Hi all,
I am glad to announce the release of WBPP 2.5.0! This release contains a list of important new features, improvements, and bug fixes. This update is distributed along with the Image Calibration v1.9.7 and Image Registration v1.2.2, and both versions are required to operate correctly.
The index of the changes is the following:
1. New Caching System
An integrated caching system allows the re-execution of WBPP after some parameter tuning without the need to execute the whole pipeline again.
The common scenario is when we tune some Local Normalization parameters, or we change the Weighting system; in such cases, the initial calibration, cosmetic correction, debayering, and registration processes do not need to be executed again since they would produce the same data that has been created in the previous WBPP run.
For each step, the caching system memorizes the input data and its timestamp, the parameters of the step, and checks if the output data of a previous run exists and remains unchanged. If all these conditions are satisfied, then WBPP uses the existing output data; otherwise, the step is regularly executed.
Each time cached data is used, a proper message is reported in the Execution Monitor, as depicted below:
The cache is persisted across sessions, so when WBPP restarts, the cache information gets reloaded. Once the dataset has been successfully processed, it's possible to purge the current cache data with the global button Purge cache, which also reports the amount of memory occupied by the cache data:
The cache is also stored inside the script icon data, so if a WBPP script icon is saved along with the project, it can be restored, and WBPP can be furtherly tuned and executed, taking advantage of the caching system.
2. New tab Pipeline
With this new WBPP version, a new Pipeline tab has been added to provide details about the execution Pipeline that has now become a primary concept.
The execution pipeline represents a sequence of operations that WBPP schedules for its execution. With this update, the list of steps has been extended to increase the granularity of the information about what is going to be executed and the results of each step. Furthermore, this concept will have a central role in further developments focused on automation capabilities.
Each step of the pipeline is called Operation and has a descriptive name and an optional reference to the group it will work with. A column Estimated Disk Space reports each operation's estimated disk space required.
The Active Steps control has been moved from the Post-Calibration panel into the Pipeline panel and is now interactive (i.e. we can enable/disable the listed steps from here).
3. Execution reorganization
The sequence of the operations has been reorganized to group the Local Normalization interactive modes together in a row. In this way, user intervention is required once for all these operations instead of sparsely during the WBPP execution.
4. Pipeline's event script
A new pipeline script can be defined to provide further support on automation. This JS script is invoked when the following events occur:
5. Astrometric Solution handling
WBPP 2.5.0 allows enabling the autonomous detection of the astrometric solution for the dataset. This works for the registered and integrated images only and leverages the ImageSolver script capabilities. The operations performed by WBPP to handle the astrometric solutions are the following:
6. New Local Normalization parameters
The scale parameter has been replaced by a new Grid size parameter. This parameter divides the maximum dimension of the image max(width, height) to obtain the normalization scale to use. This approach makes the normalization grid choice independent of the image's resolution.
A new parameter maximum integrated frames has been added to control the maximum number of frames to be integrated to generate the local normalization reference frame. It may occur that you need to increase this value (by default, set to 20) when you need more integration time to obtain a significant SNR in the integrated image. This is especially true if the session handles short-exposure frames or if your target is extremely faint.
7. Local Normalization Interactive Mode updated
It may happen that the image auto stretch needs to be manually boosted to highlight the quality of the background gradient when selecting the local normalization reference frame. For this purpose, two controls have been added to adjust the shadows and midtones of the preview.
To improve the GUI usability, the button to exclude frames from being integrated now becomes Include when an excluded frame is selected.
The integrated reference frame's file name now contains the reference to the post-calibration group:
8. Separated Channel selection
When the debayer's Separated Channel mode is active, selecting which channels we would like to proceed with is now possible. This is particularly useful when narrowband data is collected using OSC cameras; in those cases, the data may reside on one or two channels only (like H-alpha emission mainly affecting the R channel or O3 mainly affecting the G and B channels) and processing the other channels may be worthless.
As a logical consequence, RGB recombination operation will be available only if all R, G, and B separated channels are active.
9. Drizzle Integration support has been added
Drizzle Integration can be enabled on each post-calibration group. When drizzle is enabled, a subsequent drizzle operation will be scheduled after image integration.
Drizzle Integration is now suggested as the best integration method to be used when a reasonable number of well-dithered frames are provided (15-20 and above), especially for CFA images where it may produce superior results.
Drizzle is not currently available for separated channels.
10. Overscan handling improved
When overscan is enabled, the master flat and light frames will be cropped to the overscan region defined in the overscan parameters:
With this new release, two special keywords are stored in the masterFlat's header: PREOVSCW and PREOVSCH, which are the width and height of the uncalibrated flat frames. This information is handled by WBPP in order to properly match the correct flat group when the overscan is enabled. In particular, if this metadata is present, the matching group maintains the original size in order to properly match the uncalibrated light frames group:
To properly set a correct overscan handling, it is strongly recommended to let WBPP generate the master flat adding the flat frames to the session instead of using a manually generated master flat.
11. Execution Monitor updated
The Execution Monitor window has been updated to provide better feedback on the execution of each step. A new column called Elapsed shows the amount of time spent on each past step and the elapsed time of the active step, while a new column called Notes contains short messages reporting useful execution information for informative or diagnostic purposes.
Moreover, on opening, the window size is now adjusted to accommodate the content of the columns.
12. Linear Defect Detection - Linear Pattern Subtraction
When enabled, the detected linear defects are saved in a text file named with the group it refers to:
Moreover, the global rejection limit parameter is set to the maximum value between 5 and the rejection limit control value.
13. New Bad Frames Rejection operation
A new "Bad frames rejection" step is added to the pipeline right after frames measurements to remove bad frames before executing Registration, Local Normalization, and Image Integration. WBPP will normalize the weights of the frames in the post-calibration group and disables the ones with a threshold lower than the value set in the Image Integration minimum weight parameter.
The rejection is not performed if the active weighting schema is PSF Scale SNR weighting since, in that case, the local normalization process needs to be performed in order to generate the weights.
The Minimum weight's default value has also been increased to 0.05.
14. Process Container script
At the end of each execution, WBPP creates a script named with the execution timestamp and the string "_ProcessContainer."
This script can be used to launch a process container that contains all processes executed by WBPP during the correspondent run. The script can be opened from the script editor or simply dragged into the PixInsight window to be quickly loaded.
Once loaded, you can run it with the Compile & Run button, and the process container will be created:
15. Compatibility with pre-calibrated master dark removed
The option Contains Bias option, and the correspondent column in the darks table has been removed since the compatibility with pre-calibrated master darks has been removed.
WBPP is now compatible with the master darks generated by simply averaging dark frames without any preprocessing involved.
16. Compact GUI manual mode
Due to the wide variety of screen resolutions and pixel scaling on different operating systems, the compact GUI mode is no more forced but manually enabled/disabled.
A new global checkbox Compact GUI is added; by checking it, the WBPP will switch to the compact GUI mode.
When the compact GUI is enabled, the right-most global options and the panel-specific right options are mutually exclusive and only one at a time is visible. By clicking the top-right button (visible only in compact mode), the two panels can be switched on and off.
In compact mode, when the width of the window is too small, the buttons to add Bias, Dark, Flat, Light frames are grouped under a drop-down menu control:
17. Presets
Preset parameters have been updated:
Enjoy this rich new version, and, as always, feel free to report issues, feedback, and suggestions on further possible improvements!
Robyx
I am glad to announce the release of WBPP 2.5.0! This release contains a list of important new features, improvements, and bug fixes. This update is distributed along with the Image Calibration v1.9.7 and Image Registration v1.2.2, and both versions are required to operate correctly.
The index of the changes is the following:
- New caching system
- New tab Pipeline
- Pipeline's structure updated and execution order reorganized
- Pipeline's event script [more details in a dedicated thread]
- Astrometric solution handling
- New Local Normalization parameters
- Local Normalization interactive mode updated
- Separated Channel selection added
- Drizzle Integration support added
- Overscan handling improved
- Execution Monitor updated
- LDD+LPS updates
- New Bad frames rejection operation added
- New auto-generated process container script added
- Compatibility with pre-calibrated master darks removed
- Compact GUI manual enabling
- Presets updated
- Other minor updates
- Bugfixes
1. New Caching System
An integrated caching system allows the re-execution of WBPP after some parameter tuning without the need to execute the whole pipeline again.
The common scenario is when we tune some Local Normalization parameters, or we change the Weighting system; in such cases, the initial calibration, cosmetic correction, debayering, and registration processes do not need to be executed again since they would produce the same data that has been created in the previous WBPP run.
For each step, the caching system memorizes the input data and its timestamp, the parameters of the step, and checks if the output data of a previous run exists and remains unchanged. If all these conditions are satisfied, then WBPP uses the existing output data; otherwise, the step is regularly executed.
Each time cached data is used, a proper message is reported in the Execution Monitor, as depicted below:
The cache is persisted across sessions, so when WBPP restarts, the cache information gets reloaded. Once the dataset has been successfully processed, it's possible to purge the current cache data with the global button Purge cache, which also reports the amount of memory occupied by the cache data:
The cache is also stored inside the script icon data, so if a WBPP script icon is saved along with the project, it can be restored, and WBPP can be furtherly tuned and executed, taking advantage of the caching system.
2. New tab Pipeline
With this new WBPP version, a new Pipeline tab has been added to provide details about the execution Pipeline that has now become a primary concept.
The execution pipeline represents a sequence of operations that WBPP schedules for its execution. With this update, the list of steps has been extended to increase the granularity of the information about what is going to be executed and the results of each step. Furthermore, this concept will have a central role in further developments focused on automation capabilities.
Each step of the pipeline is called Operation and has a descriptive name and an optional reference to the group it will work with. A column Estimated Disk Space reports each operation's estimated disk space required.
The Active Steps control has been moved from the Post-Calibration panel into the Pipeline panel and is now interactive (i.e. we can enable/disable the listed steps from here).
3. Execution reorganization
The sequence of the operations has been reorganized to group the Local Normalization interactive modes together in a row. In this way, user intervention is required once for all these operations instead of sparsely during the WBPP execution.
4. Pipeline's event script
A new pipeline script can be defined to provide further support on automation. This JS script is invoked when the following events occur:
- the pipeline starts
- the pipeline ends
- an operation starts
- an operation ends
5. Astrometric Solution handling
WBPP 2.5.0 allows enabling the autonomous detection of the astrometric solution for the dataset. This works for the registered and integrated images only and leverages the ImageSolver script capabilities. The operations performed by WBPP to handle the astrometric solutions are the following:
- All reference frames are solved during the Plate solving reference frames operation. For each reference frame:
- The Image Solver parameters are initialized by reading the metadata of the reference frame (center's coordinates, image scale, etc.)
- Distortion Correction is enabled by default and must remain enabled even when a manual intervention is done (an alert is shown, in case)
- If the astrometric solution is found, then it's stored in the reference frame.
- If the astrometric solution fails, the Image Solver interface is presented, and the user may attempt to manually adjust the parameters and run the solver again to obtain a proper solution. After as many manual attempts as needed, if the solution is found, then it is stored in the reference frame's header.
- The astrometric solution of the reference frame is propagated to each registered frame after Image Registration.
- At the end of the integration operation, if the first integrated frame contains an astrometric solution, then this solution is used as the first guess for optimizing the astrometric solution of the integrated image.
6. New Local Normalization parameters
The scale parameter has been replaced by a new Grid size parameter. This parameter divides the maximum dimension of the image max(width, height) to obtain the normalization scale to use. This approach makes the normalization grid choice independent of the image's resolution.
A new parameter maximum integrated frames has been added to control the maximum number of frames to be integrated to generate the local normalization reference frame. It may occur that you need to increase this value (by default, set to 20) when you need more integration time to obtain a significant SNR in the integrated image. This is especially true if the session handles short-exposure frames or if your target is extremely faint.
7. Local Normalization Interactive Mode updated
It may happen that the image auto stretch needs to be manually boosted to highlight the quality of the background gradient when selecting the local normalization reference frame. For this purpose, two controls have been added to adjust the shadows and midtones of the preview.
To improve the GUI usability, the button to exclude frames from being integrated now becomes Include when an excluded frame is selected.
The integrated reference frame's file name now contains the reference to the post-calibration group:
8. Separated Channel selection
When the debayer's Separated Channel mode is active, selecting which channels we would like to proceed with is now possible. This is particularly useful when narrowband data is collected using OSC cameras; in those cases, the data may reside on one or two channels only (like H-alpha emission mainly affecting the R channel or O3 mainly affecting the G and B channels) and processing the other channels may be worthless.
As a logical consequence, RGB recombination operation will be available only if all R, G, and B separated channels are active.
9. Drizzle Integration support has been added
Drizzle Integration can be enabled on each post-calibration group. When drizzle is enabled, a subsequent drizzle operation will be scheduled after image integration.
Drizzle Integration is now suggested as the best integration method to be used when a reasonable number of well-dithered frames are provided (15-20 and above), especially for CFA images where it may produce superior results.
Drizzle is not currently available for separated channels.
10. Overscan handling improved
When overscan is enabled, the master flat and light frames will be cropped to the overscan region defined in the overscan parameters:
With this new release, two special keywords are stored in the masterFlat's header: PREOVSCW and PREOVSCH, which are the width and height of the uncalibrated flat frames. This information is handled by WBPP in order to properly match the correct flat group when the overscan is enabled. In particular, if this metadata is present, the matching group maintains the original size in order to properly match the uncalibrated light frames group:
To properly set a correct overscan handling, it is strongly recommended to let WBPP generate the master flat adding the flat frames to the session instead of using a manually generated master flat.
11. Execution Monitor updated
The Execution Monitor window has been updated to provide better feedback on the execution of each step. A new column called Elapsed shows the amount of time spent on each past step and the elapsed time of the active step, while a new column called Notes contains short messages reporting useful execution information for informative or diagnostic purposes.
Moreover, on opening, the window size is now adjusted to accommodate the content of the columns.
12. Linear Defect Detection - Linear Pattern Subtraction
When enabled, the detected linear defects are saved in a text file named with the group it refers to:
Moreover, the global rejection limit parameter is set to the maximum value between 5 and the rejection limit control value.
13. New Bad Frames Rejection operation
A new "Bad frames rejection" step is added to the pipeline right after frames measurements to remove bad frames before executing Registration, Local Normalization, and Image Integration. WBPP will normalize the weights of the frames in the post-calibration group and disables the ones with a threshold lower than the value set in the Image Integration minimum weight parameter.
The rejection is not performed if the active weighting schema is PSF Scale SNR weighting since, in that case, the local normalization process needs to be performed in order to generate the weights.
The Minimum weight's default value has also been increased to 0.05.
14. Process Container script
At the end of each execution, WBPP creates a script named with the execution timestamp and the string "_ProcessContainer."
This script can be used to launch a process container that contains all processes executed by WBPP during the correspondent run. The script can be opened from the script editor or simply dragged into the PixInsight window to be quickly loaded.
Once loaded, you can run it with the Compile & Run button, and the process container will be created:
15. Compatibility with pre-calibrated master dark removed
The option Contains Bias option, and the correspondent column in the darks table has been removed since the compatibility with pre-calibrated master darks has been removed.
WBPP is now compatible with the master darks generated by simply averaging dark frames without any preprocessing involved.
16. Compact GUI manual mode
Due to the wide variety of screen resolutions and pixel scaling on different operating systems, the compact GUI mode is no more forced but manually enabled/disabled.
A new global checkbox Compact GUI is added; by checking it, the WBPP will switch to the compact GUI mode.
When the compact GUI is enabled, the right-most global options and the panel-specific right options are mutually exclusive and only one at a time is visible. By clicking the top-right button (visible only in compact mode), the two panels can be switched on and off.
In compact mode, when the width of the window is too small, the buttons to add Bias, Dark, Flat, Light frames are grouped under a drop-down menu control:
17. Presets
Preset parameters have been updated:
- Mid-quality preset: sets the number of maximum stars to 500.
- Fast preset: leaves the automatic pedestal settings unchanged.
- The default output directory for saving the smart report content is set to be the logs folder.
- The Dialog is disabled while adding new files.
- The progress of file addition or migration is tracked in the console.
- The reset panel has been updated and allows you to select separately to reset the parameters and/or the file lists.
- The color space of colorized images in post-calibration groups is reported as "RGB" instead of "CFA."
- The log file is updated and saved after each pipeline step instead of being entirely written at the very end of the pipeline execution. This allows storing the progress until a critical error occurs (in which case, nothing was saved).
- The RGB Recombination operation is performed only if Image Integration is enabled.
- fixed the occasional local normalization's max number of stars out-of-range message on start
- best M* can now be used as criteria to generate the local normalization reference frame under Windows
- Local Normalization interactive mode: the best frame is properly selected on exit
- frame size is correctly read using the standard input hints
- fixed a keyword handling issue that may cause a random group duplication
- fixed the application of the overscan regions
- fixed the wrong reference frame by keyword reported in the GUI
- fixed the reloading of the light exposure tolerance value
- fixed the wrong saving of the output directory containing accented characters
- fixed some tooltip typo
Enjoy this rich new version, and, as always, feel free to report issues, feedback, and suggestions on further possible improvements!
Robyx
Last edited:
thanks for reporting and for your kind words!!
