WBPP 2.5.0 Released

Status
Not open for further replies.
Hi all,

I am glad to announce the release of WBPP 2.5.0! This release contains a list of important new features, improvements, and bug fixes. This update is distributed along with the Image Calibration v1.9.7 and Image Registration v1.2.2, and both versions are required to operate correctly.

The index of the changes is the following:
  1. New caching system
  2. New tab Pipeline
  3. Pipeline's structure updated and execution order reorganized
  4. Pipeline's event script [more details in a dedicated thread]
  5. Astrometric solution handling
  6. New Local Normalization parameters
  7. Local Normalization interactive mode updated
  8. Separated Channel selection added
  9. Drizzle Integration support added
  10. Overscan handling improved
  11. Execution Monitor updated
  12. LDD+LPS updates
  13. New Bad frames rejection operation added
  14. New auto-generated process container script added
  15. Compatibility with pre-calibrated master darks removed
  16. Compact GUI manual enabling
  17. Presets updated
  18. Other minor updates
  19. Bugfixes
Below you can find more details for each point.

1. New Caching System
An integrated caching system allows the re-execution of WBPP after some parameter tuning without the need to execute the whole pipeline again.
The common scenario is when we tune some Local Normalization parameters, or we change the Weighting system; in such cases, the initial calibration, cosmetic correction, debayering, and registration processes do not need to be executed again since they would produce the same data that has been created in the previous WBPP run.
For each step, the caching system memorizes the input data and its timestamp, the parameters of the step, and checks if the output data of a previous run exists and remains unchanged. If all these conditions are satisfied, then WBPP uses the existing output data; otherwise, the step is regularly executed.

Each time cached data is used, a proper message is reported in the Execution Monitor, as depicted below:
View attachment 15743

The cache is persisted across sessions, so when WBPP restarts, the cache information gets reloaded. Once the dataset has been successfully processed, it's possible to purge the current cache data with the global button Purge cache, which also reports the amount of memory occupied by the cache data:


The cache is also stored inside the script icon data, so if a WBPP script icon is saved along with the project, it can be restored, and WBPP can be furtherly tuned and executed, taking advantage of the caching system.

2. New tab Pipeline
With this new WBPP version, a new Pipeline tab has been added to provide details about the execution Pipeline that has now become a primary concept.
The execution pipeline represents a sequence of operations that WBPP schedules for its execution. With this update, the list of steps has been extended to increase the granularity of the information about what is going to be executed and the results of each step. Furthermore, this concept will have a central role in further developments focused on automation capabilities.

Each step of the pipeline is called Operation and has a descriptive name and an optional reference to the group it will work with. A column Estimated Disk Space reports each operation's estimated disk space required.

The Active Steps control has been moved from the Post-Calibration panel into the Pipeline panel and is now interactive (i.e. we can enable/disable the listed steps from here).

View attachment 15747

3. Execution reorganization
The sequence of the operations has been reorganized to group the Local Normalization interactive modes together in a row. In this way, user intervention is required once for all these operations instead of sparsely during the WBPP execution.

View attachment 15748

4. Pipeline's event script
A new pipeline script can be defined to provide further support on automation. This JS script is invoked when the following events occur:
  • the pipeline starts
  • the pipeline ends
  • an operation starts
  • an operation ends
The script receives information on which event triggered its execution and further details about it. More details are provided in a dedicated thread.
View attachment 15749

5. Astrometric Solution handling
WBPP 2.5.0 allows enabling the autonomous detection of the astrometric solution for the dataset. This works for the registered and integrated images only and leverages the ImageSolver script capabilities. The operations performed by WBPP to handle the astrometric solutions are the following:

  1. All reference frames are solved during the Plate solving reference frames operation. For each reference frame:
    1. The Image Solver parameters are initialized by reading the metadata of the reference frame (center's coordinates, image scale, etc.)
    2. Distortion Correction is enabled by default and must remain enabled even when a manual intervention is done (an alert is shown, in case)
    3. If the astrometric solution is found, then it's stored in the reference frame.
    4. If the astrometric solution fails, the Image Solver interface is presented, and the user may attempt to manually adjust the parameters and run the solver again to obtain a proper solution. After as many manual attempts as needed, if the solution is found, then it is stored in the reference frame's header.
  2. The astrometric solution of the reference frame is propagated to each registered frame after Image Registration.
  3. At the end of the integration operation, if the first integrated frame contains an astrometric solution, then this solution is used as the first guess for optimizing the astrometric solution of the integrated image.
At the end of the execution, all registered frames and master lights will contain an astrometric solution.



6. New Local Normalization parameters
The scale parameter has been replaced by a new Grid size parameter. This parameter divides the maximum dimension of the image max(width, height) to obtain the normalization scale to use. This approach makes the normalization grid choice independent of the image's resolution.

A new parameter maximum integrated frames has been added to control the maximum number of frames to be integrated to generate the local normalization reference frame. It may occur that you need to increase this value (by default, set to 20) when you need more integration time to obtain a significant SNR in the integrated image. This is especially true if the session handles short-exposure frames or if your target is extremely faint.

View attachment 15752

7. Local Normalization Interactive Mode updated
It may happen that the image auto stretch needs to be manually boosted to highlight the quality of the background gradient when selecting the local normalization reference frame. For this purpose, two controls have been added to adjust the shadows and midtones of the preview.

To improve the GUI usability, the button to exclude frames from being integrated now becomes Include when an excluded frame is selected.
View attachment 15753

The integrated reference frame's file name now contains the reference to the post-calibration group:

8. Separated Channel selection
When the debayer's Separated Channel mode is active, selecting which channels we would like to proceed with is now possible. This is particularly useful when narrowband data is collected using OSC cameras; in those cases, the data may reside on one or two channels only (like H-alpha emission mainly affecting the R channel or O3 mainly affecting the G and B channels) and processing the other channels may be worthless.

As a logical consequence, RGB recombination operation will be available only if all R, G, and B separated channels are active.


9. Drizzle Integration support has been added
Drizzle Integration can be enabled on each post-calibration group. When drizzle is enabled, a subsequent drizzle operation will be scheduled after image integration.
Drizzle Integration is now suggested as the best integration method to be used when a reasonable number of well-dithered frames are provided (15-20 and above), especially for CFA images where it may produce superior results.

Drizzle is not currently available for separated channels.

10. Overscan handling improved
When overscan is enabled, the master flat and light frames will be cropped to the overscan region defined in the overscan parameters:

With this new release, two special keywords are stored in the masterFlat's header: PREOVSCW and PREOVSCH, which are the width and height of the uncalibrated flat frames. This information is handled by WBPP in order to properly match the correct flat group when the overscan is enabled. In particular, if this metadata is present, the matching group maintains the original size in order to properly match the uncalibrated light frames group:



To properly set a correct overscan handling, it is strongly recommended to let WBPP generate the master flat adding the flat frames to the session instead of using a manually generated master flat.

I get the following error on my MAC OS (not M1) macbook.

11. Execution Monitor updated
The Execution Monitor window has been updated to provide better feedback on the execution of each step. A new column called Elapsed shows the amount of time spent on each past step and the elapsed time of the active step, while a new column called Notes contains short messages reporting useful execution information for informative or diagnostic purposes.
Moreover, on opening, the window size is now adjusted to accommodate the content of the columns.
View attachment 15741


12. Linear Defect Detection - Linear Pattern Subtraction
When enabled, the detected linear defects are saved in a text file named with the group it refers to:


Moreover, the global rejection limit parameter is set to the maximum value between 5 and the rejection limit control value.



13. New Bad Frames Rejection operation
A new "Bad frames rejection" step is added to the pipeline right after frames measurements to remove bad frames before executing Registration, Local Normalization, and Image Integration. WBPP will normalize the weights of the frames in the post-calibration group and disables the ones with a threshold lower than the value set in the Image Integration minimum weight parameter.
The rejection is not performed if the active weighting schema is PSF Scale SNR weighting since, in that case, the local normalization process needs to be performed in order to generate the weights.

The Minimum weight's default value has also been increased to 0.05.




14. Process Container script
At the end of each execution, WBPP creates a script named with the execution timestamp and the string "_ProcessContainer."


This script can be used to launch a process container that contains all processes executed by WBPP during the correspondent run. The script can be opened from the script editor or simply dragged into the PixInsight window to be quickly loaded.
Once loaded, you can run it with the Compile & Run button, and the process container will be created:




15. Compatibility with pre-calibrated master dark removed
The option Contains Bias option, and the correspondent column in the darks table has been removed since the compatibility with pre-calibrated master darks has been removed.
WBPP is now compatible with the master darks generated by simply averaging dark frames without any preprocessing involved.
View attachment 15774


16. Compact GUI manual mode
Due to the wide variety of screen resolutions and pixel scaling on different operating systems, the compact GUI mode is no more forced but manually enabled/disabled.
A new global checkbox Compact GUI is added; by checking it, the WBPP will switch to the compact GUI mode.

View attachment 15775

When the compact GUI is enabled, the right-most global options and the panel-specific right options are mutually exclusive and only one at a time is visible. By clicking the top-right button (visible only in compact mode), the two panels can be switched on and off.



In compact mode, when the width of the window is too small, the buttons to add Bias, Dark, Flat, Light frames are grouped under a drop-down menu control:



17. Presets
Preset parameters have been updated:
  • Mid-quality preset: sets the number of maximum stars to 500.
  • Fast preset: leaves the automatic pedestal settings unchanged.
18. Other improvements
  • The default output directory for saving the smart report content is set to be the logs folder.
  • The Dialog is disabled while adding new files.
  • The progress of file addition or migration is tracked in the console.
  • The reset panel has been updated and allows you to select separately to reset the parameters and/or the file lists.
  • The color space of colorized images in post-calibration groups is reported as "RGB" instead of "CFA."
  • The log file is updated and saved after each pipeline step instead of being entirely written at the very end of the pipeline execution. This allows storing the progress until a critical error occurs (in which case, nothing was saved).
  • The RGB Recombination operation is performed only if Image Integration is enabled.
19. Bugfixes
  • fixed the occasional local normalization's max number of stars out-of-range message on start
  • best M* can now be used as criteria to generate the local normalization reference frame under Windows
  • Local Normalization interactive mode: the best frame is properly selected on exit
  • frame size is correctly read using the standard input hints
  • fixed a keyword handling issue that may cause a random group duplication
  • fixed the application of the overscan regions
  • fixed the wrong reference frame by keyword reported in the GUI
  • fixed the reloading of the light exposure tolerance value
  • fixed the wrong saving of the output directory containing accented characters
  • fixed some tooltip typo

Enjoy this rich new version, and, as always, feel free to report issues, feedback, and suggestions on further possible improvements!
Robyx
 
Hi I get the following error on the latest WBPP script. MAC OS (not M1 chip). Any suggestions would be welcome.
 

Attachments

  • Screen Shot 2022-09-10 at 9.55.54 am.png
    Screen Shot 2022-09-10 at 9.55.54 am.png
    282.7 KB · Views: 88
Hi I get an error on the latest WBPP script - MAC OS 12.5.1 Monterey (Intel not M1). Is there something I need to do?
 

Attachments

  • Screen Shot 2022-09-10 at 9.55.54 am.png
    Screen Shot 2022-09-10 at 9.55.54 am.png
    282.7 KB · Views: 66
Hi I get an error on the latest WBPP script - MAC OS 12.5.1 Monterey (Intel not M1). Is there something I need to do?
Hi, for some reason, the signature of the WBPP and the WBPP script does not match. My suggestion is to ensure that the last update is installed successfully. Check the list of updates from the Resources -> Updates -> View Installed Updates; you should see something like the following for WBPP:

Installed Updates.png


is this the case?
 
I keep hitting the "check for updates and get the no updates currently available message
Are you using the last PI version 1.8.9-1? is your connection protected by a firewall? have you been able to install other updates before or does it happen with any update you try to install?
 
Hi, thank you for the update.

- in drizzle integration options I can't find Kernel function VarShape. I usually go for VarShape 1.5 which I find producing the best results with my data. Is there a reason why to limit the choice to only three options?
- I see drizzle scale is defaulted to "1". I had always assumed drizzle should be x2?
- would it be possible to apply Local Normalization only to a certain group of lights? Similar to what happens with cosmetic correction
 
- in drizzle integration options I can't find Kernel function VarShape. I usually go for VarShape 1.5 which I find producing the best results with my data. Is there a reason why to limit the choice to only three options?
a few common options have been added; there is not a particular reason why this list cannot include all other options. This could be part of the next update.

- I see drizzle scale is defaulted to "1". I had always assumed drizzle should be x2?
Drizzle can be perfectly at scale 1x as the integration method for the unscaled master lights in both monochromatic or CFA images.
Drizzle is the preferred method with enough well-dithered frames since it does not involve any interpolation.

- would it be possible to apply Local Normalization only to a certain group of lights? Similar to what happens with cosmetic correction
Not at the moment; one way with which you can obtain this is to activate the interactive mode and press "esc" to dismiss the interactive view without selecting any reference frame when the relative groups are involved. In that case, with no local normalization reference frame, local normalization will not be applied.
 
a few common options have been added; there is not a particular reason why this list cannot include all other options. This could be part of the next update.
Great thank you

Drizzle can be perfectly at scale 1x as the integration method for the unscaled master lights in both monochromatic or CFA images.
I see a marked difference in details resolution in Drizzle 2x vs 1x
1662795652828.png
 
Great thank you


I see a marked difference in details resolution in Drizzle 2x vs 1x
View attachment 15834
Of course, drizzle 2x produces smoother results with respect to 1x :)
My sentence contained a lexical error, I meant "Drizzle can be perfectly used at scale 1x" with any proper dataset. The meaningful comparison, in that sense, should be done between drizzle 1x and the standard Image Integration result since both are at the same scale.
 
Hi, and thanks for the update :)

I get an error when I try to execute the script.
Everything is up to date. I haven't used WBPP in a while so I'm not sure if it is specifically related to this version or not.
1662797712824.png1662797697310.png
 
I guess you're watching the Announcement forum section. Maybe it has been moved to the spam folder of your email? I am not aware of any problem regarding the email sending...
I received the announcement mail a couple of hours after our conversation - no issues of any sort. I was just to fast in accessing the forum after I saw the update coming on my system.
Too much PixInsight addiction!
 
The solution to this error has been posted in Bug Reports. It's a problem with an old .plist file.

Indeed, see here.
Thanks :)

EDIT : Well, it turns out there is no PixInsight.plist file on windows machines. Instead, edit Pixinsight.ini and remove all entries related to the WBPP script
 
Last edited:
Many thanks for the huge number of new features in the new script!

One comment to point 8 in you list:
8. Separated Channel selection

from what I understand this option works globally and for that reason it does not reach it's full potential. (Or did I miss something here?)
In worst case scenario I use RGB stars, a dual narrowband filter and a Sii filter with my OSC camera. I will not get any reduction of stacking time because of the RGB stars as I cannot disable any color channel.
In a scenario with Sii and dual narrowband in the stack I will gain a bit by turning off processing for blue but I still cannot reach full potential because the green channel of the Sii filter is still processed.
To reach full potential in all cases it should be possible to be able to disable processing for each set of lights, please refer to the mockup in my post:
 
I noticed about the update of WBPP V2.5 and I have a problem with the buttons of "accept" and "cancel" at the end of the window. I have tried to change the window resolution and the problem is not solved. Anyone else with the same problem? How could correct it?
I remember 3-4 months ago the same problem with the same script WBPP
My operating system is windows 10 home x64
 

Attachments

  • Captura de pantalla 2022-09-10 102646.png
    Captura de pantalla 2022-09-10 102646.png
    66.9 KB · Views: 45
I noticed about the update of WBPP V2.5 and I have a problem with the buttons of "accept" and "cancel" at the end of the window. I have tried to change the window resolution and the problem is not solved. Anyone else with the same problem? How could correct it?
I remember 3-4 months ago the same problem with the same script WBPP
My operating system is windows 10 home x64
You may enable the compact GUI by checking the correspondent global checkbox, this will resize the window enough to run the wbpp.
 
Hi, for some reason, the signature of the WBPP and the WBPP script does not match. My suggestion is to ensure that the last update is installed successfully. Check the list of updates from the Resources -> Updates -> View Installed Updates; you should see something like the following for WBPP:

View attachment 15827

is this the case?
Hi Rob, It looks OK from your screen shot. Please see attachment.
 

Attachments

  • Screen Shot 2022-09-11 at 7.16.50 am.png
    Screen Shot 2022-09-11 at 7.16.50 am.png
    429.5 KB · Views: 65
Status
Not open for further replies.
Back
Top