WBPP 2.5.0 Released

Status
Not open for further replies.

robyx

Administrator
Staff member
Hi all,

I am glad to announce the release of WBPP 2.5.0! This release contains a list of important new features, improvements, and bug fixes. This update is distributed along with the Image Calibration v1.9.7 and Image Registration v1.2.2, and both versions are required to operate correctly.

The index of the changes is the following:
  1. New caching system
  2. New tab Pipeline
  3. Pipeline's structure updated and execution order reorganized
  4. Pipeline's event script [more details in a dedicated thread]
  5. Astrometric solution handling
  6. New Local Normalization parameters
  7. Local Normalization interactive mode updated
  8. Separated Channel selection added
  9. Drizzle Integration support added
  10. Overscan handling improved
  11. Execution Monitor updated
  12. LDD+LPS updates
  13. New Bad frames rejection operation added
  14. New auto-generated process container script added
  15. Compatibility with pre-calibrated master darks removed
  16. Compact GUI manual enabling
  17. Presets updated
  18. Other minor updates
  19. Bugfixes
Below you can find more details for each point.

1. New Caching System
An integrated caching system allows the re-execution of WBPP after some parameter tuning without the need to execute the whole pipeline again.
The common scenario is when we tune some Local Normalization parameters, or we change the Weighting system; in such cases, the initial calibration, cosmetic correction, debayering, and registration processes do not need to be executed again since they would produce the same data that has been created in the previous WBPP run.
For each step, the caching system memorizes the input data and its timestamp, the parameters of the step, and checks if the output data of a previous run exists and remains unchanged. If all these conditions are satisfied, then WBPP uses the existing output data; otherwise, the step is regularly executed.

Each time cached data is used, a proper message is reported in the Execution Monitor, as depicted below:
WBPP_Cache.png


The cache is persisted across sessions, so when WBPP restarts, the cache information gets reloaded. Once the dataset has been successfully processed, it's possible to purge the current cache data with the global button Purge cache, which also reports the amount of memory occupied by the cache data:

WBPP_PurgeCache.png

The cache is also stored inside the script icon data, so if a WBPP script icon is saved along with the project, it can be restored, and WBPP can be furtherly tuned and executed, taking advantage of the caching system.

2. New tab Pipeline
With this new WBPP version, a new Pipeline tab has been added to provide details about the execution Pipeline that has now become a primary concept.
The execution pipeline represents a sequence of operations that WBPP schedules for its execution. With this update, the list of steps has been extended to increase the granularity of the information about what is going to be executed and the results of each step. Furthermore, this concept will have a central role in further developments focused on automation capabilities.

Each step of the pipeline is called Operation and has a descriptive name and an optional reference to the group it will work with. A column Estimated Disk Space reports each operation's estimated disk space required.

The Active Steps control has been moved from the Post-Calibration panel into the Pipeline panel and is now interactive (i.e. we can enable/disable the listed steps from here).

WBPP_NewPipelineTab.png


3. Execution reorganization
The sequence of the operations has been reorganized to group the Local Normalization interactive modes together in a row. In this way, user intervention is required once for all these operations instead of sparsely during the WBPP execution.

WBPP_PipelineTab_LN_Grouped.png


4. Pipeline's event script
A new pipeline script can be defined to provide further support on automation. This JS script is invoked when the following events occur:
  • the pipeline starts
  • the pipeline ends
  • an operation starts
  • an operation ends
The script receives information on which event triggered its execution and further details about it. More details are provided in a dedicated thread.
WBPP_PipelineScript.png


5. Astrometric Solution handling
WBPP 2.5.0 allows enabling the autonomous detection of the astrometric solution for the dataset. This works for the registered and integrated images only and leverages the ImageSolver script capabilities. The operations performed by WBPP to handle the astrometric solutions are the following:

  1. All reference frames are solved during the Plate solving reference frames operation. For each reference frame:
    1. The Image Solver parameters are initialized by reading the metadata of the reference frame (center's coordinates, image scale, etc.)
    2. Distortion Correction is enabled by default and must remain enabled even when a manual intervention is done (an alert is shown, in case)
    3. If the astrometric solution is found, then it's stored in the reference frame.
    4. If the astrometric solution fails, the Image Solver interface is presented, and the user may attempt to manually adjust the parameters and run the solver again to obtain a proper solution. After as many manual attempts as needed, if the solution is found, then it is stored in the reference frame's header.
  2. The astrometric solution of the reference frame is propagated to each registered frame after Image Registration.
  3. At the end of the integration operation, if the first integrated frame contains an astrometric solution, then this solution is used as the first guess for optimizing the astrometric solution of the integrated image.
At the end of the execution, all registered frames and master lights will contain an astrometric solution.

WBPP_AstrometricSolution.png
WBPP_AstrometricSolutionOperation.png


6. New Local Normalization parameters
The scale parameter has been replaced by a new Grid size parameter. This parameter divides the maximum dimension of the image max(width, height) to obtain the normalization scale to use. This approach makes the normalization grid choice independent of the image's resolution.

A new parameter maximum integrated frames has been added to control the maximum number of frames to be integrated to generate the local normalization reference frame. It may occur that you need to increase this value (by default, set to 20) when you need more integration time to obtain a significant SNR in the integrated image. This is especially true if the session handles short-exposure frames or if your target is extremely faint.

WBPP_LocalNormalizationParameters.png


7. Local Normalization Interactive Mode updated
It may happen that the image auto stretch needs to be manually boosted to highlight the quality of the background gradient when selecting the local normalization reference frame. For this purpose, two controls have been added to adjust the shadows and midtones of the preview.

To improve the GUI usability, the button to exclude frames from being integrated now becomes Include when an excluded frame is selected.
WBPP_LN_Interactive.jpg


The integrated reference frame's file name now contains the reference to the post-calibration group:
WBPP_LN_Interactive_IntegratedMasterName.jpg
WBPP_LN_Interactive_IntegratedMasterName_Dolphin.png

8. Separated Channel selection
When the debayer's Separated Channel mode is active, selecting which channels we would like to proceed with is now possible. This is particularly useful when narrowband data is collected using OSC cameras; in those cases, the data may reside on one or two channels only (like H-alpha emission mainly affecting the R channel or O3 mainly affecting the G and B channels) and processing the other channels may be worthless.

As a logical consequence, RGB recombination operation will be available only if all R, G, and B separated channels are active.

WBPP_SeparatedChannels_R_Only.png
WBPP_SeparatedChannels_R_Only_Pipeline.png

9. Drizzle Integration support has been added
Drizzle Integration can be enabled on each post-calibration group. When drizzle is enabled, a subsequent drizzle operation will be scheduled after image integration.
Drizzle Integration is now suggested as the best integration method to be used when a reasonable number of well-dithered frames are provided (15-20 and above), especially for CFA images where it may produce superior results.

Drizzle is not currently available for separated channels.
WBPP_Drizzle.png
WBPP_Drizzle_Pipeline.png

10. Overscan handling improved
When overscan is enabled, the master flat and light frames will be cropped to the overscan region defined in the overscan parameters:
WBPP_OverscanSettings.png

With this new release, two special keywords are stored in the masterFlat's header: PREOVSCW and PREOVSCH, which are the width and height of the uncalibrated flat frames. This information is handled by WBPP in order to properly match the correct flat group when the overscan is enabled. In particular, if this metadata is present, the matching group maintains the original size in order to properly match the uncalibrated light frames group:

WBPP_Overscan_MasterFlatMetadata.png
WBPP_Overscan_MasterFlatGropuing.png


To properly set a correct overscan handling, it is strongly recommended to let WBPP generate the master flat adding the flat frames to the session instead of using a manually generated master flat.

11. Execution Monitor updated
The Execution Monitor window has been updated to provide better feedback on the execution of each step. A new column called Elapsed shows the amount of time spent on each past step and the elapsed time of the active step, while a new column called Notes contains short messages reporting useful execution information for informative or diagnostic purposes.
Moreover, on opening, the window size is now adjusted to accommodate the content of the columns.
WBPP_ExecutionMonitor.png



12. Linear Defect Detection - Linear Pattern Subtraction
When enabled, the detected linear defects are saved in a text file named with the group it refers to:

WBPP_LDD_defects.png

Moreover, the global rejection limit parameter is set to the maximum value between 5 and the rejection limit control value.



13. New Bad Frames Rejection operation
A new "Bad frames rejection" step is added to the pipeline right after frames measurements to remove bad frames before executing Registration, Local Normalization, and Image Integration. WBPP will normalize the weights of the frames in the post-calibration group and disables the ones with a threshold lower than the value set in the Image Integration minimum weight parameter.
The rejection is not performed if the active weighting schema is PSF Scale SNR weighting since, in that case, the local normalization process needs to be performed in order to generate the weights.

The Minimum weight's default value has also been increased to 0.05.

WBPP_MinimumWeight.png
WBPP_MinimumWeight_Rejected.png



14. Process Container script
At the end of each execution, WBPP creates a script named with the execution timestamp and the string "_ProcessContainer."

WBPP_ProcessContainerScript.png

This script can be used to launch a process container that contains all processes executed by WBPP during the correspondent run. The script can be opened from the script editor or simply dragged into the PixInsight window to be quickly loaded.
Once loaded, you can run it with the Compile & Run button, and the process container will be created:

WBPP_ProcessContainerExecution.jpg



15. Compatibility with pre-calibrated master dark removed
The option Contains Bias option, and the correspondent column in the darks table has been removed since the compatibility with pre-calibrated master darks has been removed.
WBPP is now compatible with the master darks generated by simply averaging dark frames without any preprocessing involved.
WBPP_ContainsBiasRemoved.png



16. Compact GUI manual mode
Due to the wide variety of screen resolutions and pixel scaling on different operating systems, the compact GUI mode is no more forced but manually enabled/disabled.
A new global checkbox Compact GUI is added; by checking it, the WBPP will switch to the compact GUI mode.

WBPP_CompactGUI_Checkbox.png


When the compact GUI is enabled, the right-most global options and the panel-specific right options are mutually exclusive and only one at a time is visible. By clicking the top-right button (visible only in compact mode), the two panels can be switched on and off.

WBPP_CompactGUI_Enabled.png
WBPP_CompactGUI_Enabled_OptionsMode.png


In compact mode, when the width of the window is too small, the buttons to add Bias, Dark, Flat, Light frames are grouped under a drop-down menu control:

WBPP_CompactGUI_AddFiles.png


17. Presets
Preset parameters have been updated:
  • Mid-quality preset: sets the number of maximum stars to 500.
  • Fast preset: leaves the automatic pedestal settings unchanged.
18. Other improvements
  • The default output directory for saving the smart report content is set to be the logs folder.
  • The Dialog is disabled while adding new files.
  • The progress of file addition or migration is tracked in the console.
  • The reset panel has been updated and allows you to select separately to reset the parameters and/or the file lists.
  • The color space of colorized images in post-calibration groups is reported as "RGB" instead of "CFA."
  • The log file is updated and saved after each pipeline step instead of being entirely written at the very end of the pipeline execution. This allows storing the progress until a critical error occurs (in which case, nothing was saved).
  • The RGB Recombination operation is performed only if Image Integration is enabled.
19. Bugfixes
  • fixed the occasional local normalization's max number of stars out-of-range message on start
  • best M* can now be used as criteria to generate the local normalization reference frame under Windows
  • Local Normalization interactive mode: the best frame is properly selected on exit
  • frame size is correctly read using the standard input hints
  • fixed a keyword handling issue that may cause a random group duplication
  • fixed the application of the overscan regions
  • fixed the wrong reference frame by keyword reported in the GUI
  • fixed the reloading of the light exposure tolerance value
  • fixed the wrong saving of the output directory containing accented characters
  • fixed some tooltip typo

Enjoy this rich new version, and, as always, feel free to report issues, feedback, and suggestions on further possible improvements!
Robyx
 
Last edited:
Thank you very much @robyx (and all the Pixinsight Staff involved) for this great tool. I have already used this new version and I'm especially happy with the caching system, the improved execution monitor and the integration of the astrometric solver. Very well done

PS: I have noticed a small typo in the tooltip of the "Event script" checkbox wich doesn't seem to be the correct one.
 

Attachments

  • Event Script tooltip.jpg
    Event Script tooltip.jpg
    36.5 KB · Views: 171
Last edited:
Thank you very much @robyx (and all the Pixinsight Staff involved) for this great tool. I have already used this new version and I'm especially happy with the caching system, the improved execution monitor and the integration of the astrometric solver. Very well done

PS: I have noticed a small typo in the tooltip of the "Event script" checkbox wich doesn't seem to be the correct one.
Hi @nico1038, definitely the tooltip is wrongly mapped :) thanks for reporting and for your kind words!!
 
Hello Robyx, how can I combine images from different RGB camera setups and different binings into one Masterlight in wbpp? At the moment all images are registered to a reference image, but from the LN each setup is normalized and integrated separately. Now when these stacks are combined in pixelmath, the result is sub-optimal because the stacked images are not normalized to a common reference. Wouldn't it be possible to normalize and integrate the images in wbpp to a reference. The weighting method PSFscaleSNR would make something like that possible.

With kind regards.
 
Hello Robyx, how can I combine images from different RGB camera setups and different binings into one Masterlight in wbpp? At the moment all images are registered to a reference image, but from the LN each setup is normalized and integrated separately. Now when these stacks are combined in pixelmath, the result is sub-optimal because the stacked images are not normalized to a common reference. Wouldn't it be possible to normalize and integrate the images in wbpp to a reference. The weighting method PSFscaleSNR would make something like that possible.

With kind regards.
Hi @Minusman,

Yes, the grouping of frames with different binning is not supported in WBPP. Integrating frames with different binning together is not the way to go IMHO; any weighting system would lead to a biased weights population where the best group receives high weights while the worse group may receive very low weights, to the point that it can become irrelevant or even discarded. Differently, separating data by binning leads to a more coherent weighting system within the same group, extracting the best possible data from each master light generated. At that point, it's up to you and the objective you have to mix them properly.

I agree It could make sense to have a common local normalization reference frame (the one with the highest SNR), but WBP does not support that at all, and I think it should deserve a proper investigation to see if it's really worth the effort. In the end, you can always apply local normalization to one master referencing the other and then merge them with the approach that is best for our objective.
 
Amazing update Robyx. ? Again, lots of new features to test and enjoy. Thank you for introducing drizzle into the mix. Been waiting for that one. :)

One question: Is it possible to activate bayer drizzle to CFA data?
 
Amazing update Robyx. ? Again, lots of new features to test and enjoy. Thank you for introducing drizzle into the mix. Been waiting for that one. :)

One question: Is it possible to activate bayer drizzle to CFA data?
Thanks @Grim408!!! Yes, not only it is possible, but it is even suggested as a better approach if you have enough well-dithered frames!
Check it out!
 
Thanks @Grim408!!! Yes, not only it is possible, but it is even suggested as a better approach if you have enough well-dithered frames!
Check it out!

Great, reason I ask is that I don't see any option to enable CFA drizzle so I guess this is enabled automatically?
 
Great, reason I ask is that I don't see any option to enable CFA drizzle so I guess this is enabled automatically?
of course! When you enable drizzle for a colorized group it handles the Drizzle configuration properly. You don't need to set it since it's a forced settings that depends on the colorspace, you're only free to tune the other drizzle parameters :)
 
of course! When you enable drizzle for a colorized group it handles the Drizzle configuration properly. You don't need to set it since it's a forced settings that depends on the colorspace, you're only free to tune the other drizzle parameters :)

Yeah that's what I thought but wasn't sure, thank you for clarifying that and thank you again for the hard work. This is great stuff. :eek:
 
@robyx Incredible how many improvements are you injecting in WBPP in the last months!

One remark: I received no notification of this announcement post. Normally I do and I made no changes to the account settings. Maybe is a generalized issue?
 
@robyx Incredible how many improvements are you injecting in WBPP in the last months!

One remark: I received no notification of this announcement post. Normally I do and I made no changes to the account settings. Maybe is a generalized issue?
I guess you're watching the Announcement forum section. Maybe it has been moved to the spam folder of your email? I am not aware of any problem regarding the email sending...
 
Awesome work, tell everyone thank you on our behalf!

Quick question, is there any idea on when CUDA acceleration is planned to roll out? I'm really excited for this potential feature, as everything else its been implemented in that I use has seen immense speed of processing gains, often times being twice as fast or more than my 14core/28thread cpu. Video cards these days are monsters compared to those of yesteryear.

Keep up the good work and thank you again!
 
Awesome work, tell everyone thank you on our behalf!

Quick question, is there any idea on when CUDA acceleration is planned to roll out? I'm really excited for this potential feature, as everything else its been implemented in that I use has seen immense speed of processing gains, often times being twice as fast or more than my 14core/28thread cpu. Video cards these days are monsters compared to those of yesteryear.

Keep up the good work and thank you again!
some experimental GPU implementations exist, but this front is on hold since we're busy implementing other more strategic tools. We'll continue working on it asap.
 
When will this be available for updating? Checked 5 minutes ago and PI says no updates available.
it is already available now. Recheck and ensure your connection is good, you should see it if you have PI 1.8.9-1 installed.
 
Thanks for constantly improving WBPP!

One question. In the 'Channels Configuration' do I need to select 'Combined + Separate' output to get the advantage of distortion aligning the channels for minimum chromatic aberration? Will only the RGB channels have the distortion correction applied and then need to be combined manually in into an RGB image, or can I just output 'Combined RGB' and get the benefit of reduced CA?

All the best
 
Thanks for constantly improving WBPP!

One question. In the 'Channels Configuration' do I need to select 'Combined + Separate' output to get the advantage of distortion aligning the channels for minimum chromatic aberration? Will only the RGB channels have the distortion correction applied and then need to be combined manually in into an RGB image, or can I just output 'Combined RGB' and get the benefit of reduced CA?

All the best
The "Combined RBG" method aligns the colorized RGB image, the "Separated RGB" takes the advantage of separating the Red, Green, and Blue channels and aligns them all on the best Green frame enabling the distortion correction in order to minimize the chromatic aberration that affects the recombined RBG image.
You can select "Separate RBG" to reduce the chromatic aberration of the recombined rgb image and check the "Recombine RGB" option to let WBPP generate the recombined colorized image for you.

If you select Combined + Separated, then WBPP will generate both masters with the two approaches; this could be useful if you want to compare the two methods' differences.

[edited] by the way, nice nickname :)
 
Status
Not open for further replies.
Back
Top