WBPP 2.4.0 Released

[robyx]

Hi all,

I am glad to announce a new version of the Weighted Batch Pre Processing WBPP script 2.4.0. This update is distributed with the new PixInsight core application 1.8.9 and requires it as the minimum version to run. This release integrates the new features, including the new PSFSW and PSF SNR weighting algorithms, the new Local Normalization process and the new calibration output pedestal "auto" mode, and implements other improvements and bug fixes.


PSF Signal Weight and PSF SNR weighting algorithms
The new PSF Signal Weight and PSF SNR weighting algorithms are available in the Subframe Weighting panel:

​

and both can be used as weighting factors under the custom formula panel:

​

A new Local Normalization step
A Local Normalization step is now available as an intermediate step performed after the registration and before image integration. When activated, WBPP finds the best frame for each post-process group and normalizes the others with respect to it. If the Reference Frame Generation option is set to "Integration of best frames" then a subset of the best frames in the group are integrated and the integrated image is used as the normalization reference.

Rejection methods updates

• The new RCR rejection method is now available as a possible choice amongst the rejection methods:

• the new RCR Limit parameter has been added
• the ESD Low Rejection parameter has been removed as it has been marked as deprecated
• the rejection methods list has been shortened and now contains the set of best rejection methods while Min/Max, Sigma Clipping, Averaged Sigma Clipping and CCD Noise Model have been removed
• the "auto" rejection method now selects the rejection method depending on the number of frames as follows:

Number of frames	Method
< 6​	Percentile clipping
< 15​	Winsorized Sigma Clipping
>= 15​	Generalized Extreme Studentized Deviate

New Output Pedestal "auto" mode
The new output pedestal "auto" mode can be enabled for each light frame group by selecting the correspondent Mode in the Output Pedestal Settings panel:

Other Interface Changes

• the panel "Post-process" is now renamed "Post-Calibration"
• in Post-Calibration panel, the exposure column is renamed "Exposures" and contains all different exposures of the frames in each post-calibration group:​

• all steps can be enabled/disabled using the title checkbox (the previous "Apply" button has been removed):​

• Cosmetic Correction is now applicable by group:​

• FITS orientation can now follow the global preferences or can be overridden if a particular set of processed images have a different orientation convention:​

Global Presets
This version of WBPP introduces the concept of "presets". A preset is simply a WBPP configuration that corresponds to a certain configuration that can be manually set. The idea of having presets is to offer a convenient method to configure WBPP balancing execution time versus output quality. In this version, the presets are the following:

• Maximum quality: Local Normalization is enabled with the maximum number of stars used for scale evaluation and the PSF model is "auto"​
• Faster, good quality: Local Normalization is enabled with a reduced number of stars used for scale evaluation and the PSF model is Moffat 4​
• Fastest, lower quality: Local Normalization is disabled and the Output Pedestal is set to Literal on all groups​

New Automated Mode
WBBP can now be executed in a fully automated mode from the command line. The syntax is the following:

/opt/PixInsight/bin/PixInsight.sh --automation-mode -r="/home/robyx/Workspace/PixInsight/PJSR/src/scripts/WeightedBatchPreprocessing/WeightedBatchPreprocessing.js,automationMode=true,<key>=<value>,<key>=<value>,..." --force-exit

assuming a Linux operating system. The --automation-mode parameter is necessary to enable a special working mode optimized for non-interactive or unattended execution while --force-exit terminates PixInsight at the end of the execution. The -r parameter is a string that contains the following:

1. the full path of WBPP at the beginning
2. automationMode=true option which is mandatory to run WBPP in automation mode
3. any other comma-separated key=value sequence to set WBPP parameters. In this initial implementation, it's only possible to configure parameters that are not specific to any group since groups are dynamically generated depending on the files provided

The most useful keys are the following:

1. to add a single file use the file key and the full path as the value:
   file=/home/user/Workspace/masters/masterDark_BIN-1_EXPOSURE-1.75s.xisf
2. to add a directory and include all files in the subtree use dir as key and the directory path as the value:
   dir=/home/user/Workspace/M45/data/lights/
3. to specify the output directory use outputDirectory as key and a path as the value:
   outputDirectory=/home/user/Workspace/M45/wbpp/
   

Diagnostic Screenshots generation
Quite often, issues due to WBPP misconfigurations can be easily spotted by inspecting the panel's configuration. To facilitate the request for support, it's now possible to grab screenshots of all WBPP panels automatically. Users are encouraged to grab these screenshots and attach meaningful ones to the post when asking for support on WBPP (most probably, Lights, Calibration and Post-Calibration panels will be enough).

The screenshots can be generated by opening the Diagnostic panel and then clicking the correspondent button:

The screenshots will be saved under the logs subfolder of the output directory:

Other improvements

• the initial integration warning message is not shown any more since WBPP now represents a reliable tool to generate production master integration files
• Local Normalization, Registration and Image Integration are now enabled by default when WBPP is reset​
• the console log now includes ImageCalibration, CosmeticCorrection, Debayer, StarAlignment and ImageIntegration JavaScript process source code to
  report the complete overview of each processes configuration​
• diagnostic messages have been extended to include the proper checks related to the new features​
• the final Smart Report now shows Local Normalization information and the number of successfully integrated frames for each master
• Keywords mode loop cycle has changed to "none" -> "PRE" -> "PREPOST" -> "POST"

Bugfix
The issue of the editor remaining in an inconsistent state when the focus on the edit field was lost has been fixed.

Enjoy!

​

 

[rbotero]

Fantastic new features Roberto! So many improvements now fully accessible within the same script. I am a particular fan of the ability to apply different Cosmetic Corrections settings to different sets of frames. Thank you!

Roberto

 

[robyx]

Fantastic new features Roberto! So many improvements now fully accessible within the same script. I am a particular fan of the ability to apply different Cosmetic Corrections settings to different sets of frames. Thank you!

Roberto

yes, considering the flexibility we reached and the possibilities in terms of mixing different kinds of data it was definitely worth it making Cosmetic Correction a per-group feature!

Glad that you like it!

 

[Grim408]

Amazing improvements. Nice to se WBPP is still growing. One question, does image integration include local normalization files and drizzle files or this has to be added manually in the image integration process after WBPP is done?

 

[robyx]

Amazing improvements. Nice to se WBPP is still growing. One question, does image integration include local normalization files and drizzle files or this has to be added manually in the image integration process after WBPP is done?

Hi @Grim408,

WBPP executes Image Integration properly by handling both local normalization and drizzle files. This is one of the biggest (historical I would say) change introduced with this new release since the new Local Normalization process along with the new image quality estimators allows us to select the best reference frame and perform local normalization automatically without supervision.

So, yes, your master light will be the result of the integration of local normalized images and drizzle files will be updated too, preparing the drizzle integration as a next step at the end of the execution.

Robyx

 

[Grim408]

Hi @Grim408,

WBPP executes Image Integration properly by handling both local normalization and drizzle files. This is one of the biggest (historical I would say) change introduced with this new release since the new Local Normalization process along with the new image quality estimators allows us to select the best reference frame and perform local normalization automatically without supervision.

So, yes, your master light will be the result of the integration of local normalized images and drizzle files will be updated too, preparing the drizzle integration as a next step at the end of the execution.

Robyx

Woow, ok that's some really great news. I'm shocked. This is something I've been waiting for a long time and adding drizzle process as the final execution is even more amazing. Finally I can sleep all night and upon awaking the normalized-drizzled-master will be waiting. Uh. Thank you for that!

 

[robyx]

Woow, ok that's some really great news. I'm shocked. This is something I've been waiting for a long time and adding drizzle process as the final execution is even more amazing. Finally I can sleep all night and upon awaking the normalized-drizzled-master will be waiting. Uh. Thank you for that!

well drizzle is not yet part of the WBPP steps it's "prepared" in the sense that drizzle files are updated and ready to be used by a manual drizzle integration WBPP is done

 

[Grim408]

well drizzle is not yet part of the WBPP steps it's "prepared" in the sense that drizzle files are updated and ready to be used by a manual drizzle integration WBPP is done

Yes Yes, I got it, I'm just exited it is coming. You have shorten the manual process by two factors already so KUDOS to you and your team and thank you.

 

[chris.bailey]

Is integration of SFS part of the plan? Approval criteria could be part of WBPP.

 

[freed]

hello, can you explain how the threshold level works for Auto Pedestal.
Thank you

 

[Minusman]

Hello Robyx, the linear pattern subtraction still seems to work. I select it though and set Rows. But when WBPP 2.4.0 runs, it won't be applied. I reported it about a month ago. With version 2.3.*. And how can I disable local distortion in WBPP 2.4? In the previous versions I could do it in the Script Editor, but this is no longer possible due to script signing.

 

[bulrichl]

And how can I disable local distortion in WBPP 2.4? In the previous versions I could do it in the Script Editor, but this is no longer possible due to script signing.

In the 'Lights' tab, click on 'Registration parameters' and disable 'Distortion correction'.

Bernd

 

[robyx]

Hello Robyx, the linear pattern subtraction still seems to work. I select it though and set Rows. But when WBPP 2.4.0 runs, it won't be applied. I reported it about a month ago. With version 2.3.*.

Hi @Minusman,
yes this bug is confirmed and the fix on the way, it will be released with the incoming WBPP 2.4.1.

Thank you!
Robyx

 

[robyx]

hello, can you explain how the threshold level works for Auto Pedestal.
Thank you

Hi @freed,
quoting the announcement post:

When a prescribed minimum fraction of negative or insignificant pixels are detected after calibration, the process gathers the whole set of negative pixels and computes a robust and significant mean value, and the absolute value of this mean is the automatic pedestal.

I hope it's clear but let me know if you need further clarifications!
Robyx

 

[johnpane]

@robyx, in this other thread I requested a change to logging that enables users to monitor execution while it is ongoing.

 

[robyx]

Is integration of SFS part of the plan? Approval criteria could be part of WBPP.

Hi @chris.bailey, let's say that technically the codebase is ready to implement it easily, but would you really discard frames blindly? I think you should give a try to the new weighting algorithms which are able to address quite robustly the problem of bad frames assigning very low weights (close to zero) when they are "really bad".

I personally think that in the average case where bad frames are the minority, there is no more a real need of throwing away frames since the combination of weighting algorithms and rejection is essentially already handling bad frames for you. Anyway, I guess this discussion will be refreshed in the future

There is also an internal discussion about implementing an optimization to automatically discard frames with very low weights, but it's more for performance reasons than for quality reasons.

 

[freed]

re Pedestal, I was asking as i used it last night and the automatic pedestal value was insufficient (many clipped pixels post calibration on narrowband datas o I was trying to understand how to modify the threshold (I assume by reducing it ?)

 

[robyx]

re Pedestal, I was asking as i used it last night and the automatic pedestal value was insufficient (many clipped pixels post calibration on narrowband datas o I was trying to understand how to modify the threshold (I assume by reducing it ?)

reducing the threshold percentage won't help since it is a parameter that just "triggers" the addition of the pedestal, while the value is computed as a robust mean of the negative pixel values.
This is a new feature that may require refinements and fine-tuning in some cases so if you're willing to share an uncalibrated light frame along with the calibration masters it would be appreciated!

 

[jlake]

I am loving a lot of these additions! The more robust this tool, the better, and I'm so happy it's finally recognized as a valid way to generate masters. I often return to a manual Image Integration but I rarely get a significantly better result.

I am running into a script-stopping issue though, and I'm not totally sure why. Error message:

It's not a signficantly clipped image, and while it does look a bit on the dark side, it's not that different from the frames on either side.

It's frustrating to have the script just stop and close part way through. I would appreciate a 'continue on errors' option although that can be playing with fire.

 

[chris.bailey]

Hi @chris.bailey, let's say that technically the codebase is ready to implement it easily, but would you really discard frames blindly? I think you should give a try to the new weighting algorithms which are able to address quite robustly the problem of bad frames assigning very low weights (close to zero) when they are "really bad".

I personally think that in the average case where bad frames are the minority, there is no more a real need of throwing away frames since the combination of weighting algorithms and rejection is essentially already handling bad frames for you. Anyway, I guess this discussion will be refreshed in the future

There is also an internal discussion about implementing an optimization to automatically discard frames with very low weights, but it's more for performance reasons than for quality reasons.

I think when you have run with a setup for a while you get a good ‘feel’ for what approval of frames is likely to look like. In my, limited, testing so far PSF based weighting is pretty robust BUT with large sets of large frame OSC files there is a processing overhead. I tend to approve frames in the 2 sigma beneficial band based on PSF weight, FWHM and eccentricity. This typically culls about 5-10%. That saves a little in integration time.