WBPP v2.1 Released

robyx

Administrator
Staff member
Hi all,

We are glad to announce a new version of PixInsight's Weighted Batch PreProcessing script 2.1. This new version is distributed as part of the PixInsight 1.8.8-8 Release and requires it as the minimum version of PixInsight's core to run.

The new WBPP v2.1 represents the completion of the intense reworking of the script's engine and interface started with the new version 2.0 and the introduction of the Control Panel and the Grouping Keywords features.

What's new: Main Features in Summary.
  • WBPP now generalizes the grouping strategy allowing to decouple the calibration groups in pre-processing phase from the groups that generate the master light frames in the post-processing phase, extending the roles of the grouping keywords, which now can be assigned to pre-processing, post-processing or both phases, thus allowing a complete custom logic to generate any desired master light frame groups.
  • To consistently support multi-panel sessions, a new reference frame mode "per post-processing keyword" is optionally selectable when a keyword has the special role of separating panels in the post-process phase; this is mandatory to properly align light frames within the same panel of the mosaic on a separated reference frame.
  • A new step that integrates the Linear Defect Detection and Linear Pattern Subtraction methods by Vicent Peris has been added.
  • Image Integration Output Pedestal parameter has been added (a contribution from Chris Bledsoe)
  • rangeClipHigh is now disabled in ImageIntegration: solving the questioned black pixels issue in the drizzled images
  • A brand new "continue on error" strategy is now in place such that all successfully processed frames will continue to be processed while frames that failed will be just skipped.
  • The number of stars detected on an image is now used to select the best reference frame (the higher number of stars, the better), and it's also included as a weight in the light frame's weighting formula.
  • The Subframe Selector process is now used for measuring the light frames, boosting the measurement phase that represented the main bottleneck in the previous versions.

Several other improvements are part of this release and will be detailed in the following sections.

PRE processing / POST processing grouping
WBPP now separates the two stages of pre-processing and post-processing. Pre-processing is the calibration phase that generates the master calibration files along with the calibrated light frames. In that phase:
  • calibration master files are generated first (master bias, master dark and master flat)
  • light frames are calibrated using the matching calibration master files
After calibration, all calibrated light frames are regrouped into the post-processing groups, and any further processing is performed on that new set of groups. This decouples the grouping criteria for calibration purposes from the grouping criteria used to generate the master light frames. LDD+LPS, Cosmetic Correction, Measurements / Weighting, Registration and Image Integration are now performed per post-process groups.

The post-processing minimum criteria always separate frames with different binning, exposure time (according to the exposure tolerance), and a filter name.

You can switch between the pre-processing and post-processing groups using the dedicated selector:

PrePostProcessView.png

The pre-processing view is similar to what was already visible in the control panel. The status column has been moved to the 4th position in the pre-processing view to be almost always visible on smaller screens. The post-processing view shows a shorter set of columns, including a new total integration time per group column and the overall total integration time of the session in the column's header.

Grouping Keywords Modes
Grouping keywords can be enabled for the pre-processing or post-processing stage or both. In the following example, we group by NIGHT for calibration purposes (multi-night session), but then we group by PANEL keyword to generate separate master light frames of a mosaic.

KW_00.png

How does it work:
  • by selecting a keyword (row highlighted in orange), you can
    • modify it by typing the new value in the text edit control and clicking the save button
    • increase or decrease the priority by moving it up or down clicking the arrows
    • switch the mode by clicking the wheels button
    • delete it by clicking the thrash button
  • keywords order matter in the "rare" case where the same number (but a different set) of matching keywords matches more than one calibration master file. In that case, the matching keyword on top has precedence
  • keywords can be globally enabled/disabled using the checkbox on top in case you need to quickly switch between the groping, including/excluding keywords (mostly for verification)
New Reference Frame Mode
When one or more keywords are enabled in post-process mode, a separate reference frame could be associated with a specific keyword. This is convenient for mosaics, where light frames in each panel should be registered separately from the others. If registration is enabled, the Registration Reference Frame's mode dropdown provides one option for each post-process keyword to automatically use a separate reference frame associated with each selected keyword's value.

In the following example, a multi-night mosaic session uses the keyword PANEL to group light frames of the same panel; the Registration Reference Image's mode dropdown detects the post-processing PANEL keyword and offers the option to detect a separated reference frame for each PANEL automatically.

NewReferenceFrameMode.png

Linear Defect Detection + Linear Pattern Subtraction Step Added
Vicent Paris scripts are used to correct defects usually emerging with dated sensors, and they are now part of the processing pipeline (reference here).
If enabled, this step is the first applied after the calibration. The LDD is used to analyze and extract partial or full defective columns or rows or both from a reference image automatically generated by WBPP after the calibration; the LPS procedure then corrects the detected lines.

LDD's engine must be fed with a good reference frame to detect defects successfully; for this purpose, WBPP conventionally assumes that the post-processing group with the highest total exposure time is most willing to be the best candidate in terms of SNR. It's worth mentioning that this step must not be enabled if multiple the session contains frames from different cameras since this kind of defect is bound on the sensor and differs from camera to camera.

LDD_LPS.png

Image Integration "Output Pedestal" added
it's now possible to add a pedestal to calibrated light frames. This option avoids negative values resulting after the dark subtraction to be rounded to zero when the calibrated light frame file is saved (code contribution from Chris Bledsoe).

AddPedestal.png

New "Continue on Error" Strategy
WBPP now will not stop the execution if one file (or an entire group) fails to be processed at any step. In the reports, there is new information called "Active Frames" that reports the number of frames in a successfully processed group; the failed files are disabled and will just get discarded in the following steps.

ActiveFrames.png
The Number of Stars
The detected number of stars is a value extracted from image measurements and is used both as a weighting parameter and as a metric to determine the best reference frame. As a weighting parameter, the higher the weight, the more weight is given to the frames with a higher number of stars. As the new criteria to detect the best reference frame, WBPP now selects the frame with the highest number of stars.

NumberOfStars.png

Global Option "Include file path to detect Masters"
WBPP auto-detects master files by inspecting the fits header's IMAGETYP value, searching for the "master" word or, as a fallback, searches the word "master" in the file's full path. The latter strategy could rarely cause an unintended detection of a master file if the full path contains the word "master" not intended to classify the files type.

If a collision of this kind occurs, you can uncheck this option to exclude the full path and restrict the search of the word "master" strictly to the file name.

CheckMasterInFilePAth.png

ESD Low Relaxation Parameter Added
ESD Low Relaxation parameter has been added to provide finer control of ESD rejection in the Image Integration process.

ESD_Low_Relaxation.png

rangeClipHigh is now disabled in ImageIntegration.
This option was enabled by default in WBPP, setting the level to 0.98. This drawback generates black pixels at the centre of the saturated stars when performing drizzle integration using the drizzle files generated by WBPP.

Auto-Matching Heuristic Improved
The WBPP auto-matching behaviour has been improved to cover more cases and reduce the manual adjustments.
Master Bias is now used only if needed, so if no matching master dark is found or if it's found and optimized.
Master Dark's matching logic differs between flat and light frames:
  • flat frames: the behaviour depends on the presence of a matching master bias,
    • if a Master Bias matches, then the Master Dark exposure must strictly correspond to match; if it matches, then only the Master Dark matches and Master Bias is unmatched (no need for it); otherwise, Master Bias only matches
    • if a Master Bias does not match, then the Master Dark with the closest exposure is selected
  • light frames: the Master Dark with the closest exposure time is always selected
The matching keywords always filter the master dark to match. The general rule is the following: the auto-selected Master Dark is the one that has the closest exposure amongst the set of matching darks that have the highest number of matching keywords or no keywords at all. If more than one Master Dark with the same exposure match, the one with the higher number of matching keywords gets auto-selected.
This logic guarantees the possibility to use Master Darak from a library in a multi-session, having WBPP auto-matching the Master Dark from the library when no in-session darks are provided.

New "Add Directory" Button
A convenient button is introduced to add all image files found starting from a root folder and traversing all subdirectories to quickly bulk-load several files in the session.

AddDirectory.png

Paths and Master Files Name Convention Changed

The output paths structure has been updated to ensure the unicity of the file's full paths. A folder name is associated to each group and it's constructed adopting the following template:

<IMAGE TYPE>_BIN-<binning>_EXPSURE-<seconds>s_FILTER-<filter name>_[Mono|RGB]_<keyword>-<value>​

and the folder structure is now organized as follows, assuming group A and B be pre-process groups and groups C and D be post-process gropus :

WBPP_output: the main WBPP output folder
  • calibrated: contains the calibrated, cosmetized and debayered files​
    • <groupName A>: contains all the calibrated files for group A​
      • cosmetized: cosmetized files​
      • debayered: debayered files​

    • <groupName B>: contains all the calibrated files for group B​
      • cosmetized: cosmetized files
      • debayered: debayered files

  • ldd_lps: contains the files corrected by the LDD+LPS processes
    • <groupName C>: corrected files for group C
    • <groupName D>: corrected files for group D
  • registered: contains the registered files
    • <groupName C>: corrected files for group C
    • <groupName D>: corrected files for group D
  • master: the master files
Master file naming now adopts the following convention:

master<IMAGE TYPE>_BIN-<binning>_EXPSURE-<seconds>s_FILTER-<filter name>_[Mono|RGB]_<keyword>-<value>.xisf​

This convention is twofold: first of all, we would summarize the main properties of the master files to identify their main properties by looking at their filename quickly; secondly, this convention ensures that master file names are unique across any session configuration.
An example of maser file names is be the following:
masterFlat_BIN-1_FILTER-LPro_Mono_DATE-20210421.xisf​
masterLight_BIN-1_EXPOSURE-120.00s_FILTER-LPro_Mono_TARGET-hd172102.xisf​
masterLight_BIN-1_EXPOSURE-300.00s_FILTER-LPro_Mono_TARGET-ngc4568.xisf​
masterLight_BIN-1_EXPOSURE-300.00s_FILTER-LPro_Mono_TARGET-ngc5846.xisf​


Other Improvements

  • Image Integration scaling method is BWMV for all frame types
  • SubframeSelector process is used to measure images; this speeds up significantly the time spent on measuring frames which now is executed in parallel
  • The dash character is now allowed in both grouping keyword name and value
  • Control Panel is now activated at script launch
  • smart report updated and readability improved; in particular, the group information now contains
    • the number of active frames
    • the group's mode (pre/post processing)
    • the frames color space (CFA/RGB or Gray)
  • various bugfixes
 
Last edited:
Very good. One humble request: On the Control Panel, I can't see important things (like CFA status) without either widening the window or scrolling horizontally. The window doesn't remember its width from one session to another. Perhaps it could do so, and also the default widths of some of the columns could be smaller.
 
Another small request. Can we have the option of turning off the "Do you really want to exit the Weighted Batch Preprocessing Script?" message when exiting the script?
At least if "Save frame groups on exit" is checked, so that it can be reopened with everything there.
 
What *did* change in 2.1.1? The release note (under View Installed Updates) sounded as if it was addressing exactly that exposure match problem.
 
Great work, thanks to you all! I add a comment. Disk space requirement increases with CMOS cameras, both in terms of speed and capacity, so large SSD become necessary with consequent high purchase costs. This is even more true with multi-night sessions that have to be integrated all together.
So why don't you let WBPP delete intemediate files, upon user choice through check boxes in the user interface, DURING the processing once they are no more needed? Example to make things clearer: cosmetic corrected subs are no lenger needed once calibrated ones are produced, and calibrated can be no more necessary once registered ones are produced. So the user could optionally choose to renounce to have calibrated subs or cosmetic corrected subs at the end of the process, letting WBPP delete them as soon as they no longer needed by the process. The saving in space occupation would be very consistent and let user use smaller SSDs with an important reduction of costs (memory prices are lowering, but space requirement is increasing instead...).
Thanks for your attention and, again, for you very valuable work! Davide
 
Hello,

I upgraded today to v1.8.8 and installed all its updates to WBPP v2.1.2. Since then, I have trouble running it and can't figure out what I did wrong. Attached are the screenshots for WBPP parameters and the log file showing the errors I'm having...

This is a video I captured for what is happening to me...

And this is part of the attached log file showing the error for the Blue filter
******************** LIGHT FRAMES CALIBRATION ********************
Group of 7 Light frames (7 active)
BINNING : 1
Filter : B
Exposure : 180.00s
Keywords : []
Mode : pre-processing
Color : Gray

Master Dark automatic match.
Master Flat automatic match.

Master bias: none
Master dark: X:/Dropbox/Personal Docs/Astro/Lib/ASI533MC/Darks/masterDark-BINNING_1-EXPTIME_180.xisf
Master flat: C:/Users/hammady/Pictures/2021-05-08 Flat/masterFlat-BINNING_1-FILTER_B.xisf
Light Output Pedestal: 0
*** Warning: Image Calibration failed.
!!! Error: Light frames calibration failed.
******************************************************************




Another issue, after the upgrade, the L filter is showing "No filter". Is there is something I need to do about that?

Any help would be appreciated as I'm at this point and cannot go past WBPP

Thank you
Mohamed
 

Attachments

  • ProcessLogger.txt
    6 KB · Views: 85
  • bias.png
    bias.png
    75.6 KB · Views: 82
  • darks.png
    darks.png
    82.9 KB · Views: 78
  • flats.png
    flats.png
    95 KB · Views: 78
  • lights.png
    lights.png
    136 KB · Views: 87
  • ControlPanel.png
    ControlPanel.png
    88.7 KB · Views: 90
Hi @mhammady, your master dark looks to be 3008x3008 while the maser flat is 5496x3672 pixels.

Most probably they have been generating with different cameras.

Regarding the issue with the filter name, could you upload 1 single light frame? we'll get quickly to the reason by inspecting it.

Robyx
 
Hi @mhammady, your master dark looks to be 3008x3008 while the maser flat is 5496x3672 pixels.

Most probably they have been generating with different cameras.

Regarding the issue with the filter name, could you upload 1 single light frame? we'll get quickly to the reason by inspecting it.

Robyx

Thank you Robyx for your reply. Yes, it was my fault. I used ASI533MC as dark for ASI183MM. I should pay more attention in the future... sorry about that...

Just a suggestion, is it possible to show the frame size discrepancy in the Diagnostics function?


About the NoFilter issue, Pixinsight was correct, I discovered a bug in my capture software that some frames at the same night don't have the /Filter tag in the header. I reported this to the software support group.

Best regards,
Mohamed
 
Just wanted to say a big thank you to Roberto (and whoever else is contributing. Adam?) for this script!
The new pre/post processing keywords behavior allowed me to process a multi night and multi session 2 panel mosaic loading all the images at once!
That saves a lot of time and prevents tons of errors.

And all the possible issues have been taken in account, for example a different reference frame can be set for any post-process keywords.
Thanks!

Ferrante
 

Attachments

  • Screenshot from 2021-06-22 15-04-21.png
    Screenshot from 2021-06-22 15-04-21.png
    94.9 KB · Views: 100
Just wanted to say a big thank you to Roberto (and whoever else is contributing. Adam?) for this script!
The new pre/post processing keywords behavior allowed me to process a multi night and multi session 2 panel mosaic loading all the images at once!
That saves a lot of time and prevents tons of errors.

And all the possible issues have been taken in account, for example a different reference frame can be set for any post-process keywords.
Thanks!

Ferrante
Thank you very much @fenriques! it's great to see real cases successfully managed by the latest WBPP upgrade!
I'm glad that you enjoyed it!
robyx
 
The evolution of this script is fab! Thanks so much... a few improvements if I may suggest (maybe peculiar to my workflow?)

Ability to turn off the WBPP_output calibrated, cosmetized path as it now is, to remove the grouping level of directory. I am having to go through all the directories to move the cosmetized files in to one directory per filter so I can then use the rest of my workflow... not a problem when just doing say a nights worth of data, but if I reprocess a target from say last year, I am having to copy files from 10 different directories, 2 deep per filter or do a lot of clicking to get all the files I want to in the star alignment process :)

It would be great if I could set the pedestal once and it's saved to be used the next time, I am currently remembering to go in to the control panel and adjust it for each group, but easy to forget to do so.

My data is a mix of 1x1 and 2x2 binning, would be great to be able to associate my cosmetic correction process icons with them somehow - at the moment, I have to do 2 runs of the script to cater for the two binning icons.

Not sure I've missed it, but I couldn't see any notification of the highest weighting per filter processed (I only use the weighting, not the registration or integration options in my workflow). At present I am having to use the fitsview script to get to the weighting to see which came out top...
 
Hi @jiberjaber and @freed,

thank you very much for your feedback, indeed I received this feedback also from other users that find awkward the new folder grouping of files.
I will consider to revert it to single folders and a proper naming strategy in oder to avoid name overlapping within the same session (which was one of the reasons I implemented the file grouping by folder).

Moreover, the CC suggestion is also very welcome, I will implement a strategy to select different instances of CC depending on the binning.

Robyx
 
Hi All
I do not understand Why It´s not to use Master Darks to calibrate ! ¡ ¿ ?... I load all Darks files As it could see. (x 33)..
The Othera rare is that at the end It give me a master Dark on the master file

1625442760292.png

Could any Know that ?
Thanks !
Federico
 
Hi All
I do not understand Why It´s not to use Master Darks to calibrate ! ¡ ¿ ?... I load all Darks files As it could see. (x 33)..
The Othera rare is that at the end It give me a master Dark on the master file

View attachment 11529
Could any Know that ?
Thanks !
Federico

Hi @freddy1,
the warning regards the Flat Frames. If you look carefully at your configuration you can notice that the dark frames you've added have an exposure of 300 seconds while your flats exposure is just 0.01 sec.
WBPP detects that your dark frames are not suitable for calibrating the flat frames and so it subtracts only the bias only, which is a good assumption since your "flat dark" should have exposure if 0.01 seconds i.e. they are exactly equal to bias frames.

So, light frames are correctly dark subtracted, flat frames are correctly bias-subtracted. Your configuration looks good to me.

The warning is just telling you that despite you provided dark frames, no suitable dark flats have been found. If you know that this is correct, just proceed, a warning is simply a reminder that the configuration may be wrong but it's not blocking you from running the WBPP.

Hope this helps,
Robyx
 
Executed WBPP gives me the following warning calibrating the flats frames with the masterbias and its masterdark:

[2021-07-11 09:42:07] * Writing output file: D: / IC 1318 Sadr Region / IC 1318 SESSION 001 / WBPS / calibrated / Flat_BIN-2_FILTER-BLUE_Mono_SESSION-001 / 2021-06-24_01-24- 23_BLUE_4936pos_-9.40C_0.84seg_BIN2x2_0003_c.xisf
[2021-07-11 09:42:07] Dark scaling factors:
[2021-07-11 09:42:07] k0 = 0.000
[2021-07-11 09:42:07] ** Warning: No correlation between the master dark and target frames (channel 0).

The funny thing is that it only gives me the warning with some flats frames. Others calibrate them without any error.

On the other hand, the WBPP log does not indicate any error:

Master Dark automatic match.
Master bias: C: /CAPTURAS/MASTERS/masterBias_BIN-2.xisf
Master dark: C: /CAPTURAS/MASTERS/masterDark_BIN-2_EXPOSURE-0.84s.xisf
Master flat: none
Master Dark is optimized.
Calibration completed: 50 flat frames calibrated.
Rejection method auto-selected: Generalized Extreme Studentized Deviate
Integration completed: master flat saved at path D: / IC 1318 Sadr Region / IC 1318 SESSION 001 / WBPS / master / masterFlat_BIN-2_FILTER-BLUE_Mono_SESSION-001.xisf

What am I doing wrong?
Thank you so much
 
Back
Top