Hi all,
We are glad to announce a new version of PixInsight's Weighted Batch PreProcessing script 2.1. This new version is distributed as part of the PixInsight 1.8.8-8 Release and requires it as the minimum version of PixInsight's core to run.
The new WBPP v2.1 represents the completion of the intense reworking of the script's engine and interface started with the new version 2.0 and the introduction of the Control Panel and the Grouping Keywords features.
What's new: Main Features in Summary.
Several other improvements are part of this release and will be detailed in the following sections.
PRE processing / POST processing grouping
WBPP now separates the two stages of pre-processing and post-processing. Pre-processing is the calibration phase that generates the master calibration files along with the calibrated light frames. In that phase:
The post-processing minimum criteria always separate frames with different binning, exposure time (according to the exposure tolerance), and a filter name.
You can switch between the pre-processing and post-processing groups using the dedicated selector:
The pre-processing view is similar to what was already visible in the control panel. The status column has been moved to the 4th position in the pre-processing view to be almost always visible on smaller screens. The post-processing view shows a shorter set of columns, including a new total integration time per group column and the overall total integration time of the session in the column's header.
Grouping Keywords Modes
Grouping keywords can be enabled for the pre-processing or post-processing stage or both. In the following example, we group by NIGHT for calibration purposes (multi-night session), but then we group by PANEL keyword to generate separate master light frames of a mosaic.
How does it work:
When one or more keywords are enabled in post-process mode, a separate reference frame could be associated with a specific keyword. This is convenient for mosaics, where light frames in each panel should be registered separately from the others. If registration is enabled, the Registration Reference Frame's mode dropdown provides one option for each post-process keyword to automatically use a separate reference frame associated with each selected keyword's value.
In the following example, a multi-night mosaic session uses the keyword PANEL to group light frames of the same panel; the Registration Reference Image's mode dropdown detects the post-processing PANEL keyword and offers the option to detect a separated reference frame for each PANEL automatically.
Linear Defect Detection + Linear Pattern Subtraction Step Added
Vicent Paris scripts are used to correct defects usually emerging with dated sensors, and they are now part of the processing pipeline (reference here).
If enabled, this step is the first applied after the calibration. The LDD is used to analyze and extract partial or full defective columns or rows or both from a reference image automatically generated by WBPP after the calibration; the LPS procedure then corrects the detected lines.
LDD's engine must be fed with a good reference frame to detect defects successfully; for this purpose, WBPP conventionally assumes that the post-processing group with the highest total exposure time is most willing to be the best candidate in terms of SNR. It's worth mentioning that this step must not be enabled if multiple the session contains frames from different cameras since this kind of defect is bound on the sensor and differs from camera to camera.
Image Integration "Output Pedestal" added
it's now possible to add a pedestal to calibrated light frames. This option avoids negative values resulting after the dark subtraction to be rounded to zero when the calibrated light frame file is saved (code contribution from Chris Bledsoe).
New "Continue on Error" Strategy
WBPP now will not stop the execution if one file (or an entire group) fails to be processed at any step. In the reports, there is new information called "Active Frames" that reports the number of frames in a successfully processed group; the failed files are disabled and will just get discarded in the following steps.
The Number of Stars
The detected number of stars is a value extracted from image measurements and is used both as a weighting parameter and as a metric to determine the best reference frame. As a weighting parameter, the higher the weight, the more weight is given to the frames with a higher number of stars. As the new criteria to detect the best reference frame, WBPP now selects the frame with the highest number of stars.
Global Option "Include file path to detect Masters"
WBPP auto-detects master files by inspecting the fits header's IMAGETYP value, searching for the "master" word or, as a fallback, searches the word "master" in the file's full path. The latter strategy could rarely cause an unintended detection of a master file if the full path contains the word "master" not intended to classify the files type.
If a collision of this kind occurs, you can uncheck this option to exclude the full path and restrict the search of the word "master" strictly to the file name.
ESD Low Relaxation Parameter Added
ESD Low Relaxation parameter has been added to provide finer control of ESD rejection in the Image Integration process.
rangeClipHigh is now disabled in ImageIntegration.
This option was enabled by default in WBPP, setting the level to 0.98. This drawback generates black pixels at the centre of the saturated stars when performing drizzle integration using the drizzle files generated by WBPP.
Auto-Matching Heuristic Improved
The WBPP auto-matching behaviour has been improved to cover more cases and reduce the manual adjustments.
Master Bias is now used only if needed, so if no matching master dark is found or if it's found and optimized.
Master Dark's matching logic differs between flat and light frames:
This logic guarantees the possibility to use Master Darak from a library in a multi-session, having WBPP auto-matching the Master Dark from the library when no in-session darks are provided.
New "Add Directory" Button
A convenient button is introduced to add all image files found starting from a root folder and traversing all subdirectories to quickly bulk-load several files in the session.
Paths and Master Files Name Convention Changed
The output paths structure has been updated to ensure the unicity of the file's full paths. A folder name is associated to each group and it's constructed adopting the following template:
and the folder structure is now organized as follows, assuming group A and B be pre-process groups and groups C and D be post-process gropus :
WBPP_output: the main WBPP output folder
This convention is twofold: first of all, we would summarize the main properties of the master files to identify their main properties by looking at their filename quickly; secondly, this convention ensures that master file names are unique across any session configuration.
An example of maser file names is be the following:
Other Improvements
We are glad to announce a new version of PixInsight's Weighted Batch PreProcessing script 2.1. This new version is distributed as part of the PixInsight 1.8.8-8 Release and requires it as the minimum version of PixInsight's core to run.
The new WBPP v2.1 represents the completion of the intense reworking of the script's engine and interface started with the new version 2.0 and the introduction of the Control Panel and the Grouping Keywords features.
What's new: Main Features in Summary.
- WBPP now generalizes the grouping strategy allowing to decouple the calibration groups in pre-processing phase from the groups that generate the master light frames in the post-processing phase, extending the roles of the grouping keywords, which now can be assigned to pre-processing, post-processing or both phases, thus allowing a complete custom logic to generate any desired master light frame groups.
- To consistently support multi-panel sessions, a new reference frame mode "per post-processing keyword" is optionally selectable when a keyword has the special role of separating panels in the post-process phase; this is mandatory to properly align light frames within the same panel of the mosaic on a separated reference frame.
- A new step that integrates the Linear Defect Detection and Linear Pattern Subtraction methods by Vicent Peris has been added.
- Image Integration Output Pedestal parameter has been added (a contribution from Chris Bledsoe)
- rangeClipHigh is now disabled in ImageIntegration: solving the questioned black pixels issue in the drizzled images
- A brand new "continue on error" strategy is now in place such that all successfully processed frames will continue to be processed while frames that failed will be just skipped.
- The number of stars detected on an image is now used to select the best reference frame (the higher number of stars, the better), and it's also included as a weight in the light frame's weighting formula.
- The Subframe Selector process is now used for measuring the light frames, boosting the measurement phase that represented the main bottleneck in the previous versions.
Several other improvements are part of this release and will be detailed in the following sections.
PRE processing / POST processing grouping
WBPP now separates the two stages of pre-processing and post-processing. Pre-processing is the calibration phase that generates the master calibration files along with the calibrated light frames. In that phase:
- calibration master files are generated first (master bias, master dark and master flat)
- light frames are calibrated using the matching calibration master files
The post-processing minimum criteria always separate frames with different binning, exposure time (according to the exposure tolerance), and a filter name.
You can switch between the pre-processing and post-processing groups using the dedicated selector:
The pre-processing view is similar to what was already visible in the control panel. The status column has been moved to the 4th position in the pre-processing view to be almost always visible on smaller screens. The post-processing view shows a shorter set of columns, including a new total integration time per group column and the overall total integration time of the session in the column's header.
Grouping Keywords Modes
Grouping keywords can be enabled for the pre-processing or post-processing stage or both. In the following example, we group by NIGHT for calibration purposes (multi-night session), but then we group by PANEL keyword to generate separate master light frames of a mosaic.
How does it work:
- by selecting a keyword (row highlighted in orange), you can
- modify it by typing the new value in the text edit control and clicking the save button
- increase or decrease the priority by moving it up or down clicking the arrows
- switch the mode by clicking the wheels button
- delete it by clicking the thrash button
- keywords order matter in the "rare" case where the same number (but a different set) of matching keywords matches more than one calibration master file. In that case, the matching keyword on top has precedence
- keywords can be globally enabled/disabled using the checkbox on top in case you need to quickly switch between the groping, including/excluding keywords (mostly for verification)
When one or more keywords are enabled in post-process mode, a separate reference frame could be associated with a specific keyword. This is convenient for mosaics, where light frames in each panel should be registered separately from the others. If registration is enabled, the Registration Reference Frame's mode dropdown provides one option for each post-process keyword to automatically use a separate reference frame associated with each selected keyword's value.
In the following example, a multi-night mosaic session uses the keyword PANEL to group light frames of the same panel; the Registration Reference Image's mode dropdown detects the post-processing PANEL keyword and offers the option to detect a separated reference frame for each PANEL automatically.
Linear Defect Detection + Linear Pattern Subtraction Step Added
Vicent Paris scripts are used to correct defects usually emerging with dated sensors, and they are now part of the processing pipeline (reference here).
If enabled, this step is the first applied after the calibration. The LDD is used to analyze and extract partial or full defective columns or rows or both from a reference image automatically generated by WBPP after the calibration; the LPS procedure then corrects the detected lines.
LDD's engine must be fed with a good reference frame to detect defects successfully; for this purpose, WBPP conventionally assumes that the post-processing group with the highest total exposure time is most willing to be the best candidate in terms of SNR. It's worth mentioning that this step must not be enabled if multiple the session contains frames from different cameras since this kind of defect is bound on the sensor and differs from camera to camera.
Image Integration "Output Pedestal" added
it's now possible to add a pedestal to calibrated light frames. This option avoids negative values resulting after the dark subtraction to be rounded to zero when the calibrated light frame file is saved (code contribution from Chris Bledsoe).
New "Continue on Error" Strategy
WBPP now will not stop the execution if one file (or an entire group) fails to be processed at any step. In the reports, there is new information called "Active Frames" that reports the number of frames in a successfully processed group; the failed files are disabled and will just get discarded in the following steps.
The detected number of stars is a value extracted from image measurements and is used both as a weighting parameter and as a metric to determine the best reference frame. As a weighting parameter, the higher the weight, the more weight is given to the frames with a higher number of stars. As the new criteria to detect the best reference frame, WBPP now selects the frame with the highest number of stars.
Global Option "Include file path to detect Masters"
WBPP auto-detects master files by inspecting the fits header's IMAGETYP value, searching for the "master" word or, as a fallback, searches the word "master" in the file's full path. The latter strategy could rarely cause an unintended detection of a master file if the full path contains the word "master" not intended to classify the files type.
If a collision of this kind occurs, you can uncheck this option to exclude the full path and restrict the search of the word "master" strictly to the file name.
ESD Low Relaxation Parameter Added
ESD Low Relaxation parameter has been added to provide finer control of ESD rejection in the Image Integration process.
rangeClipHigh is now disabled in ImageIntegration.
This option was enabled by default in WBPP, setting the level to 0.98. This drawback generates black pixels at the centre of the saturated stars when performing drizzle integration using the drizzle files generated by WBPP.
Auto-Matching Heuristic Improved
The WBPP auto-matching behaviour has been improved to cover more cases and reduce the manual adjustments.
Master Bias is now used only if needed, so if no matching master dark is found or if it's found and optimized.
Master Dark's matching logic differs between flat and light frames:
- flat frames: the behaviour depends on the presence of a matching master bias,
- if a Master Bias matches, then the Master Dark exposure must strictly correspond to match; if it matches, then only the Master Dark matches and Master Bias is unmatched (no need for it); otherwise, Master Bias only matches
- if a Master Bias does not match, then the Master Dark with the closest exposure is selected
- light frames: the Master Dark with the closest exposure time is always selected
This logic guarantees the possibility to use Master Darak from a library in a multi-session, having WBPP auto-matching the Master Dark from the library when no in-session darks are provided.
New "Add Directory" Button
A convenient button is introduced to add all image files found starting from a root folder and traversing all subdirectories to quickly bulk-load several files in the session.
Paths and Master Files Name Convention Changed
The output paths structure has been updated to ensure the unicity of the file's full paths. A folder name is associated to each group and it's constructed adopting the following template:
<IMAGE TYPE>_BIN-<binning>_EXPSURE-<seconds>s_FILTER-<filter name>_[Mono|RGB]_<keyword>-<value>
and the folder structure is now organized as follows, assuming group A and B be pre-process groups and groups C and D be post-process gropus :
WBPP_output: the main WBPP output folder
- calibrated: contains the calibrated, cosmetized and debayered files
- <groupName A>: contains all the calibrated files for group A
- cosmetized: cosmetized files
- debayered: debayered files
- <groupName B>: contains all the calibrated files for group B
- cosmetized: cosmetized files
- debayered: debayered files
- ldd_lps: contains the files corrected by the LDD+LPS processes
- <groupName C>: corrected files for group C
- <groupName D>: corrected files for group D
- registered: contains the registered files
- <groupName C>: corrected files for group C
- <groupName D>: corrected files for group D
- master: the master files
master<IMAGE TYPE>_BIN-<binning>_EXPSURE-<seconds>s_FILTER-<filter name>_[Mono|RGB]_<keyword>-<value>.xisf
This convention is twofold: first of all, we would summarize the main properties of the master files to identify their main properties by looking at their filename quickly; secondly, this convention ensures that master file names are unique across any session configuration.
An example of maser file names is be the following:
masterFlat_BIN-1_FILTER-LPro_Mono_DATE-20210421.xisf
masterLight_BIN-1_EXPOSURE-120.00s_FILTER-LPro_Mono_TARGET-hd172102.xisf
masterLight_BIN-1_EXPOSURE-300.00s_FILTER-LPro_Mono_TARGET-ngc4568.xisf
masterLight_BIN-1_EXPOSURE-300.00s_FILTER-LPro_Mono_TARGET-ngc5846.xisf
Other Improvements
- Image Integration scaling method is BWMV for all frame types
- SubframeSelector process is used to measure images; this speeds up significantly the time spent on measuring frames which now is executed in parallel
- The dash character is now allowed in both grouping keyword name and value
- Control Panel is now activated at script launch
- smart report updated and readability improved; in particular, the group information now contains
- the number of active frames
- the group's mode (pre/post processing)
- the frames color space (CFA/RGB or Gray)
- various bugfixes
Last edited:
