WBPP 2.0 released

robyx

PixInsight Staff
Jul 17, 2019
262
77
44
Zurich
Hi all,

We are glad to announce the release of WBPP 2.0. This is a major update that significantly enhances the control over the calibration process and removes some limitations present in the previous WBPP 1.5.3.

Main Improvements

In this release we introduced a brand new Control Panel view that gives a comprehensive and interactive overview of the entire WBPP session. The purpose of this panel is to have both a global dashboard and calibration control center in a single place, providing a summary of the groups in the session and their configuration, and giving the possibility to manually override any specific configuration or calibration master file assigned to each group. The second main feature introduced is called Grouping Keywords, which enables the possibility to group frames by arbitrary criteria, thus addressing the increasing demand of managing complex cases like multi-night session and separate pre-post meridian frames, to name a few.

The new Control Panel

A new tab Control Panel is now available. This panel provides an overview of the current status of the whole WBPP session with regards the calibration settings:

WBPP_00_Overview.png

This overview allows to:
  • See the full list of bias/dark/flat/light groups.
  • Differentiate between groups that contain a master file and groups that contain frames to be calibrated and integrated.
  • See the binning, the reference exposure and the filter name for each group accordingly to its type.
  • See if proper bias/dark/flats are matched for each group that needs to be calibrated.
  • See the calibration settings for each group ("Contains Bias" option for darks, "Optimize Master Dark" and "Separate CFA Scaling factors" for flat/lights).
  • See groups that contain monochrome or mosaiced images and, in the latter case, see which debayer pattern will be used to debayer the images after their calibration.
  • See the status of the calibration settings for each group. The status highlights the result of the sanity check that is performed automatically in order to identify potential issues or blocking errors due to a wrong configuration.
  • See the list of grouping keywords for each group (more on that below).
  • See the warnings and/or errors detected on each group's settings.
  • Manually change settings and override master files for each group.
  • Inspect the Calibration Diagram that depicts the calibration process performed by WBPP on each group with the current configuration.
This overview is interactive, you can select any group to:
  • Highlight in green which master bias/dark/flat will be used to calibrate the group.
  • Highlight in black bold which groups will be calibrated using the master file corresponding to the selected group.
  • Enable the editing of the calibration settings.
  • Enable the "show Calibration Diagram" button to see the graphical representation of the calibration process that WBPP will perform on the selected group.
Dark Frames Settings

WBPP_01_DarkSettings.png

By selecting a dark group you can:
  • Set/unset the "Contains Bias checkbox".
Flat Frames Settings

WBPP_02_FlatSettings.png

By selecting a flat group you can :
  • Override the Master Dark that will be used to calibrate flat frames (the drop down shows only the compatible Master Darks available).
  • Enable/disable the Master Dark optimization (the Optimize Dark column will change accordingly).
  • Mark the flats as containing CFA or monochrome images (the CFA Images column will change accordingly).
  • Enable/disable the "Separate CFA Flat Scaling factors" option (if group contains CFA images).
  • Apply the same settings to all flat frames.
  • Read the warnings and/or errors detected.
  • Show the calibration diagram for the selected flat frames group.
Light Frames Settings

WBPP_03_LightSettings.png


By selecting a light group you can:
  • Override the master dark and the master flat that will be used for frames calibration.
  • Set/unset the master dark optimization (the Optimize Dark column will change accordingly).
  • Mark the flats as containing CFA or monochrome images (the CFA Images column will change accordingly).
  • Set the Bayer pattern and debayer method to be used during Debayering, only if group contains CFA Images (the CFA Images column will change accordingly).
  • Apply the same settings to all light frames.
  • Read the warnings or errors detected.
  • Show the calibration diagram for the selected light frames group.

Calibration Diagram

WBPP_04_CalibrationDiagram.png

Any time a group that needs to be calibrated is selected, the "Show Calibration Diagram" button gets enabled and by pressing it the Calibration Diagram dialog will be presented. It depicts the calibration process that will be preformed by WBPP and it could be a good reference to double check if the calibration settings are properly configured.


Grouping Keywords

[Representation in Control Panel]
WBPP_05_GroupingKeywords.png

[Representation in frame Panels]
WBPP_06_GroupingKeywordsBIAS.pngWBPP_07_GroupingKeywordsDARKS.pngWBPP_08_GroupingKeywordsFLAT.pngWBPP_09_GroupingKeywordsLIGHT.png

The new Grouping Keywords feature provides the flexibility to group frames by custom matching criteria. A keyword is a name that can optionally be found into the FITS header or stored either in the file name or in the file path (like in the containing folder's name or the parent's parent folder name, etc). Any keyword has a value that is read from the FITS header (if the keyword is found) or from the file name or folder path if it has been added using the following syntax: "<keyword>_<value>". Few examples for the keyword SESSION are:

Keyword in the containing folderKeyword in the file namesKeyword in the file path
M42/SESSION_01/light_600sec_01.fits
M42/SESSION_01/light_600sec_02.fits
M42/SESSION_01/light_600sec_03.fits
M42/SESSION_02/light_600sec_01.fits
M42/SESSION_02/light_600sec_02.fits
M42/SESSION_02/light_600sec_03.fits
M42/SESSION_02/light_600sec_04.fits
M42/SESSION_03/light_600sec_01.fits
M42/light_600sec_01_SESSION_01.fits
M42/light_600sec_02_SESSION_01.fits
M42/light_600sec_03_SESSION_01.fits
M42/light_600sec_01_SESSION_02.fits
M42/light_600sec_02_SESSION_02.fits
M42/light_600sec_03_SESSION_02.fits
M42/light_600sec_04_SESSION_02.fits
M42/light_600sec_04_SESSION_03.fits
M42/SESSION_01/lights/light_Ha_600sec_01.fits
M42/SESSION_01/lights/light_Ha_600sec_02.fits
M42/SESSION_01/lights/light_Ha_600sec_03.fits
M42/SESSION_02/lights/light_Ha_600sec_04.fits
M42/SESSION_02/lights/light_Ha_600sec_05.fits
M42/SESSION_02/lights/light_Ha_600sec_06.fits
M42/SESSION_02/lights/light_Ha_600sec_07.fits
M42/SESSION_03/lights/light_Ha_600sec_08.fits

Even if we assume that light frames have the same binning, same filter name and same duration, by using the keyword SESSION three separated light frame groups will be created: SESSION 01, SESSION 02 and SESSION 03. Each group will be calibrated, debayered (eventually), registered and integrated separately to generate three master light frames.

To generically understand flexibility provided by this grouping strategy, here are the rules that determines the matching logic:
  1. Grouping keywords works on top of the standard matching criteria which are:
    • Single frames
      • Same binning between bias frames and
      • Same binning and duration within the provided tolerance for dark frames.
      • Same binning and filter name for flat frames.
      • Same binning, duration within the provided tolerance and filter name for light frames.
    • Master frames
      • Master bias must have the same binning.
      • Master dark must have the same binning and the closest duration.
      • Master flat must have the same binning and filter name.
  2. Missing keywords are ignored in the comparison.
  3. Only the values for the keywords that are found in both the frames or the groups to be matched are compared, if there is a mismatch (even a single one) then the match fails.
  4. If more than one frame or group matches (i.e. all the keywords found on both have the same values), the precedence is given to the frame or group that has the highest number of matches.
These four rules allows to address a large amount of possibilities like handling multi-session, multi-camera, pre-post meridian flip sessions to mention few.

Keywords are case insensitive while values are case sensitive. Keywords can be added/updated/removed at any time by means of the provided control above the global options, as soon as the keywords list will get updated the groups will be regrouped accordingly. A good and simple approach to learn how to take advantage from grouping keywords is to practice by adding/removing keywords and see how groups get reorganized.

Once the generation of the master frame is completed, the corresponding grouping keywords will be saved into both its FITS header and its file name.


Master Frame automatic detection

Master frames are now automatically recognized by checking the presence of the word "master" in either the "IMAGETYP" FITS keyword's value or in the file path (case insensitive). This removes the limitation of the previous WBPP 1.5.3 with respect the possibility to mix master files and single frames in a session.

Note: Dark Tolerance parameter is applied only to dark frames, thus when a Master Dark file is added the dark tolerance value is ignored and the master dark frame is added to an existing group only if the duration exactly matches.


Global checkboxes removed

Several global checkboxes are removed from the "Global Options" right panel. In particular:
  • Flat panel's "calibrate with dark flats" checkbox has been removed, this matching logic is no more required since now we can check the matching Master Dark to calibrate the flat frames and manually override it if needed.
  • Light panel's "calibrate only" has been removed, to calibrate only you can simply disable image registration and integration.
  • Global option "use master bias / dark / flat" checkboxes removed and replaced by the automatic detection of master frames feature.
  • Global option "master dark contains bias" checkbox removed and replaced by the "per-group" settings in the Calibration Panel.
  • Global option "CFA images" checkbox removed and replaced by the "per-group" settings in the Calibration Panel.

Global Defaults updated

Some default values have been updated:
  • Globals
    • "Save frame groups on exit" is enabled by default.
    • Generate Subframe Weights is disabled by default.
    • Image Registration is disabled by default.
    • Image Integration is disabled by default.
  • Image Registration
    • "Use triangle similarity" in the registration panel is disabled by default, (pentagons is the default value).
    • "Maximum stars" in the registration panel is set to "Auto" by default.

New Diagnostics

Diagnostic messages have been redesigned to provide better readability. Messages are now grouped by the following topics:
  • XISF writer check (library sanity check) errors.
  • Output directory error checks (existence, permission to write).
  • Missing of bias/dark/flat/light frames notes.
  • For each group:
    • Less than 3 frames to be integrated warning.
    • Frame file not existing error.
    • Usage of bias-subtracted master dark warning.
    • Filter name containing invalid characters to be replaced in master file name warning.
    • No master calibration files found warning.
    • Missing master bias and master dark warning.
    • Master bias found but master dark missing warning.
    • Missing of master flat warning.
    • Wrong configurations that leads to a non bias-subtracted calibrated frames warning.
    • Optimization of a master dark without providing a master bias warning.
    • Master dark optimization with significant exposure difference warning.
    • Unoptimized master dark with significant exposure difference warning.
    • Inappropriate rejection method warning.
  • Various Overscan warning/errors.
  • Various Cosmetic Correction warnings/errors.
  • Reference image settings checks.

Process Logger Updated

Process logger has been updated to provide the tracing information that reflects the new features.
The whole process logger's final report can now be exported as a text file.

Dedicated Script Icon

A dedicated icon has been designed and assigned to the new WBPP script. This icon is the original work of our new icon designer Ana Carracedo.


—————————————

Thank you for your attention.
 
Last edited:

sbenz

Member
Jul 8, 2017
16
0
Wow, thanks for providing this update!

One quick question: I struggle with assigning a single flat frame as a master flat. Do I need to rename that file "masterflat" in order to make this work, or is there a switch than I can tell the script that the single file provided is already a master (irrespective its name)?
Thanks.

CS
Stefan
 

robyx

PixInsight Staff
Jul 17, 2019
262
77
44
Zurich
Wow, thanks for providing this update!

One quick question: I struggle with assigning a single flat frame as a master flat. Do I need to rename that file "masterflat" in order to make this work, or is there a switch than I can tell the script that the single file provided is already a master (irrespective its name)?
Thanks.

CS
Stefan
Hi Stefan, yes adding the word “master” to the file name helps to mark it as a master file.
 

rbotero

Well-known member
Jan 19, 2005
57
12
Kent, England
www.astrobin.com
Hi Robyx

This is looking great! Thank you and the team behind WBPP for all the hard work!

I have a question about Cosmetic Correction settings when using Darks. If I have different exposure times for my lights, do I need to process each set separately with its corresponding Template Icon for CC or can map more than one icon to match the exposure length?

Roberto
 

robyx

PixInsight Staff
Jul 17, 2019
262
77
44
Zurich
Th
Hi Robyx

This is looking great! Thank you and the team behind WBPP for all the hard work!

I have a question about Cosmetic Correction settings when using Darks. If I have different exposure times for my lights, do I need to process each set separately with its corresponding Template Icon for CC or can map more than one icon to match the exposure length?

Roberto
Thanks for your kind words!

At the moment, CC can refer to a single Template Icon to be applied to all light frames to be calibrated. You can use the "auto" hot-cold pixel thresholding which should be fine for all frames. If, differently, you have other methods, I they could not be suitable for all group of light frames and if this would be the case you still need to run separate instances of WBPP in order to reference different CC Template Icons.

Robyx
 
  • Like
Reactions: rbotero

IamTomzo

Member
Dec 6, 2018
18
0
This looks awesome! One quick question - I have a ZWO ASI1600 and advice seen across the web says not to use bias frames for flats calibration for this type of CMOS camera. How would a calibration proceed in this case? I have generally created a master flat using dark flats rather than bias frames and wonder if that process might change with this new, and excellent, tool?

Thanks

Tom
 

robyx

PixInsight Staff
Jul 17, 2019
262
77
44
Zurich
[edited]
If you have dark flats to calibrate your flats then I don't see any issue, a master Bias should not be used for either flats nor lights if you have darks with proper duration and temperature.

Is this your case?
 
Last edited:
  • Like
Reactions: IamTomzo

Linwood

Well-known member
Jul 28, 2020
148
16
This begs a question I have been bothered by for a while: How does it know?

Specifically - I have master darks built by hand. When I load them into WBPP (old or new) it could find their exposure time and groups fine, but when I look at the FITS header I do not see it in there anywhere. I have them in the name, but only as Master_Dark_0300.00s_0C_Gain0_Offset50_binx1x1.xisf. Nice that it works, but always puzzled me as I don't see an exposure in the fits headers anywhere, nor the number 300. I doubt (especially the old one) it is parsing the name. I see miles of history in the fits header from integration but no exposure that I can find, and exposure appears removed as part of the integration.

So it can find exposure, but where?

The reason this is of interest is it would be nice if I could now have it separate by sensor temp and gain. Gain works nicely (I put in EGAIN and it found them all and separated). But in the file name I have target sensor temperature only as "_xxC_" no keywords.

Should I mass-update by folder and include sensor temp (target) in the FITS header somewhere?

Or mass update and try to change the file name structure so there's a _TEMP_xxC_ in it?

Or.... my real question... like exposure, is it stored somewhere I cannot see, but should be accessible? That's why I ask about exposure, was really hoping for temp?

Finally -- any chance of having "add files" parse subfolders? I have things broken down (for the human in me) by temp, gain, etc. But with this feature it works best if I just suck in ALL my master darks at once. I could drag and drop but that doesn't work, you need the add dialog?
 

Linwood

Well-known member
Jul 28, 2020
148
16
Sorry... new question (or related)... EGAIN didn't work well. My master darks have 0.78. Same camera and gain my individual flats are 0.779999971389771. I suspect those are not going to match? Unless that's a change in the driver I assume PI rounded when it did the integration to master darks?

Is there a provision in the matching for numeric parameters being close enough, like exposure? Or does that need file name restructuring also (I use 0x or 100x now, but could change to Gain_0 and Gain_100 of course).
 

robyx

PixInsight Staff
Jul 17, 2019
262
77
44
Zurich
This begs a question I have been bothered by for a while: How does it know?

Specifically - I have master darks built by hand. When I load them into WBPP (old or new) it could find their exposure time and groups fine, but when I look at the FITS header I do not see it in there anywhere. I have them in the name, but only as Master_Dark_0300.00s_0C_Gain0_Offset50_binx1x1.xisf. Nice that it works, but always puzzled me as I don't see an exposure in the fits headers anywhere, nor the number 300. I doubt (especially the old one) it is parsing the name. I see miles of history in the fits header from integration but no exposure that I can find, and exposure appears removed as part of the integration.

So it can find exposure, but where?
Indeed it's actually read from the file name. The so called "Smart Naming" capability allows you to add informations to the file name or in any point of the entire file path (so parent folders). in your case, the exposures searches for a sequence of numbers (integer or floating point) terminated with an "s", if you look carefully you can find it:

Master_Dark_0300.00s_0C_Gain0_Offset50_binx1x1.xisf

The reason this is of interest is it would be nice if I could now have it separate by sensor temp and gain. Gain works nicely (I put in EGAIN and it found them all and separated). But in the file name I have target sensor temperature only as "_xxC_" no keywords.

Should I mass-update by folder and include sensor temp (target) in the FITS header somewhere?

Or mass update and try to change the file name structure so there's a _TEMP_xxC_ in it?
I suggest the latter: use the grouping keyword syntax <name>_<value> so, like you mentioned, TEMP_10C for example and add TEMP as a grouping keyword to get all frames grouped by similar temperature values. Keep in mind that values can only be alphanumeric so don't use dots to store temperature as a floating point value (the integer part would be taken anyway).

You have two options: add it to each file or group files with same temperature in a subfolder and add the TEMP_10C word in the folder name. The latter will work as well.

Finally -- any chance of having "add files" parse subfolders? I have things broken down (for the human in me) by temp, gain, etc. But with this feature it works best if I just suck in ALL my master darks at once. I could drag and drop but that doesn't work, you need the add dialog?
If I understand correctly you're referring to the recursive addition of files starting from a subfolder. If this is the case, this is not implemented.
 

Linwood

Well-known member
Jul 28, 2020
148
16
@robyx, thanks. That's fascinating, I had no idea it was pulling exposure from the pattern in the name. One mystery solved. And I can do the same for temp.

Recursive adding of subfolders: Thanks. Please consider it maybe for the future. Or allow drag and drop into the file selections (it's easy to display a list of all files recursively, and I could drag them into a process file list, but apparently not into a script file list).
 
  • Like
Reactions: robyx

baron555

New member
Nov 18, 2015
1
0
Newbie here, is there something that I need to download to obtain the new WBPP script? Or does it automatically update in Pix?
 

Linwood

Well-known member
Jul 28, 2020
148
16
Resources, Updates, Check for updates. Follow from there, a restart of PI is needed to install, then it's all there.
 

IamTomzo

Member
Dec 6, 2018
18
0
[edited]
If you have dark flats to calibrate your flats then I don't see any issue, a master Bias should not be used for either flats nor lights if you have darks with proper duration and temperature.

Is this your case?
I generally create my master flats by calibrating my flats with dark flats and then integrating them into masters. This practice comes from my reading of several PI processing workflows from various folks around the web. Is this the correct method?
 

robyx

PixInsight Staff
Jul 17, 2019
262
77
44
Zurich
I generally create my master flats by calibrating my flats with dark flats and then integrating them into masters. This practice comes from my reading of several PI processing workflows from various folks around the web. Is this the correct method?
yep there is nothing wrong in doing this, with WBPP you can add the dark flats and flat frames and it will do the job. I guess light frames will have their proper darks too and no bias is involved. With this configuration I don't see any issue, just ensure that exposure matches between flats and their master flat dark as well as for light and their master dark.
 
  • Like
Reactions: IamTomzo

IamTomzo

Member
Dec 6, 2018
18
0
yep there is nothing wrong in doing this, with WBPP you can add the dark flats and flat frames and it will do the job. I guess light frames will have their proper darks too and no bias is involved. With this configuration I don't see any issue, just ensure that exposure matches between flats and their master flat dark as well as for light and their master dark.
Thanks - I generally don't take flats every night - unless I rotate the camera or change something else. Otherwise, I do it every few weeks just to account for random dust motes. I also don't have flats automated yet, but once I do I may just do them for each session.
 

robyx

PixInsight Staff
Jul 17, 2019
262
77
44
Zurich
You can add your master flats created in a previous session without issues, until you'll decide to integrate new flat frames.
 

interplanetary

New member
Oct 27, 2019
2
0
The Control Panel and Grouping Keywords are a great improvement! Finally I can co-process Lights with different GAIN and OFFSET, with their appropriate calibration frames.

Unfortunately, it doesn't look like I can group by Keyword CCD-TEMP, as the script does not permit "-" (hyphen) in the keyword.
 

robyx

PixInsight Staff
Jul 17, 2019
262
77
44
Zurich
Correct, keyword and value can be alphanumeric only in this first implementation. The reason is to provide a simple strategy that is mainly intended to be used in file names and folders, FITS header is optional and is not meant for the moment to be fully compatible with the whole standard keywords, considering that extending the possible character set for values (so integer and float numbers, dates etc) has an impact that should be evaluated accurately.

As a drawback, you can put files with different temperature in subfolders adding TEMP_10 (for example) to the containing folder.

Not being compatible with all FITS keywords is not a blocker, despite that enabling dashes in keyword names could be a nice to have feature that we'll investigate for a next release.
 

aatdalt

Member
Feb 24, 2020
8
0
There's two features I've asked about in the past and I'm curious if they were added or planned to be added:

Custom weighting options? Would be amazing to be able to load our own weighing formula. Like using Stars instead of or in addition to median.

Saving the weighing keyword into the fits header without going past the registration step? Iirc, WBPP would run its weighting but not actually save anything into the files at that stage, which was frustrating for those of us who like to look at SFS manually.

Updated UI looks nice! Good sanity checks before pressing go.