WBPP 2.4.3 Released

[robyx]

Hi all,

I am pleased to announce the new WBPP 2.4.3 distributed with PixInsight version 1.8.9-1. This new version integrates with the new Star Detection Engine and PSF Scale SNR Estimator and includes additional new features, improvements and bug fixes. As being updated to the last features, this version requires PI 1.8.9-1 as the minimum version.

New Features

Execution Monitor

This feature has been introduced to keep track of the execution status of WBPP. While running, now WBPP shows the Execution Monitor window that reports in real-time:

• the list of the scheduled operations to be executed and the relative status
• the execution time of each completed operation
• the overall elapsed time
• A cancel button to interrupt the execution if needed. Pressing the cancel button does not interrupt the execution immediately but schedules the interruption of the current process, which could take a while before stopping in order to complete the currently ongoing operations.

PSF Scale SNR weighting mode

​

The new PSF Scale SNR weighting mode is selectable amongst the other image weighting schemas.

Estimated space occupied

​

At the bottom of the Diagnostic Message view, WBPP will report the estimated space required by each step type and the overall total amount of space required to complete the overall steps compared with the available space in the destination drive. In case there is not enough available space or the estimated remaining space is less than 5% of the current free space, a proper warning is raised.

Frame size is now used as a matching factor

WBPP now reads and takes into account the image size as a matching factor to ensure that images with different sizes are never wrongly associated.
This involves both the image grouping (images with different sizes never get grouped together) along with the master file matching (images are never calibrated with master files with a different size).

GUI Improvements: addressing small screens

​

WBPP interface now adapts to small screens (theoretically down to 800x600). When the screen resolution is not enough to contain the regular WBPP window, a new "compact GUI mode" is activated, enabling the selective collapsing of the calibration, post-calibration and global settings controls. The screenshot shows what this new interface looks like on a resolution of 1024x768 pixels.

Adaptation to the new Stars Detector

Default parameter values and interface controls have been adapted to the new Star Detector V2 engine, including parameter ranges and tooltips.
New StarAlignment parameters have been exposed under the Image Registration controls:

• Local distortion checkbox
• Hot pixel removal (filter radius)
• Bright threshold
• Allow clustered sources

Local Normalization

Reference frame evaluation criteria

The criteria used to select the local normalization reference frame can now be selected amongst the following:

• PSF Signal Weight: the higher better
• PSF SNR: the higher, the better
• M*: the lower, the better
• Median: the lower, the better
• Number of Stars: the higher, the better​

Robust generation of the Local Normalization reference frame
When the reference frame is generated by integrating the set of the best frames, Local Normalization is applied to such frames using the very best one as a reference. This ensures that any gradient potentially present in one of such frames will not contaminate the final integrated reference frame.

High and Low clipping level

Low and high clipping level parameters have been added.

Image Integration Minimum Weight

​

A new threshold Minimum weight has been added to discard frames with a weight lower than this threshold.

Global Settings Updates

​

The previous option "Save logs" has been removed since logs are now always generated.
A new "Smart naming override" checkbox has been added; when enabled, the metadata found in the full pathname, accordingly with the smart naming syntax convention, will override the corresponding data stored in the FITS header.

Automated Mode: Grouping Keywords handling
This new version allows specifying the grouping keywords to be used when WBPP is launched in automation mode.
The inline syntax is shown in the following example:

keywords=EXPOSURE;PANEL post;FILTER prepost;CAMERA pre;​

The chunk keywords= denotes the beginning of a keywords list. Keywords must be separated by a semicolon, and each keyword must be defined by the name optionally followed by its mode separated by a space. If the mode is missing, the value assumed is pre. If defined, the keyword mode must be one of pre, post or prepost.

In the above example, 4 keywords are defined: EXPOSURE, PANEL, FILTER and CAMERA. EXPOSURE is pre by default, PANEL is in post-calibration mode, FILTER is in both pre and post-calibration mode, and CAMERA is in pre mode.

An example of a full automated WBPP execution command line is the following (valid on Linux):

/opt/PixInsight/bin/PixInsight.sh -n --automation-mode -r="/opt/PixInsight/src/scripts/WeightedBatchPreprocessi
ng/WeightedBatchPreprocessing.js,automationMode=true,outputDirectory=/home/users/me/M31/wbpp,dir=/home/users/me/M31/data,keywords=SESSION;PANEL post,
smartNamingOverride=true" --force-exit

This command:

• runs WBPP in automation mode (automationMode=true)
• runs PixInsight in a free slot ( -n, instantiate a new dedicated PixInsight process if other PixInsight instances are running already)
• adds all files under the folder /home/users/me/M31/data (including the subfolders)
• outputs the results into the folder /home/users/me/M31/wbpp
• enables the smart naming override option (smartNamingOverride=true)
• adds two keywords, SESSION in pre and PANEL in post (common configuration of a multi-session multi-panel data)

Other improvements and bugfixes

• When a file type is not recognised, it is now added as a light frame (before it was discarded).
• This version comes with the following bugfixes:
  • the character ":" is now replaced by "-" in the group folder name to avoid folder creation errors in some OS
  • image measurements are disabled if both registration and subframe weighting is disabled
  • the maximum number of stars was not correctly handled in the Local Normalization panel
  • the master file names sanitization was corrupting the keyword values in some cases

As usual, feel free to provide any feedback; constructive critics and suggestions are always welcome!
Enjoy!

Robyx

 

[ngc1535]

Roberto,

Quite a thing you have done~! Congrats!!

The grouping of binning only applies to calibration correct? It is still possible to later register images of different sizes if desired?
(otherwise, a binning tolerance will be necessary.. oy!)
-adam

 

[robyx]

Roberto,

Quite a thing you have done~! Congrats!!

The grouping of binning only applies to calibration correct? It is still possible to later register images of different sizes if desired?
(otherwise, a binning tolerance will be necessary.. oy!)
-adam

Thanks Adam!

As it occurs for the binning, images of different sizes are never grouped together. This constraint is in place for post-calibration groups too in the current version, it could make sense to relax it on post-calibration groups since the geometry of frames aligned on the same reference frame should be the same, disregarding the pre-registration size.

To be discussed

 

[vland]

Wow, an impressive set of improvements.

I have a question: is the total integration time for each channel saved somewhere after a WBPP process?
I think it would be helpful to find a simple log/txt file somewhere with this information, since we often end up uploading our images to some website and then writing the integration times

 

[robyx]

Wow, an impressive set of improvements.

I have a question: is the total integration time for each channel saved somewhere after a WBPP process?
I think it would be helpful to find a simple log/txt file somewhere with this information, since we often end up uploading our images to some website and then writing the integration times

Hi @vland, it's not saved, but it's accessible in the post-calibrated panel.
It could be anyway nice information to be shown in the final smart report for the post-calibration groups, I think I'll integrate it.

thanks for the suggestion,
Robyx

 

[silentrunning]

Fantastic. WBPP really is something to behold.

Would it be possible to make the monitor window resizable and to keep the column sizes constant etc each time the running task changes? At the moment they reset each time it advances - my file names are very long!

Thanks so much.

John

 

[robyx]

Fantastic. WBPP really is something to behold.

Would it be possible to make the monitor window resizable and to keep the column sizes constant etc each time the running task changes? At the moment they reset each time it advances - my file names are very long!

Thanks so much.

John

Thanks a lot @silentrunning for your kind words!
Keeping the column width unchanged after a task is completed is for sure a visual improvement. I'll note it down thanks!

 

[gregwjones]

I love the new features in WBPP 2.4.3.

I ran a set of 700 light frames from several nights and several filters and all worked well, as expected, except the first group reported an error on the calibration. It then skipped that failed calibration step and completed all other tasks, including running all other steps for that failed group. The log is attached.

The error line is:
"
[2022-05-19 14:50:48] Operation queue error: TypeError: null is not a function
"

I reprocessed just that group and it completed with no errors.

Please advise,
Greg

 

[robyx]

I love the new features in WBPP 2.4.3.

I ran a set of 700 light frames from several nights and several filters and all worked well, as expected, except the first group reported an error on the calibration. It then skipped that failed calibration step and completed all other tasks, including running all other steps for that failed group. The log is attached.

The error line is:
"
[2022-05-19 14:50:48] Operation queue error: TypeError: null is not a function
"

I reprocessed just that group and it completed with no errors.

Please advise,
Greg

Thanks, I’ll investigate it!

 

[Linwood]

Hi @vland, it's not saved, but it's accessible in the post-calibrated panel.
It could be anyway nice information to be shown in the final smart report for the post-calibration groups, I think I'll integrate it.

thanks for the suggestion,
Robyx

Is there a place in the fits header of the integration that it might be appropriate to include this info (number of subs and total integration time)? A more permanent record. It's something of a pain to go back when you might want to post this info, especially since it is not obvious which subs may have been rejected along the way if all you saved were the raw subs. Maybe even a history like breakdown of the groupings with a total. It's not like there's a huge price to pay for more history in there.

 

[apolkowski]

Hi, I tried to run new WBPP to process lights and flats without any darks and bias frames. The monitor is showing flats integration and then debayer without any lights calibration. Is it a bug or am I doing something wrong?

The calibration diagram shows, that lights should be calibrated with flats:

When I add bias frames, calibration step is back.

 

[robyx]

Hi @apolkowski,

The reason why there is no calibration step is that there is no matching between the flats and any master bias/dark. Double-check the calibration panel and you'll see that most probably you've added a master dark with mismatching duration. In this case, the "auto" matching logic of WBPP won't use that master dark, but you can still manually assign it if you really want to use it (be careful when you have mismatching durations, there are many cases where this causes problems).

 

[astrokari]

I'm experiencing some unexpected behavior in the latest WBPP script. WBPP is correctly grouping frames by filter/exposure/resolution in the "Lights" tab, but over in the "Calibration" tab, the images are separated into two sets:

(note: none of these are CFA images, and overriding them all to "no" still doesn't resolve the grouping)

Here's a look at the "Diagnostics" section:

I'm only seeing this behavior with 1 (small) .fits dataset so far — it's possible that something in the headers is contributing to this issue. Will WBPP separate Lights sets based on any metadata besides binning, resolution, exposure, and filter?

 

[pfile]

seems like maybe WBPP has intepreted some images as CFA incorrectly? what happens if you override that?

 

[astrokari]

seems like maybe WBPP has intepreted some images as CFA incorrectly? what happens if you override that?

I unchecked "CFA Images" and clicked "Apply to all light frames". Unfortunately, it didn't help.

Could there be some FITS header information that contributes to this? The images that came in as "CFA = no" were from a different imaging session from the others that came in as "CFA = Auto". (data from iTelescope.net, so I didn't personally specify the metadata settings)

 

[astrokari]

seems like maybe WBPP has intepreted some images as CFA incorrectly? what happens if you override that?

OK, I figured it out! It was a problematic FITS header entry in the older part of my dataset which caused this issue:

I discovered this by doing some A/B comparisons between the subs, and removing this entry from the FITS headers fixed the issue. So it's not a WBPP problem at all. In fact, I would expect WBPP to behave just as it did wrt/ this header. My issue just coincided with a new release and confounded me for a bit ?. Thanks for your troubleshooting help and looking into this!

 

[robyx]

View attachment 14795

I unchecked "CFA Images" and clicked "Apply to all light frames". Unfortunately, it didn't help.

Could there be some FITS header information that contributes to this? The images that came in as "CFA = no" were from a different imaging session from the others that came in as "CFA = Auto". (data from iTelescope.net, so I didn't personally specify the metadata settings)

looking at these settings, I would expect WBPP to group the files with the same size, binning, exposure and Filter unless you're also using a keyword to separate the lights. Is that the case?

 

[astrokari]

looking at these settings, I would expect WBPP to group the files with the same size, binning, exposure and Filter unless you're also using a keyword to separate the lights. Is that the case?

I did not use any keywords. I even tried to use some keywords at one point to see if I could override the behavior (it did not, so I abandoned the idea). I fully reset the WBPP script settings in my own troubleshooting, so the issue I described happens after simply loading Lights into it.

Re: my previous message, removing the BAYERPAT FITS headers resolved the issue — so I believe the WBPP script (on some level) is grouping images based on this and not only the criteria you listed.

I remember that the telescope system I used had some camera / software pipeline issues around the time I took the images with "BAYERPAT = 'INVALID'". More recent images (after those issues were fixed) correctly do not show this header. Under most circumstances, I'd imagine that an 'INVALID' Bayer pattern entry wouldn't be present in the headers, so this is most likely an edge case. But it sounds like it's still revealing some unexpected behavior in WBPP?

 

[robyx]

But it sounds like it's still revealing some unexpected behavior in WBPP?

It could be, I will investigate using the same keyword value!

 

[MLM3]

Really like the WBPP Execution monitor and the calculation of disc space requirements, as well as the other enhancements.