Batch Preprocessing upgrade: the Weighted Batch Preprocessing

[robyx]

Weighted Batch Preprocessing (WBPP)

Hi all,

I would announce a side project I'm working on with Tommaso Rubechi: an upgraded version of Batch Preprocessing script called Weighted Batch Preprocessing.

The reason behind this upgrade is that many mid and advanced users are not using BPP because they interpone the Subframe Selector weighting during the process. Our new upgrade integrates this step, providing a user friendly interface to combine SNR, FWHM and eccentricity to generate the light frame weights.

The list of new features are the following:

• possibility of weighing our Light frames thanks to the Subframe Selector
• possibility to automatically select the reference frame for Light frames registration
• possibility of grouping Light frames based on different integration times, useful for creating HDR images later
• automatic choice of the best pixel rejection algorithm in the integration phase
• smart file naming
• an udpdated approach to flat frames calibration
• extended diagnostic
• smart reporting
• session saving

You can find a detailed description and you can download the current script version 1.3.5 of on this page:

english: https://www.tommasorubechi.it/2019/11/15/the-new-weighted-batchpreprocessing/
italian: https://www.tommasorubechi.it/2019/11/15/the-new-weighted-batchpreprocessing/
why WBPP and how it works in details: https://www.tommasorubechi.it/wp-content/uploads/2019/11/WBPP-Guide.pdf

I hope you will appreciate and use this new version.
Several improvements are already planned and new versions will come.

BUG REPORTING:
mail to weightedbpp@gmail.com

• If you have errors while using the dialog widow try to attach a screenshot of the error
• if you have issues during the integration execution activate the "Save process log" option and attach the last created log file you can find into the "logs" folder.

Stay tuned!
Robyx

[Juan Conejero]

Hi Robyx,

Very nice work. Why don't you contribute this new version to our official open-source repositories:

https://gitlab.com/pixinsight/

Of course, you are free to create a side project and stay that way if you want, since BPP is open source and its license gives you the right to do that. However, since BPP is a crucial component of the PixInsight distribution, your new version of this script should eventually replace the current version, instead of living as a side project isolated from the official distribution. This is just my point of view, and how I think your work would be more useful for most PixInsight users. At any rate, irrespective of how you are going to manage this, nice work.

[robyx]

Hi Juan,

thank you very very much!! indeed the sources are already there in the branch WBPP (not updated to 1.3.5 but I will do it), I planned to propose this as the new BPP 2.0 but I prefer at the moment to pass a deep public testing phase before making the big step to push it into PI.

There are also other features we didn't mentioned which we would like to implement like multiple sessions integration, processing completion time estimation and more but they will come with no pressure.

As soon as this version will be proved to be stable for a while (I need people to test it with different PI versions and many many different sessions in terms of combinations of binning/filter/cameras etc) I will be pleased to merge it into the BPP to become the new BPP 2.0.

So for sure we'll get in touch soon.

Regards,
Robyx

[jwaters125]

Ho do we report bugs?

[robyx]

Ho do we report bugs?

I added bug report instructions at the bottom of the original message.

Regards,
Robyx

[jwaters125]

Warning Messages

Here’s an example of warning messages.  One for each Light Sub.  The WBPP output directory was empty.  No other runs or files.  I am guessing that the new scrip does double passes on existing data.  I only see this if I rerun BPP and overwrite a previous run.

It does this 3 times - Begin computation of weights for images.


************************************************************
* Begin computation of weights for images
************************************************************
Loading image: w=5496 h=3670 n=3 RGB Float32
10 image properties
67 FITS keyword(s) extracted.
Weights of image: C:/Users/JTWaters/Desktop/registered/NoFilter/M4_LIGHT_90s_6D_1600_+61f_0343600027_c_d_r.xisf
-----------------------
 FWHM         : 53.08 %
 eccentricity : 79.00 %
 SNR          : 57.97 %
 Image weight : 87.15
-----------------------

Writing file:
C:/Users/JTWaters/Desktop/registered/NoFilter/M4_LIGHT_90s_6D_1600_+61f_0343600027_c_d_r.xisf
** Warning: Overwriting already existing file: C:/Users/JTWaters/Desktop/registered/NoFilter/M4_LIGHT_90s_6D_1600_+61f_0343600027_c_d_r.xisf
Writing image: w=5496 h=3670 n=3 RGB Float32
68 FITS keyword(s) embedded.
ICC profile embedded: 'Lenovo ThinkPad LCD Monitor', 1038 bytes.
10 image properties embedded.
Loading image: w=5496 h=3670 n=3 RGB Float32
10 image properties
67 FITS keyword(s) extracted.

[robyx]

Warning Messages

Here’s an example of warning messages.  One for each Light Sub.  The WBPP output directory was empty.  No other runs or files.  I am guessing that the new scrip does double passes on existing data.  I only see this if I rerun BPP and overwrite a previous run.

Weights are computed and saved into the image's fit header, this operation requires to overwrite the existing file with the new version that includes the weight. In the console logs you see the warning about this file overwriting operation.

I assume you're weighting the light frames after the registration, that's why you should be warned about overwriting the registered frames.

regards,
Robyx

[jwaters125]

YES - It does this 3 times - "Begin computation of weights for images."

What would happen if this BOX is Unchecked - Subframe Weighting - "Compute weights after registration"? 

Is this necessary?

[robyx]

YES - It does this 3 times - "Begin computation of weights for images."

What would happen if this BOX is Unchecked - Subframe Weighting - "Compute weights after registration"? 

Is this necessary?

The log "Begin computation of weights for images" is printed for each light frame group at the time all the information needed to compute the weights are available.

If you selected the option to compute the weights after the registration each group will be first registered and then registered light frames are measured and weights are stored. If, differently, you uncheck the flag the light frames measurement and weighting is executed before the registration (so the warning will be referred to the calibrated or debayered images depending on which is the last step before the registration).

If you want to know exactly how the processing workflow works depending on the configuration of these options you can read this document https://www.tommasorubechi.it/wp-content/uploads/2019/11/WBPP-Guide.pdf that I wrote to describe in detail how the WBPP works.

In here I explain the exact sequence of processing flow depending on the configuration of the weighting flags.

Regards,
Robyx

[jwaters125]

Thank-you Robyx

[dchamberlain]

Hi,

I've sent you an email about this problem I've encountered with the script. It seems it is not reading the exposure times correctly from the FITS header.  This causes the image files to be grouped incorrectly for calibration and integration. For example all the darks and flat darks are getting integrated together and not by the exposure time. My 60 second light frames are showing as "0.00 secs".

All subframes are captured by TheSkyX Pro which uses the FITS keyword EXPTIME. Not sure if this is the cause or perhaps a misinterpretation of the actual value itself.

[jwaters125]

Would this also impact Canon CR2 subs?

[robyx]

Hi,

I've sent you an email about this problem I've encountered with the script. It seems it is not reading the exposure times correctly from the FITS header.  This causes the image files to be grouped incorrectly for calibration and integration. For example all the darks and flat darks are getting integrated together and not by the exposure time. My 60 second light frames are showing as "0.00 secs".

All subframes are captured by TheSkyX Pro which uses the FITS keyword EXPTIME. Not sure if this is the cause or perhaps a misinterpretation of the actual value itself.

Hi,
the reason why you see zero seconds is that the duration is not stored in the file meta data or for some reason cannot be read by the PI apis. WBPP s compatible with both EXPTIME and EXPOSURE keywords but at the end, I don't know why, they are not read from the file (the reason can be investigated only by owning the files).

This is a perfect example where you can take advantage of the new smart naming feature. Just rename the folders that contain your files by adding the missing data. For example, if the folder containing light frames is called "lights" just rename it "lights_60s" and the duration will be taken from it. You can rename single files if you prefer adding the same info to each.

You can also add binning and filter with the following syntax: "FILTER-Ha_BINNING-1", both, if missing, will be extracted by the file name or the containing folder.

So, at the end, if you rename
- "dark_BINNING-1_60s" for the dark frames folder
- "dark_BINNING-1_2s" for the flat darks folder
- "flat_BINNING-1_FILTER-Ha_2s" for the flats folder
- "light_BINNING-1_FILTER-Ha_60s" for the lights folder

the missing information will be read from these names.

NOTE: the filter name read by smart naming is alphanumeric only so the name H-a will be interpreted as "H" only. In your case you don't even need to add it since it's correctly read form the FITS header as H-a.

Let me know if you face further issues
Robyx

[dchamberlain]

Hi robyx

I'll give that a try. I will be transitioning from TheSkyX Pro to SGP for image acquisition.  Will this correct things or should I just incorporate it into the file naming?

Dale

[robyx]

Hi robyx

I'll give that a try. I will be transitioning from TheSkyX Pro to SGP for image acquisition.  Will this correct things or should I just incorporate it into the file naming?

Dale

I cannot guarantee since it depends on the metadata stored into the files and I don't have any experience with SGP. What I know is that we had testers that used SGP and they had no issues.

Please check also the new release 1.4.0 posted here: https://pixinsight.com/forum/index.php?topic=14232.0

Keep me updated