PixInsight Forum (historical)
PixInsight => General => Topic started by: robyx on 2019 November 15 10:57:30
-
Weighted Batch Preprocessing (WBPP)
Hi all,
I would announce a side project I'm working on with Tommaso Rubechi: an upgraded version of Batch Preprocessing script called Weighted Batch Preprocessing.
The reason behind this upgrade is that many mid and advanced users are not using BPP because they interpone the Subframe Selector weighting during the process. Our new upgrade integrates this step, providing a user friendly interface to combine SNR, FWHM and eccentricity to generate the light frame weights.
The list of new features are the following:
- possibility of weighing our Light frames thanks to the Subframe Selector
- possibility to automatically select the reference frame for Light frames registration
- possibility of grouping Light frames based on different integration times, useful for creating HDR images later
- automatic choice of the best pixel rejection algorithm in the integration phase
- smart file naming
- an udpdated approach to flat frames calibration
- extended diagnostic
- smart reporting
- session saving
You can find a detailed description and you can download the current script version 1.3.5 of on this page:
english: https://www.tommasorubechi.it/2019/11/15/the-new-weighted-batchpreprocessing/
italian: https://www.tommasorubechi.it/2019/11/15/the-new-weighted-batchpreprocessing/
why WBPP and how it works in details: https://www.tommasorubechi.it/wp-content/uploads/2019/11/WBPP-Guide.pdf
I hope you will appreciate and use this new version.
Several improvements are already planned and new versions will come.
BUG REPORTING:
mail to weightedbpp@gmail.com
- If you have errors while using the dialog widow try to attach a screenshot of the error
- if you have issues during the integration execution activate the "Save process log" option and attach the last created log file you can find into the "logs" folder.
Stay tuned!
Robyx
-
Hi Robyx,
Very nice work. Why don't you contribute this new version to our official open-source repositories:
https://gitlab.com/pixinsight/
Of course, you are free to create a side project and stay that way if you want, since BPP is open source and its license gives you the right to do that. However, since BPP is a crucial component of the PixInsight distribution, your new version of this script should eventually replace the current version, instead of living as a side project isolated from the official distribution. This is just my point of view, and how I think your work would be more useful for most PixInsight users. At any rate, irrespective of how you are going to manage this, nice work.
-
Hi Juan,
thank you very very much!! indeed the sources are already there in the branch WBPP (not updated to 1.3.5 but I will do it), I planned to propose this as the new BPP 2.0 but I prefer at the moment to pass a deep public testing phase before making the big step to push it into PI.
There are also other features we didn't mentioned which we would like to implement like multiple sessions integration, processing completion time estimation and more but they will come with no pressure.
As soon as this version will be proved to be stable for a while (I need people to test it with different PI versions and many many different sessions in terms of combinations of binning/filter/cameras etc) I will be pleased to merge it into the BPP to become the new BPP 2.0.
So for sure we'll get in touch soon.
Regards,
Robyx
-
Ho do we report bugs?
-
Ho do we report bugs?
I added bug report instructions at the bottom of the original message.
Regards,
Robyx
-
Warning Messages
Here’s an example of warning messages. One for each Light Sub. The WBPP output directory was empty. No other runs or files. I am guessing that the new scrip does double passes on existing data. I only see this if I rerun BPP and overwrite a previous run.
It does this 3 times - Begin computation of weights for images.
************************************************************
* Begin computation of weights for images
************************************************************
Loading image: w=5496 h=3670 n=3 RGB Float32
10 image properties
67 FITS keyword(s) extracted.
Weights of image: C:/Users/JTWaters/Desktop/registered/NoFilter/M4_LIGHT_90s_6D_1600_+61f_0343600027_c_d_r.xisf
-----------------------
FWHM : 53.08 %
eccentricity : 79.00 %
SNR : 57.97 %
Image weight : 87.15
-----------------------
Writing file:
C:/Users/JTWaters/Desktop/registered/NoFilter/M4_LIGHT_90s_6D_1600_+61f_0343600027_c_d_r.xisf
** Warning: Overwriting already existing file: C:/Users/JTWaters/Desktop/registered/NoFilter/M4_LIGHT_90s_6D_1600_+61f_0343600027_c_d_r.xisf
Writing image: w=5496 h=3670 n=3 RGB Float32
68 FITS keyword(s) embedded.
ICC profile embedded: 'Lenovo ThinkPad LCD Monitor', 1038 bytes.
10 image properties embedded.
Loading image: w=5496 h=3670 n=3 RGB Float32
10 image properties
67 FITS keyword(s) extracted.
-
Warning Messages
Here’s an example of warning messages. One for each Light Sub. The WBPP output directory was empty. No other runs or files. I am guessing that the new scrip does double passes on existing data. I only see this if I rerun BPP and overwrite a previous run.
Weights are computed and saved into the image's fit header, this operation requires to overwrite the existing file with the new version that includes the weight. In the console logs you see the warning about this file overwriting operation.
I assume you're weighting the light frames after the registration, that's why you should be warned about overwriting the registered frames.
regards,
Robyx
-
YES - It does this 3 times - "Begin computation of weights for images."
What would happen if this BOX is Unchecked - Subframe Weighting - "Compute weights after registration"?
Is this necessary?
-
YES - It does this 3 times - "Begin computation of weights for images."
What would happen if this BOX is Unchecked - Subframe Weighting - "Compute weights after registration"?
Is this necessary?
The log "Begin computation of weights for images" is printed for each light frame group at the time all the information needed to compute the weights are available.
If you selected the option to compute the weights after the registration each group will be first registered and then registered light frames are measured and weights are stored. If, differently, you uncheck the flag the light frames measurement and weighting is executed before the registration (so the warning will be referred to the calibrated or debayered images depending on which is the last step before the registration).
If you want to know exactly how the processing workflow works depending on the configuration of these options you can read this document https://www.tommasorubechi.it/wp-content/uploads/2019/11/WBPP-Guide.pdf that I wrote to describe in detail how the WBPP works.
In here I explain the exact sequence of processing flow depending on the configuration of the weighting flags.
Regards,
Robyx
-
Thank-you Robyx
-
Hi,
I've sent you an email about this problem I've encountered with the script. It seems it is not reading the exposure times correctly from the FITS header. This causes the image files to be grouped incorrectly for calibration and integration. For example all the darks and flat darks are getting integrated together and not by the exposure time. My 60 second light frames are showing as "0.00 secs".
All subframes are captured by TheSkyX Pro which uses the FITS keyword EXPTIME. Not sure if this is the cause or perhaps a misinterpretation of the actual value itself.
-
Would this also impact Canon CR2 subs?
-
Hi,
I've sent you an email about this problem I've encountered with the script. It seems it is not reading the exposure times correctly from the FITS header. This causes the image files to be grouped incorrectly for calibration and integration. For example all the darks and flat darks are getting integrated together and not by the exposure time. My 60 second light frames are showing as "0.00 secs".
All subframes are captured by TheSkyX Pro which uses the FITS keyword EXPTIME. Not sure if this is the cause or perhaps a misinterpretation of the actual value itself.
Hi,
the reason why you see zero seconds is that the duration is not stored in the file meta data or for some reason cannot be read by the PI apis. WBPP s compatible with both EXPTIME and EXPOSURE keywords but at the end, I don't know why, they are not read from the file (the reason can be investigated only by owning the files).
This is a perfect example where you can take advantage of the new smart naming feature. Just rename the folders that contain your files by adding the missing data. For example, if the folder containing light frames is called "lights" just rename it "lights_60s" and the duration will be taken from it. You can rename single files if you prefer adding the same info to each.
You can also add binning and filter with the following syntax: "FILTER-Ha_BINNING-1", both, if missing, will be extracted by the file name or the containing folder.
So, at the end, if you rename
- "dark_BINNING-1_60s" for the dark frames folder
- "dark_BINNING-1_2s" for the flat darks folder
- "flat_BINNING-1_FILTER-Ha_2s" for the flats folder
- "light_BINNING-1_FILTER-Ha_60s" for the lights folder
the missing information will be read from these names.
NOTE: the filter name read by smart naming is alphanumeric only so the name H-a will be interpreted as "H" only. In your case you don't even need to add it since it's correctly read form the FITS header as H-a.
Let me know if you face further issues :)
Robyx
-
Hi robyx
I'll give that a try. I will be transitioning from TheSkyX Pro to SGP for image acquisition. Will this correct things or should I just incorporate it into the file naming?
Dale
-
Hi robyx
I'll give that a try. I will be transitioning from TheSkyX Pro to SGP for image acquisition. Will this correct things or should I just incorporate it into the file naming?
Dale
I cannot guarantee since it depends on the metadata stored into the files and I don't have any experience with SGP. What I know is that we had testers that used SGP and they had no issues.
Please check also the new release 1.4.0 posted here: https://pixinsight.com/forum/index.php?topic=14232.0
Keep me updated :)
-
i think you should be OK, all of my files captured with SGP across many versions have the EXPOSURE keyword.
rob
-
Would it help if I send you a couple of raw FITS subframes via email?
-
absolutely yes: weightedbpp@gmail.com
-
Thank you Robyx and team, for this excellent update!! I have run BPP for over 1000 DSO images over the past 3 years, and I love the time savings of BPP vs. the manual process, so this is a valuable improvement for me. I haven't run the new script yet, but it looks very promising.
A few suggestions for future updates:
- It would be great if the end summary printout could include a summary of how many subs were successfully registered for each group (or ideally, how many subs FAILED!). Often I am surprised to see that a number of subs haven't registered. If I catch this right away, it is then a simple matter of running the alignment process manually and using noise reduction to achieve alignment of all subs.
- It would be great is BPP included the option for large scale pixel rejection. This is the main reason now why I will often have to make the final run manually instead of with BPP.
Thanks again,
Gary
-
Excellent upgrade to this very useful tool! Thanks a lot!
Would it be possible to integrate LocalNormalization to the script ? One could prepare the reference frame before starting the script, just like we are preparing the CosmeticCorrection process icon.
-
Hi Sedat,
Thank you.
The local normalization problem is ill-posed by its nature. LocalNormalization should never be used blindly; you should apply it only when necessary, and always under strict control, verifying its results. An improper application of local normalization can easily generate large-scale artifacts that invalidate any subsequent result. For this reason I implemented many control resources in the LN tool, such as background models, rejection maps and 3-D function representations.
If you want to use LN, use WBPP to calibrate and register your frames, then apply LN to the registered frames.
Besides that, this cannot be stressed enough: ImageIntegration must always be performed manually; the integrated image generated by WBPP is just a rough preview that cannot be used for production purposes.
-
Thank you Juan! I was expecting this response :) LN is magical when composing mosaics and these days I'm into more mosaics then I can handle. I'll take my time and do the job manually.
-
Please verify the check box to measure quality descriptor after registration is working . It does it on on the calibrated frames when checked on my system.
Also, pause/abort would not stop the script.
I vote for large scale rejection too.
It would be nice to do calibrate and registration only as an option
-
WBPP has got me back to using the script again and saving time. Thank you.
Some things I note:
When the script is closed and reopened, the Output folder isn't recalled and the Cosmetic Correction checkbox gets un-ticked.
It might be useful if the Smart Report could be saved as a text file (or added to the end of the log file?) for future reference, especially if you forget to make a note of the reference frame!
I stopped using the BPP script when I realised many of my images required me to add a pedestal value. Could this be included in WBPP in future?
Many thanks for improving this script for us all.
John
-
Juan, Robyx and the other developers of WBPP. I have spent this week going back and reprocessing my images with WBPP. I have noticed an improvement in my images. They show more detail and I believe less background noise. Previously I use the BPP script.
Thank-you for releasing this script. The script is an excellent upgrade to PixInsight.
Some suggestions for the next release:
- It would be great if the script could calculate the 'approximate' execution time.
- The Output Folder location gets cleared after the script is closed. Can this be retained?
- Include a Flat Darks option.
- Append the Output summary to the execution log file.
-
This has been super helpful, and a wonderful addition to the program, thank you for working on it.
I had a question -- as I shoot in monochrome, is there a way to do registration on a per filter basis for the lights? If I add in all my lights, WBPP successfully identifies and groups the files per filter, but then only registers them against the single best light frame, whereas if my understanding is right, one should register and integrate each filter. I'm not sure if this functionality exists within the WBPP and I just missed it, or if there's a technical limitation preventing it, so I figured I'd ask.
Thanks again, great work!
-
no, you want to register against the best frame across all filters. if you do it how you are proposing you end up with the channel masters being misaligned to one another, and then you'd have to register two of them to the third... and that's a double interpolation which should be avoided.
rob
-
I got some problems with file registration using WBPP, and ot looks like it blocks using WBPP at all. Because if WBPP fails with file registration, it does no write weights calculated to files. There is no options "calc weights, but do not make registration".
The problem is disscussed in separated thread: https://pixinsight.com/forum/index.php?topic=14478.msg87016#msg87016