Weighted Batch Preprocessing update - v1.4.0

[robyx]

Hi all,

due to the early feedbacks provided by several users we released an update of our Weighted Batch Preprocessing to version 1.4.0.

The updates are:

- auto-selection of reference frame update: in version 1.3.5 the reference frame selection is performed within the whole set of light frames. Since light frames with higher binning has always smaller and rounder stars then the reference frame is probably chosen among the light frames with the highest BINNING. That's exactly the opposite that what is typically done since light frames with higher binning are typically aligned to light frames with lower binning. In LRGB composition, for example, it's not uncommon to take L with BINNING 1 and RGB with BINNING 2 then RGB are aligned to L (and not viceversa). Now the reference frame is selected within the set of light frames with the lowest BINNING.

- light frame exposure tolerance: many users reported that their light frames have small exposure differences (fraction of seconds) which caused the light frames to be grouped separately (like groups of 300, 300.1 and 300.2 secs). We introduced the light frame exposure tolerance as for dark frames such that you can provide the tolerance you need to properly group light frames with short exposure time differences.

- Drizzle Files update (not a bugix as reported below): it has been reported that drizzle files generated by BPP (and so WBPP) do not work with drizzle integration. The reason is that these files are not updated during Light Frame integration in order to add the additional rejection map, statistical and extra metadata digested by Drizzle Integration process. This was done by purpose. With this update we proposed a change to this approach and now you can execute a Drizzle Integration at the end of the process using the generated drizzle files.

- minor bugfixes

You can download the new version here: https://www.tommasorubechi.it/wp-content/uploads/2019/11/weightedBatchPreprocessing1.4.0.zip

----------------------

english tutorial (version 1.3.5): https://www.tommasorubechi.it/2019/11/15/the-new-weighted-batchpreprocessing/
italian tutorial (version 1.3.5): https://www.tommasorubechi.it/2019/11/15/the-new-weighted-batchpreprocessing/
why WBPP and how it works in details: https://www.tommasorubechi.it/wp-content/uploads/2019/11/WBPP-Guide.pdf

BUG REPORTING:
mail to weightedbpp@gmail.com
If you have errors while using the dialog widow try to attach a screenshot of the error
if you have issues during the integration execution activate the "Save process log" option and attach the last created log file you can find into the "logs" folder.

Regards,
Robyx

[Juan Conejero]

Hi Robyx,

Absolutely stunning work. You are doing a very good work, and I feel fortunate for having such a team of developers with your expertise and freshness working on our platform.

- Drizzle Files bug fix: it has been reported that drizzle files generated by BPP (and so WBPP) do not work with drizzle integration. The reason is that these files are not updated during Light Frame integration in order to add the additional rejection map, statistical and extra metadata digested by Drizzle Integration process. This bug is fixed and now you can execute a Drizzle Integration at the end of the process using the generated drizzle files.

This is not a bug. It is intentional. The reason is that integration of light frames with BPP is *not* intended to generate a final integrated image. It is just a rough preview of the achievable image, but by no means the script's result should be used for production purposes. Drizzle files are not updated because there is no reason to do that with invalid rejection data.

The decision to include integration of light frames in BPP was unfortunate in my opinion. Integration of light frames (and, for that matter, drizzle integration) is not part of preprocessing, but the very first step of postprocessing. If I were to release a first version of BPP now, this feature would probably not be included.

[robyx]

Hi Juan,

your concern makes perfectly sense and I also would agree with you. Of course we're both aware that at this point users are so used to get a master integration frame out of the bpp execution that practically it's just a non-return point, I cannot imagine how many people will complain if we remove it

[Juan Conejero]

I cannot imagine how many people will complain if we remove it

Better don't try to imagine things like that, if you want to sleep well from now on 

[dchamberlain]

Hello

I've been working with Roberto on a strange problem I've run into with WBPP. Maybe someone here might have some insight on this.

I have light frames binned 1x1, for H-a, O-III and S-II that are 60 seconds in exposure. They were created from TheSkyX Pro. When I add these light frames into WBPP, they are grouped by the right bin, and filter type, but all show as "0.00 secs" for exposure time. I sent a couple of the FITS files to Roberto who verified that the FITS header had the correct exposure times and was interpreted by WBPP correctly. The key difference is I am running Windows version 10.0.17763 Build 17763 and Roberto is on a Mac.

If I put the lights into a folder named "LIGHT 60s" it will pick up the correct exposure time of 60 seconds. This is the fallback method in case the FITS header fails.

I am running PI 1.8.8 and WBPP version 1.4.0.

Any ideas on what could be happening?

Thanks

Dale

[mexbound]

I was wondering how this script uses "Flat Darks" instead of "Bias" frames. Would I just load my "Flat Darks" into the "Bias" tab? The script looks great!

[robyx]

Hi mexbound

- answer -
to use the flat darks you should simply add them as dark frames, they will be grouped by time exposure and they will remain separated from dark frames groups which have for sure longer exposure time.

- a bit deeper into the topic -
When calibrating flat frames, two scenarios are possible:

1. "Calibrate with flat dark only" is checked: WBPP searches the master dark with the EXACTLY same exposure as flat frames, if found it will be used to calibrate the flat frames (which is your case since you will add flat darks in the darks panel)

2. "Calibrate with flat dark only" is NOT checked: in this case WBPP searches for the masterDark that has the closest exposure time - but not necessarily a similar exposure time. A common situation is when you add only dark frames of 300s and your flats are 2 sec, the masterDark of 300s will be used to calibrate flat frames. Depending on your camera and other settings of WBPP such difference could work or this huge exposure difference could result in an issued masterFlat (and thus masterLight) frame .

If you add your flat darks as bias, differently, they will behave like bias frames so they will be used to calibrate all groups with the same binning. Most probably that's not what you're looking for unless flat darks are so short that they are simply undistinguishable from bias frames (that's not usually the case). But in this latter case I would prefer to call them bias frames and not flat darks.

Regards,
Robyx

[mexbound]

Thanks you for your answer to my question. Worked perfectly!

[Sedat]

Thanks for all the time and effort you put into this script. I've just installed it and hope to use it soon on my huge back log of targets.

Clear skies

[dchamberlain]

Everyone,

Disregard my earlier post about the EXPTIME not being picked up in my subframes. I was clearly an idiot because I selected the .xisf files I had from a previous calibration run rather than the raw FITS files. Of course, I would not expect the script to treat the .xisf the same as the FITS.

Once I selected all FITS files it worked perfectly.

Thanks to Roberto and team for creating this wonderful script. I will now be incorporating it into my workflow and I hope that PixInsight will make it part of the standard package.

Clear Skies,

Dale

[gnewell]

I realize this is a bit of a corner case but would it be possible to add functionality for the Weighted Batch Preprocessing script to work with files that had already been calibrated (outside of pixinsight)?

For instance fits files acquired via iTelescope.net are precalibrated (or if you choose the not already calibrated ones it is difficult/impossible to find the matching calibration files).

[robyx]

Hi,

indeed WBPP can proceed without the need of masterBias, masterDark and masterFlat. You can add already debayered images as Lights, disable the "CFA images" to skip the debayering and it will start from the measurement and weighting of your lights to subsequently register and do the integration.

If your images are not yet debayered just check the "CFA images" option.

I hope this helps!

[gnewell]

Very  Helpful. It works!

You just ignore the warnings about no calibration files and proceed.

Thanks.

[robyx]

Very  Helpful. It works!

You just ignore the warnings about no calibration files and proceed.

Thanks.

Great!!

[aatdalt]

Hello, I've noticed (and had a friend confirm) that when using the WBPP for calibration only, that the image weights can be calculated but the information isn't written anywhere. It seems like this only happens if registration is being done as well. Is there a way to enable the keywords (FWHM, eccen, and weight) to be written even if we'd like to manually register and integrate later? Really appreciate the batch features for LRGB and SHO data sets. Saves a ton of effort.