Weighted Batch Preprocessing update - v1.4.0

[robyx]

Hi all,

due to the early feedbacks provided by several users we released an update of our Weighted Batch Preprocessing to version 1.4.0.

The updates are:

- auto-selection of reference frame update: in version 1.3.5 the reference frame selection is performed within the whole set of light frames. Since light frames with higher binning has always smaller and rounder stars then the reference frame is probably chosen among the light frames with the highest BINNING. That's exactly the opposite that what is typically done since light frames with higher binning are typically aligned to light frames with lower binning. In LRGB composition, for example, it's not uncommon to take L with BINNING 1 and RGB with BINNING 2 then RGB are aligned to L (and not viceversa). Now the reference frame is selected within the set of light frames with the lowest BINNING.

- light frame exposure tolerance: many users reported that their light frames have small exposure differences (fraction of seconds) which caused the light frames to be grouped separately (like groups of 300, 300.1 and 300.2 secs). We introduced the light frame exposure tolerance as for dark frames such that you can provide the tolerance you need to properly group light frames with short exposure time differences.

- Drizzle Files update (not a bugix as reported below): it has been reported that drizzle files generated by BPP (and so WBPP) do not work with drizzle integration. The reason is that these files are not updated during Light Frame integration in order to add the additional rejection map, statistical and extra metadata digested by Drizzle Integration process. This was done by purpose. With this update we proposed a change to this approach and now you can execute a Drizzle Integration at the end of the process using the generated drizzle files.

- minor bugfixes

You can download the new version here: https://www.tommasorubechi.it/wp-content/uploads/2019/11/weightedBatchPreprocessing1.4.0.zip

----------------------

english tutorial (version 1.3.5): https://www.tommasorubechi.it/2019/11/15/the-new-weighted-batchpreprocessing/
italian tutorial (version 1.3.5): https://www.tommasorubechi.it/2019/11/15/the-new-weighted-batchpreprocessing/
why WBPP and how it works in details: https://www.tommasorubechi.it/wp-content/uploads/2019/11/WBPP-Guide.pdf

BUG REPORTING:
mail to weightedbpp@gmail.com
If you have errors while using the dialog widow try to attach a screenshot of the error
if you have issues during the integration execution activate the "Save process log" option and attach the last created log file you can find into the "logs" folder.

Regards,
Robyx

 

[Juan Conejero]

Hi Robyx,

Absolutely stunning work. You are doing a very good work, and I feel fortunate for having such a team of developers with your expertise and freshness working on our platform.

- Drizzle Files bug fix: it has been reported that drizzle files generated by BPP (and so WBPP) do not work with drizzle integration. The reason is that these files are not updated during Light Frame integration in order to add the additional rejection map, statistical and extra metadata digested by Drizzle Integration process. This bug is fixed and now you can execute a Drizzle Integration at the end of the process using the generated drizzle files.

This is not a bug. It is intentional. The reason is that integration of light frames with BPP is *not* intended to generate a final integrated image. It is just a rough preview of the achievable image, but by no means the script's result should be used for production purposes. Drizzle files are not updated because there is no reason to do that with invalid rejection data.

The decision to include integration of light frames in BPP was unfortunate in my opinion. Integration of light frames (and, for that matter, drizzle integration) is not part of preprocessing, but the very first step of postprocessing. If I were to release a first version of BPP now, this feature would probably not be included.

 

[robyx]

Hi Juan,

your concern makes perfectly sense and I also would agree with you. Of course we're both aware that at this point users are so used to get a master integration frame out of the bpp execution that practically it's just a non-return point, I cannot imagine how many people will complain if we remove it

 

[Juan Conejero]

I cannot imagine how many people will complain if we remove it

Better don't try to imagine things like that, if you want to sleep well from now on  ;D

 

[dchamberlain]

Hello

I've been working with Roberto on a strange problem I've run into with WBPP. Maybe someone here might have some insight on this.

I have light frames binned 1x1, for H-a, O-III and S-II that are 60 seconds in exposure. They were created from TheSkyX Pro. When I add these light frames into WBPP, they are grouped by the right bin, and filter type, but all show as "0.00 secs" for exposure time. I sent a couple of the FITS files to Roberto who verified that the FITS header had the correct exposure times and was interpreted by WBPP correctly. The key difference is I am running Windows version 10.0.17763 Build 17763 and Roberto is on a Mac.

If I put the lights into a folder named "LIGHT 60s" it will pick up the correct exposure time of 60 seconds. This is the fallback method in case the FITS header fails.

I am running PI 1.8.8 and WBPP version 1.4.0.

Any ideas on what could be happening?

Thanks

Dale

 

[mexbound]

I was wondering how this script uses "Flat Darks" instead of "Bias" frames. Would I just load my "Flat Darks" into the "Bias" tab? The script looks great!

 

[robyx]

Hi mexbound

- answer -
to use the flat darks you should simply add them as dark frames, they will be grouped by time exposure and they will remain separated from dark frames groups which have for sure longer exposure time.

- a bit deeper into the topic -
When calibrating flat frames, two scenarios are possible:

1. "Calibrate with flat dark only" is checked: WBPP searches the master dark with the EXACTLY same exposure as flat frames, if found it will be used to calibrate the flat frames (which is your case since you will add flat darks in the darks panel)

2. "Calibrate with flat dark only" is NOT checked: in this case WBPP searches for the masterDark that has the closest exposure time - but not necessarily a similar exposure time. A common situation is when you add only dark frames of 300s and your flats are 2 sec, the masterDark of 300s will be used to calibrate flat frames. Depending on your camera and other settings of WBPP such difference could work or this huge exposure difference could result in an issued masterFlat (and thus masterLight) frame .

If you add your flat darks as bias, differently, they will behave like bias frames so they will be used to calibrate all groups with the same binning. Most probably that's not what you're looking for unless flat darks are so short that they are simply undistinguishable from bias frames (that's not usually the case). But in this latter case I would prefer to call them bias frames and not flat darks.

Regards,
Robyx

 

[mexbound]

Thanks you for your answer to my question. Worked perfectly!

 

[bilgebay]

Thanks for all the time and effort you put into this script. I've just installed it and hope to use it soon on my huge back log of targets.

Clear skies

 

[dchamberlain]

Everyone,

Disregard my earlier post about the EXPTIME not being picked up in my subframes. I was clearly an idiot because I selected the .xisf files I had from a previous calibration run rather than the raw FITS files. Of course, I would not expect the script to treat the .xisf the same as the FITS.

Once I selected all FITS files it worked perfectly.

Thanks to Roberto and team for creating this wonderful script. I will now be incorporating it into my workflow and I hope that PixInsight will make it part of the standard package.

Clear Skies,

Dale

 

[gnewell]

I realize this is a bit of a corner case but would it be possible to add functionality for the Weighted Batch Preprocessing script to work with files that had already been calibrated (outside of pixinsight)?

For instance fits files acquired via iTelescope.net are precalibrated (or if you choose the not already calibrated ones it is difficult/impossible to find the matching calibration files).

 

[robyx]

Hi,

indeed WBPP can proceed without the need of masterBias, masterDark and masterFlat. You can add already debayered images as Lights, disable the "CFA images" to skip the debayering and it will start from the measurement and weighting of your lights to subsequently register and do the integration.

If your images are not yet debayered just check the "CFA images" option.

I hope this helps!

 

[gnewell]

Very  Helpful. It works!

You just ignore the warnings about no calibration files and proceed.

Thanks.

 

[robyx]

Very  Helpful. It works!

You just ignore the warnings about no calibration files and proceed.

Thanks.

Great!!

 

[aatdalt]

Hello, I've noticed (and had a friend confirm) that when using the WBPP for calibration only, that the image weights can be calculated but the information isn't written anywhere. It seems like this only happens if registration is being done as well. Is there a way to enable the keywords (FWHM, eccen, and weight) to be written even if we'd like to manually register and integrate later? Really appreciate the batch features for LRGB and SHO data sets. Saves a ton of effort.

 

[tommy_nawratil]

dear Robyx,

thank you for your work on modernizing BPP, I am just about to discover the new possibilities!
Its an important help to keep lifetime used for processing lower, and nevertheless get better results.

But I have some questions I cannot get answered even from the more technical document:

- What are the calculation formulas WBPP uses to calibrate light frames?
Like: calibrated light = (light - masterdark) / (masterflat - masterbias) * (average flat - median bias)

- How are flatdarks entering that formula?

For example: I have a Pana CMOS sensor that shows a little diffuse ampglow.
Filling in Lights, Flats, Darks and Bias as usual, I tried the 4 possibilities switching the Optimize Dark and Calibrate Dark flags:

- optimize dark yes, calibrate dark yes: calculated masterlight shows ampglow imprinted
- optimize dark no, calibrate dark no: masterlight shows dark vignetting on rim, flat is apparently not applied
- optimize dark yes, calibrate dark no: masterlight shows ampglow imprinted
- optimize dark no, calibrate dark yes: masterlight is calculated correctly, same result as in old BPP
(but Sigma 1.096e-04 instead of 1.161e-04 - noise slightly improved)

So I'd really like to know what are the used calculation formulas, and what is changed using the different switches.
Getting to know this would be very helpful in choosing the proper switches!

Another question is about weighting before or after registering: What is the recommendation?
Registering of dithered subs means there will be some black areas, are they excluded at weighting?

thanks again for you efforts
Tommy

 

[robyx]

Hi Tommy,
first of all, let me tell you that your astroBin page is amazing!

I'll go through your questions:

- What are the calculation formulas WBPP uses to calibrate light frames?
- How are flatdarks entering that formula?

When generating the masterFlat the calibration formula is

calibratedFlat = flat - (masterDarkFlat - masterBias) - masterBias

The formula for lights depends on which frames are provided, the most general one is the following:

calibratedLight = (light - (masterDark - masterBias) - masterBias) / masterFlat * avg(masterFlat)

Consider that WBPP generally expect masterDarks to contain the bias so we need to remove the masterBias from the masterDark when calibrating lights in order not to have a double-subtraction.
If you don't provide some masters then you should just remove them from the formula but you should be aware of what you're doing.
For example, if you don't provide a masterBias then

calibratedLight = (light - masterDark) / masterFlat * avg(masterFlat)

In this case your masterDark should absolutely contain the bias.

or if you don't provide flats then simply

calibratedLight = (light - masterDark)
etc...

[QUOTE]- optimize dark no, calibrate dark yes: masterlight is calculated correctly, same result as in old BPP[/QUOTE]
Here you should be aware of what calibration purpose is, this is the only configuration that compensates corerctly the signals you want to remove in order to extract the real target information. When you deal with ampglow the correct way is to take a dark with the exact same exposure to gather the same ampglow signal average to be subtracted from your lights. Optimization works only for isolated dark current signal.
Moreover, avoid to use the darks containing the ampglow to calibrate your flats otherwise the ampglow region will be overcorrected and will be visible as a glow in the calibrated lights.

[QUOTE]So I'd really like to know what are the used calculation formulas, and what is changed using the different switches.[/QUOTE]
- [B]dark optimization flag[/B]: by subtracting the masterBias from the masterDark you obtain dark current only, if masterDark duration dies not match your lights this signal can be scaled to generate a synthetic estimation of the dark current signal in yout lights to be subtracted. Keep in mind that you should not have ampglow in this process, this signal will be scaled wrongly resulting in a potentially visible ampglow halo in the final integrated image

- [B]calibrated darks[/B]: always leave this flag checked unless you're providing an already bias-subtracted master dark (this is a very rare case and in general you should not do that)

- [B]calibrate with flat darks only[/B]: this flat forces the WBPP to use a masterDark with the exact duration of your flats, if not then flats will not be dark subtracted. This helps if you don't have nether masterBias nor masterDark that matches the flat duration. In the latter case, with CMOS cameras, the masterDark provided could contain a significant ampglow wihch is not present in the flats so the usage of that dark must be avoided otherwise you will probably have the ampglow halo as a result in your calibrated lights.


I hope that I covered all your doubts, please let me know if you need more clarity on any topic!

Clear skies!
Robyx

 

[tommy_nawratil]

dear Robyx,

thank you, really glad you like my work, and thank you for the exhaustive explanations!

When one comes to use WBPP newly, it is a little confusing with the new options.
So its good to make the math behind the switches clear, thats the hard facts one needs to know to decide.

One thing:

Code:
calibratedLight = (light - (masterDark - masterBias) - masterBias) / masterFlat * avg(masterFlat)

This calibration formula is used when doing dark scaling, introducing the scaling factor k multiplied to (masterDark - masterBias),
so I understand WBPP uses the formula generally, with k=1 if no scaling, and no Flat pedestal?

Then, working with properly made masterdarks and masterFlatdarks, one should not choose "Optimize dark frames", and use that flag only if scaling of darks is needed.

"Calibrate master dark" - this was confusing, as I never calibrated masterDarks by subtracting masterBias and was frightened the script will do this. Its only needed when scaling the darks, and this is covered already by the previous flag "Optimize darks". But ok, its for the special case someone has a reason to provide masterDarks that are already Bias subtracted. I think 99,9% of WBPP users are happy to throw in their dark subs and let the mill run. Afterwards, they usually store the calculated masterdarks in a library and reuse it without modification.

With old BPP, I put my flatDarks into the Bias section instead of true Bias and it worked ok. I did the same with WBPP, and worked too. So, thats not too bad I think but what is your recommendation? ASI1600 true (shortest exposed) Bias value is known to go up sometimes after some ms of exposure and getting stable then only, so many people just avoid that and use Flatdarks instead.

Thank you for your explanations, you need to know if I understand it all other people also will ?

Tommy

 

[tommy_nawratil]

Weighting subframes is a real advantage, WBPP does great!
But I am not sure about subtracting Bias from Light and Darks,
especially when one uses CMOS cameras that sometimes have notoriously unstable Bias and Darks.
I had less concern if (unstable) Bias were subtracted from Flats only. But maybe, its equivalent math.

Tommy

 

[jpd321]

Not sure if this is the best place to pose this question, so please redirect me if needed. I have been using WBPP to calibrate/register images each night after a session, including integration to get an idea of stacked image quality. WBPPWGHT seems to be written OK to debayered images, but not to registered images (no FITS keywords at all as far as I can see.). I would like to be able to use the WBPPWGHT keyword for integration of multiple nights images, and ensure that the same reference frame is used for registration on subsequent nights. Am I missing something? How to get the WBPPWGHT information so that integration can use it? Thanks.