Weighted Batch PreProcessing script issues with frame calibration

dchamberlain

Well-known member
Nov 24, 2017
58
3
Hello,

Hopefully, someone will be able to shed some light on this. I've been using WBPP off and on again and several times I find the calibration of the light frames is not correct. The resulting integrated image after calibration and registration looks horrible with the appearance the flat frames didn't do their job. To test if this was a script issue or frame issue, I took the very same light, dark, bias, and flat frames and ran them through the manual steps. In fact, I subsequently ran WBPP using the very same calibration masters I used in the manual process. I created a video file from Blink to demonstrate the differences between the two processing methods. One video is of the calibrated and registered frames from WBPP, and the other video is from the calibrated and registered frames using the manual method. The one from WBPP shows uneven exposure (flickering) while the one from the manual process is even (except toward the end as some clouds moved in). As I stated, both come from the same set of data but generate very different results. Link for videos: https://www.dropbox.com/sh/b42894qrm6se4wy/AAA0K_F15ybdDZR0v9g2ODBea?dl=0

Here is a screenshot of the integration parameters I use in the manual method:

IC parameters.jpg

I hope it is something I am doing wrong with WBPP, because at this point I will have to stick with the manual method.

Clear skies!

Dale
 

pfile

PTeam Member
Nov 23, 2009
5,625
105
those settings seem like they can't possibly work - the 3 Master boxes need to point explicitly to files, not folders.

unless the master dark was precalibrated, ticking optimize by itself will lead to bad calibration. if you are using a CMOS camera you should probably just skip the master bias and turn off optimization of the master dark.

rob
 

dchamberlain

Well-known member
Nov 24, 2017
58
3
those settings seem like they can't possibly work - the 3 Master boxes need to point explicitly to files, not folders.

unless the master dark was precalibrated, ticking optimize by itself will lead to bad calibration. if you are using a CMOS camera you should probably just skip the master bias and turn off optimization of the master dark.

rob
This was a screen print of the form I use. I fill in the actual filenames of the bias, darks and flat. The point here is there really isn't much to the settings I use, but it works which is more than I can say for WBPP. I cannot use WBPP in it's present form because of what it does to the calibration of my light frames and I don't see any settings in WBPP to control the calibration, other that setting the master flat, dark and bias. As I said earlier, I point WBPP to the very same calibration masters as I do in the manual calibration, except WBPP yields entirely different results.
 

pfile

PTeam Member
Nov 23, 2009
5,625
105
This was a screen print of the form I use. I fill in the actual filenames of the bias, darks and flat. The point here is there really isn't much to the settings I use, but it works which is more than I can say for WBPP. I cannot use WBPP in it's present form because of what it does to the calibration of my light frames and I don't see any settings in WBPP to control the calibration, other that setting the master flat, dark and bias. As I said earlier, I point WBPP to the very same calibration masters as I do in the manual calibration, except WBPP yields entirely different results.
well you just haven't given enough information to give much advice. WBPP works for lots of folks, so in the absence of what kind of camera you are using, how you took the darks and biases, what program you used to capture those frames, and what settings you used in WBPP it is very hard to help.
 

dchamberlain

Well-known member
Nov 24, 2017
58
3
well you just haven't given enough information to give much advice. WBPP works for lots of folks, so in the absence of what kind of camera you are using, how you took the darks and biases, what program you used to capture those frames, and what settings you used in WBPP it is very hard to help.
Ok, here goes.
Camera: ZWO ASI1600MM-Pro sensor temp 10C unity gain
Darks: 40x180 sec., Flats: 40 each for Ha, O-III and S-II (exposure times 1.81 sec, 1.41 sec., 1.74 sec respectively), Bias: 100. The flat frames were also calibrated with dark-flats of the same exposure lengths. All captured using Sequence Generator Pro 3. Calibrated and integrated into masters using PixInsight processes outside of WBPP (I used the same file naming convention as WBPP uses).

Here are the screenshots from WBPP:

WBPP 1.jpg
WBPP 2.jpg
WBPP 3.jpg
WBPP 4.jpg

This run of WBPP uses the very same lights, darks, flats, and bias frame files as I did with the manual calibration, registration, and integration. I have included in the Dropbox folder the Blink statistics for both sets of resulting calibrated and registered H-a frames. They do show differences in the numbers which I can only attribute to a difference in how WBPP calibrates. What I am curious to know is what those differences are because there are no exposed parameters for calibration I can see except what is in Global Options.

I look forward to your advice.

Thanks

Dale
 

pfile

PTeam Member
Nov 23, 2009
5,625
105
since this is a CMOS camera i would not tick "optimize dark frames" in either WBPP or ImageIntegration. of course this means that your lights and dark duration must match.

also i know some CMOS cameras have problems with bias frames - they behave strangely with 0 or very short length exposures and produce files which are not proper biases. so another tack is to remove the bias frames from the equation entirely. particularly if your darks have evidence of amp glow then then it's not possible to optimize the darks anyway and that obviates the need for separate bias frames.

you have flat darks but how did you use them to calibrate the flats? separately or with WBPP?
 

dchamberlain

Well-known member
Nov 24, 2017
58
3
I did the flat-darks calibration outside WBPP. I'll try again without the bias frames, but I get the warning message when I uncheck the "Calibrate dark frames". I should ignore this? I did notice I had dark frame calibration ticked off in my manual calibration run. Could this make such a big difference in the resulting calibrated frames between the two methods?

My dark and light frames are of the same duration, sensor temp and gain.


Thanks for your help!

Dale
 

pfile

PTeam Member
Nov 23, 2009
5,625
105
the dark portion of calibration can look like any of these three equations:

1. light - masterdark (no dark scaling, uncalibrated dark [bias signal is in the dark])
2. light - masterbias - calibrated masterdark (no bias signal in the dark, so it must be subtracted separately)
3. light - masterbias - (UN-calibrated masterdark - masterbias) [the bias signal would be subtracted twice if it were not first subtracted from the uncalibrated dark frame, as it is also present in the uncalibrated master dark frame.]

pixinsight is trying to use #3 when bias and darks are loaded. as the warning states, the standard workflow is to make uncalibrated master darks. this is because subtracting a bias master from dark subframes can lead to negative values in the result (which get clamped to 0s.) however, you might have made a calibrated dark separately by executing the (uncalibrated masterdark - masterbias) step in #3 yourself, which is OK and not prone to these 0 valued-pixels. if you've done this you need to tell PI that the master dark is pre-calibrated, hence that checkbox. otherwise the bias will be subtracted from the dark again, and that's an error.

if you never load any bias frames at all, and only load darks, then PI can't subtract a master bias from anything, and thus #3 degenerates to #1. and then the "calibrate master darks" checkbox can't and won't do anything no matter what state it is in, because there are no bias frames with which to do the calibration. or, maybe it generates an error later in the pipeline since it's checked but there's no bias frames to work with. not sure.

if you have bias frames loaded and a calibrated master dark loaded, then #2 occurs when the checkbox is unticked. if you untick the box but load an uncalibrated master dark, that leads to bad calibration, then the bias is subtracted twice as described above.

i'm not necessarily saying your bias frames are bad. i just know that certain CMOS cameras have problems. i haven't had a chance to look at the data you posted so it's entirely possible there is some other problem here.

rob

edit: i should add that there is a #4, which is:

4. light - masterbias - (UN-calibrated masterdark - masterbias)*ScalingFactor

ScalingFactor is 1.0 if "optimize dark frames" is unticked, and equation 4 is then equivalent to #3 above. if "optimize dark frames" is ticked, then PI computes a scaling factor for the dark by iteratively subtracting a test area of the dark from the same test area of the light and measuring the noise in the calibrated test patch. if it finds a scaling factor where the noise is mininized, it sets the scaling factor to that fraction and executes #4. if it can't find a noise minimum it gives a warning that there was "no correlation" between the dark and light frame and sets the ScalingFactor to 1.0.

note i've omitted the flattening step in all of these equations but what happens there is that the flat is scaled by some factor and divided into the light after application of one of the above equations. if the bias signal (offset) is not removed from the master flat the scaling goes wrong and then the flat overcorrects the light. so it's pretty important to calibrate the flats. when WBPP makes a master flat, it calibrates the flat subs and then integrates the calibrated flats into a master flat. so WBPP-created master flats are calibrated and WBPP-created master darks are uncalibrated, unless you check that box.
 
Last edited:

pfile

PTeam Member
Nov 23, 2009
5,625
105
also to answer your question directly, it really depends on how you made the master dark when you ran manually.

if the master dark was uncalibrated, then you can load only that master dark, make sure optimization is off, and run. this is what i was essentially suggesting to do since scaling CMOS darks can be problematic.

if the master dark was uncalibrated, and you wanted to use dark optimization, then you need to load a master bias and tick both "calibrate master dark" and "optimize dark"

if the master dark was uncalibrated, and you didn't want to use dark optimization but you also loaded a master bias, you need to tick "calibrate master dark".

if the master dark was pre-calibrated, then you need to load the master bias no matter what you do. at that point you're set up properly to use dark optimization or not use dark optimization.

rob
 

dchamberlain

Well-known member
Nov 24, 2017
58
3
Thanks, rob,

This is very helpful information. I ran everything through WBPP, with the same lights, darks, flat-darks and flats (individual frames not precalibrated masters), and NO bias frames. I unchecked the "optimize dark frames" and "calibrate dark frames" and gave it a run. Looks like everything calibrated correctly and I was able to combine the Ha, OIII and SII into a workable RGB image which looked great!

I know that sometime during the run of the script the "Use Master Dark" and "Use Master Flat" get checked by the script presumably after the script creates the master dark and flat. The "Calibrate master darks" also got checked during the run, but not by me from the start.

I'm happy now!

Thanks a bunch!

Dale
 
  • Like
Reactions: pfile

dchamberlain

Well-known member
Nov 24, 2017
58
3
As a follow-up, when I ran WBPP with the master darks previously, I had "Calibrate Darks" checked. My master dark had been previously calibrated outside of WBPP with the masterbias, and the masterbias was also included in the WBPP run. This yielded undesirable results, which if I understood your explanation above was because the bias was removed from the dark again.

So I removed the bias frames entirely and ran uncalibrated dark frames, uncalibrated flat frames with dark flats through WBPP with both optimize and calibrate darks unchecked and the result looked very good.

Dale
 
  • Like
Reactions: pfile