More Bias Frames makes things worse

[jeffweiss9]

I've always used just 10 bias frames and 10 dark frames.  However, I had believed that more bias frames and/or more dark frames should give improved results. However, when I actually did an experiment to test that, I found the exact opposite.  Maybe someone can explain this. 

As the image quality metric, I use the NoiseEvaluation and FWHMEccentricity scripts to generate sigmaN, FWHM(px) and Eccentricity of the 16-frame stacked image that results from the Batch PreProcessing script (in all cases I use the same 10 flats and 16 lights) with the following results for different numbers of bias and dark frames.

10 bias and 10 darks  from July 2014:   signmaN=1.675E-4, FWHM=2.184px, Ecc=0.3367
30 bias and 41 darks  from Mar 2015:                  1.798E-4,             2.201px,         0.3964
30 bias and 10 darks     "         "                           1.742E-4,             2.204px,         0.3969
10 bias and 10 darks     "         "                           1.657E-4,             2.181px,         0.3365

Thus I find using 10 bias rather than 30 bias (or 15 bias, not shown here) came out always better by these metrics even if I used year-old biases and darks instead of brand new ones, while the number of dark frames made no difference once I had 10. 
I'd sure like to understand this.  The camera is an FLI ML-8300 cooled to -30C.   Blinking the bias frames doesn't show any anomalous ones.

Thanks.
-Jeff
 

[Dimitris Platis]

It might be possible that the reason u are getting these ''odd'' results is that the way there are combined is less efficient at 30 frames than at 30 (u know what I mean).
Also, sometimes using metrics is not the best way to determine efficiency...(sometimes in certain conditions) especially with such low noise frames

[RickS]

Are you getting any drift in average bias value?  I have a couple of cameras that show significant bias drift over time - so much so that I need to use overscan calibration for best results.

Cheers,
Rick.

[mschuster]

Hi Jeff,

At least on my integrations, Noise/FWHM/Eccen depends on many factors (subpixel frame registration, interpolation algorithm, etc), and so are not very useful as calibration master quality measures. I think it is better to measure noise in calibration frames and masters directly. In addition to NoiseEvaluation, the script DarkBiasNoiseEstimator is a new option that does this, using pairs of inputs so to discount fixed pattern noise.

Mike

[jeffweiss9]

Thanks, Dimitris, Rick, and Mike-
There was some drift at the 0.5% level of the bias frames and I did run the DarkBiasNoiseEstimator script on pairs of bias frames at the beginning of each 10-frame section (see attached plot).   The integration option I used was Windsorized Sigma Clipping which which should be reasonable for the fixed number of 16 light frames and those stacks are my bottom-line indicator on quality and the NoiseEvaluation and FWHMEccentricity scripts are the only way I have to generate metrics on those.  Aren't those the metrics people use to evaluate the best ImageIntegration options, for example?   In any case, all three metrics usually tracked up or down together, giving me some confidence.  After running BPP a total of 10 different times with different sets of bias frames (same 10 darks, 16 lights), I still didn't get results consistent with the measured small drifts in the bias frames.

I got the best result by my metrics, by running the last 25 Bias frames where the drift was relatively constant.  But I got essentially the same result using the first 10 and last 10 bias frames (20 bias frames total), that you would think would include the worst of the bias from my plot. Almost equivalent by my metrics was to simply use the first 10 bias frames only or the last 10 frames only (10 bias frames total), but if I used the middle 10 frames only, I got poorer results, similar to using all 30 bias frames. 

So either my metrics are no good and perhaps more bias is always better, in which case I'm in trouble in evaluating ImageIntegration options and other raw image quality comparisons that I need to make, or I'm just not seeing any evidence whatsoever that more bias frames are helping me.

Either way, I have no evidence that taking more than 10 bias frames (or darks with less study) helps me at all.
-Jeff

[mschuster]

Hi Jeff,

You might try measuring star shape on subframes rather than integrations. Try measuring stars on subframes calibrated multiple times with different masters. Does tweaking masters affect star shape on subframes? My guess is that on average across lots of frames and lots of telescope setups it does not. Star shape is going to be determined mostly by seeing, tracking, focus, and wavefront aberrations. To get better star shape I think work needs to be spend on these items. On the other hand, noise in calibrated subframes and integrations is going to be a function of master quality.

Mike

[RickS]

Jeff,

Might be worth working out what that 0.5% means in terms of electrons.  In my case it was significantly more than the read noise so I thought it was worth addressing (with overscan calibration.)

Another more random suggestion: have you tried Superbias?

Cheers,
Rick.

[jeffweiss9]

I tried using the metrics FWHM, SNRWeight, Eccentricity and Noise (DN) from SubframeSelector operating on the 16 calibrated lights for each combination of bias and dark frames that I collected, paying most attention to Noise (DN). Doing this for all 11 calibration runs of BPP gave results that made a little more sense in that the lowest median noise on the calibrated lights from SubframeSelector was for using 30 Bias and 41 Dark frames, as well as the best median values for the other three metrics as well. But the differences quantitatively between that 'best' and the next 4 or 5 methods were very small. In fact, using 10 year-old bias frames and 10 year-old darks gave very close to the same results for all 4 metrics as the 30/41 frame 'best'.

I think I have to conclude that I can now see the benefits of more bias and dark frames on the quality of the calibrated lights but they appear to be very small and didn't really show up at all in the similar set of metrics measured on the cosmetically-corrected, registered and integrated image (the bottom line).

Unless I'm still missing something, I'll probably stick with 10 bias / 10 dark frames although a some extra bias frames doesn't cost much.

Thanks for the suggestions, but that seems to be my conclusion.
Cheers,
-Jeff

[RickS]

Jeff,

What was the target object and are you doing LRGB or narrowband?  Any noise contribution from the calibration masters is likely to be swamped by shot noise if you're doing LRGB.  These smaller gains are much more significant if you're chasing dim narrow band targets.

Cheers,
Rick.

[jeffweiss9]

Rick-
The target was NGC2403, so not dim and not NB as it was just LRGB, although it was quite a dark sky (SQM 21.75 !!).  All the calibration testing after the first two were just done with the luminosity data.   When I say small gains, the lowest median noise of the calibrated luminosity frames from SubframeSelector was 50.90 DN and the highest of the 11 different calibrations was 51.48 DN, so it was a very narrow range. 
-Jeff

[mschuster]

Hi Jeff,

Thanks for reporting back on your calibration tests.

Note that small dim structures (unresolved stars and galaxies) are a source of confusion when measuring noise in lights. Noise estimates are often inflated as a result and hence do not reflect the true improvement obtained with better masters and more integrated lights.

Note that the calibration process adds read noise to the lights. Say you are integrating 16 lights each calibrated with 10 frame masters. In this case the dominate source of read noise in the resulting integration is from the masters. Not so good. I think master frame count needs to be several times light integration count so that master read noise is sufficiently buried by the read noise from the lights.

Mike

Regarding my master read noise thinking, take a special case where calibration is just master dark subtraction, all frames are perfectly aligned, and integration is the average:

i = average(u - d) = average(u) - d,

where i is the integration, u is an uncalibrated frame, and d is the master dark.

The read noise in the result is the quadrature sum of the read noises in average(u) and in d. If d has fewer frames than average(u), then the read noise in d dominates.

[jeffweiss9]

Hi,Mike-
   Thanks, I agree that your equation roughly applies and your conclusion if a) the SubframeSelector and NoiseEvaluation noise estimates are inflated so they're not really sensitive to what I'm trying to measure and b) in the absence of shot noise in the lights which should normally dominate the read noise in them.  However, as I mentioned, these lights were taken in a particularly dark sky, SQM 21.75 (21.70 for hours), so maybe in the case of these lights where I tried my calibration experiments, the read noise was more significant, although probably not dominant.  In that case, I could understand your argument that the read noise of the darks was degrading the image and requires several times the number of darks as lights to avoid that happening. The CalculateSkyLimitedExposure script (ML8300 camera) came up with an Anstey limit (dark skies) of 111 sec so, if that is accurate, my 480sec exposures were only ~4x that limit. The standard readout model for 5% contribution of read noise, however, came out 250 sec so I only had ~2x that limit.  So I guess I should/could have been sensitive to the number of bias and darks.
   Oh well, I've probably exhausted my time and mental energy on this problem, so I better go out tomorrow night, if the forecast holds, and just get some more data. I will try further increasing the number of bias and darks, however, at some point.
Thanks for everyone's help.
Cheers,
-Jeff