Dark Frame Optimization: Optimization Threshold

[bulrichl]

I searched in this forum for the term "Optimization Threshold" and found the thread

https://pixinsight.com/forum/index.php?topic=10123.0

The author of this thread got the message "Warning: No correlation between the master dark and target frames (channel 0)" and wanted to know what this means and how to prevent it. In Reply #3, Vicent Peris suggested to do a PixelMath operation on the MasterDark and then report the statistics to him. Sadly the author was not responsive, and the thread came to nothing.

I am unsure as well about the correct setting of this parameter in the ImageCalibration, section MasterDark. Also it is
unclear to me, whether the default setting (= 3, in units of sigma) is suitable for my configuration. So I performed the suggested PixelMath operation

Code: [Select]

iif($T > (med($T) + adev($T) * k), $T, 0)on one of my typical, not precalibrated MasterDarks and want to present the result here, hoping that an explanation will be given. The MasterDark is integrated from 40 dark frames of a DSLR (Canon EOS 600D), ISO 800, exposure 360 s, camera temperature 24 °C, ambient temperature 11.0 -> 9.4 °C in the course of the dark frame acquisition.

Vicent, if you read this, please answer my following questions:

1) For the data I supplied, what would be the correct setting of the optimization threshold?
2) Up to now I always left the optimization threshold at the default value of 3. Is it expected to get better results when I change this value?
3) Is the default setting of the size of the optimization window (1024 x 1024 px) suitable for my camera (w=5202 h=3465)?
4) How can I determine whether the quality of my MasterDark is sufficient (number of dark frames, constancy of temperature, ...)?

Bernd

[bulrichl]

OK, I re-read the text that appears when mouse-overing the input field 'Optimization thresold:' carefully again:

"Optimization threshold:
Lower bound for the set of dark frame optimization pixels, measured in sigma units from the median. This parameter defines the set of dark frame pixels that will be used to compute dark frame optimization factors adaptively. By restricting this set to relatively bright pixels, the optimization process can be more robust to readout noise present in the master bias and dark frames. Increase this parameter to remove more dark pixels from the optimization set."

and finally figured out what that means. In my words:
PixInsight computes median and standard deviation of the MasterDark. From the input value "Optimization threshold" (in sigma units) the corresponding value of Td0 is evaluated (a floating point number, range 0 - 1) so that (Median + Td0) defines the lower bound for the set of dark frame pixels that are used for the dark frame optimization. Td0 is outputted to the console as well as the number of pixels and the the number of pixels in %. So for dark frame optimization PixInsight uses only the pixels with intensity between (Median + Td0) and 1.

I calibrated one typical light frame with MasterDark and MasterBias and tried several input values for the Optimization threshold and the default setting of the size of the optimization window (1024 x 1024 px), in the Master Dark section, Optimize and Calibrate were checked. For my camera (Canon EOS 600D, 14 bit) the offset is 2048.

The MasterDark is integrated from 40 dark frames of a DSLR (Canon EOS 600D), ISO 800, exposure 360 s, camera temperature 24 °C, ambient temperature 11.0 -> 9.4 °C in the course of the dark frame acquisition:
count (%)   100.00000
count (px)  18024930
mean        2060.415
median      2047.600
stdDev      68.049
avgDev      37.687
MAD         26.550
minimum     1940.825
maximum     15321.250

The MasterBias is integrated from 600 bias frame of the same camera, ISO 800, exposure 1/4000 s:
count (%)   100.00000
count (px)  18024930
mean        2047.390
median      2047.847
stdDev      4.723
avgDev      3.721
MAD         3.108
minimum     2009.821
maximum     2125.879

The results are shown in the table and plot. k0 is the dark frame optimization coefficient applied by ImageCalibration.

I was astonished to see that the plot of k0 vs. Optimization Threshold [sigma] shows a sharp minimum just at the default value of Optimization Threshold = 3. Honestly, I don't know what to begin with these findings. My question: Is it wise to leave the default setting of the Optimization Threshold or is it better to set it so that k0 is maximized?

Bernd

[Warhen]

Bernd, "Warning: No correlation between the master dark and target frames (channel 0)" usually appears upon flat frame calibration. If that's the case, it's generally just dark subtraction being skipped, because noise analysis shows that it's not needed.

[bulrichl]

Hi Warren,

presumably you misinterpreted my text. I have no issue with this warning, and it has nothing to do with the calibration of flat frames. The calibration of the flat frames was done separately, before the light frame calibration - as suggested in https://www.pixinsight.com/tutorials/master-frames/ , BUT a MasterBias or a MasterDark-Flat was used for it, WITHOUT optimization. The dark frame optimization was used in the calibration of the light frames, and the warning "No correlation between the master dark and target frames (channel 0)" did not occur at all.

I refer to https://pixinsight.com/forum/index.php?topic=7365.msg49400#msg49400 (reply 2) where Juan Conejero on 2nd August 2014 gave a detailed explanation of optimization threshold. He wrote at that time:

"In the latest version of the ImageCalibration tool we have implemented a new feature that greatly improves dark frame optimization, especially when working with master dark frames of relatively poor quality. It is the optimization threshold parameter in the Master Dark section of the tool. This parameter is also available on the BatchPreprocessing script since version 1.37 (on the Darks tabsheet). Unfortunately, we released these new versions just before Summer vacations, so we haven't had the time to document them yet. I'm going to explain how to use this feature briefly here, but we'll publish a more detailed description with examples when possible...".

That was the starting point for my trials setting different input values for the optimization threshold in the light frame calibration and plot the resulting correlation coefficient k0 (see the plot in my second post).

My question is:
What is the crierion that tells me which optimization threshold yielded the best calibration result?

Bernd

[bulrichl]

This thread seems to remain a monologue. That's a pity, I would like to learn more about the thoughts of the developers on this topic.

Anyway, I re-preprocessed an image project and executed the calibration once with an optimization threshold (OT) of 3.0 sigma (the default value) and then with 0.1 sigma, which in my case had yielded the highest correlation coefficient k0 during dark frame optimization (see above). Comparing the calibrated images visually shows that there are less hot pixels with OT = 0.1. Here are the statistics (16-bit [0-65535], both 'Normalzed' and 'Unclipped' unchecked):

OT          3.0 sigma      0.1 sigma
            -> k0=0.851    -> k0=0.972
--------------------------------------
count (%)   99.99464       99.98250
count (px)  18023963       18021775
mean        276.601        274.955
median      260.545        259.373
stdDev      247.094        247.153
avgDev      71.897         72.025
MAD         49.165         49.255
minimum     0.099          0.012
maximum     20186.059      20194.270

(Total number of pixels: 18024930)

Mean and median of the calibrated light frame with OT = 0.1 are lower than with 3.0, but standard deviation is slightly higher. Please note the difference of pixel counts. The "missing" pixels are zero values, so an OT of 0.1 results in 3155 pixels with zero value, whereas an OT of 3.0 results in 967 pixels with zero value.

This result was surprising for me. Generally I want to avoid pixels with zero value, whereas hot pixels that remained after calibration can easily be removed by the CosmeticCorrection process. In order to find out how this correlates I calibrated one light frame with varying OTs. The result is shown in Image_1 (count [%] vs. OT in sigma units):

Pixel count runs through a maximum (= minimum zero values) at an OT of 3. No relevant change is visible below an OT of 0.2.

Finally I compared the end results after ImageCalibration (OT = 0.1 and 3.0 respectively), CosmeticCorrection, Debayer, StarAlignment and ImageIntegration processes. The image with an OT of 3.0 had a slightly higher reference noise reduction.

From these observations I conclude that the important criterion for the right choice of the OT must not be, to maximize the correlation coefficient k0, but instead to maximize the pixel count in the calibrated images. Therefore I advise against blindly follow recommendations to set OT = 0, like e. g. https://pixinsight.com/forum/index.php?topic=7485.msg50009#msg50009 or Charles Bracken: "The Deep-Sky Imaging Primer", 2nd ed., p. 100, although I found one example where an OT of 0 was reported to be favorable (camera: QSI 683ws-8):
https://www.cloudynights.com/topic/593623-deer-lick-group-at-f10-ongoing-acquisition-cc-please/

I found that I needed to set the Optimization Threshold of the Master Dark to 0, otherwise I clipped the black level, particularly on the R channel.

For my camera (Canon EOS 600D), the default value of 3.0 for the OT (PixInsight v1.8.5.1353) works well. Obviously this is different with different cameras, so I cannot generally recommend this setting - you will have to experiment.

Bernd

[Juan Conejero]

Hi Bernd,

Thank you for your detailed analysis and insights on this important topic, and sorry for being inattentive.

As you have pointed out, I described the purpose of the dark frame optimization threshold parameter in a forum post, back in 2014. As noted in that post, the main problem here is contamination of master dark frames with readout noise. This noise is additive and normally distributed, and acts like poison for our adaptive dark frame optimization algorithm. Basically, the problem is that the model this optimization algorithm is based on:

I = I₀ + k*D + B

is no longer valid if the master dark frame contains a significant amount of additive noise. Recall that in the above equation, I is the uncalibrated science frame, I₀ is the dark-and-bias-subtracted frame, k >= 0 is the dark frame optimization factor, D is the master dark frame, and B is the master bias frame. Strictly, this model requires D to contain pure dark current signal exclusively. Since our algorithm uses noise evaluation techniques, when D contains additive noise the computed optimization factor k will be too low, and hence will tend to undercorrect for dark current.

The optimization threshold feature is a brute force, but efficient, solution to this limitation. It simply discards very low pixels in the master dark frame before computing the optimization factor k. This is based on the assumption that additive noise happens at relatively low values. By setting all master dark frame pixels below the optimization threshold to zero (zero pixels will be ignored by the algorithm when computing statistical moments and noise estimates), we can remove most of the additive noise component. By doing this we also remove part of the true dark current signal, but if the threshold is reasonably set, the surviving master dark frame pixels are enough to calculate a good optimization factor. In practice, this only fails when the master dark frame is of very poor quality, but when this happens, the entire calibration process is typically meaningless anyway.

Your analysis is very useful because it shows that the noise thresholding technique implemented by our optimization threshold feature is quite sensitive. Your final conclusion is very interesting:

the important criterion for the right choice of the OT must not be, to maximize the correlation coefficient k0, but instead to maximize the pixel count in the calibrated images.

It is clearly true that trying to maximize the optimization factor makes no sense. After all, we are designing an algorithm precisely to compute k0, so trying to force it to have a minimum or maximum value is conceptually wrong, at least without some suitable a priori knowledge, which we obviously don't have here. Minimizing the number of zero pixels after calibration, as a way to "optimize the dark frame optimization" (sounds nice! ), had not occurred to me. This is something we'll have to think more thoroughly. Again, thank you for your elaborated review on this topic.