future fix BasicCCDParameters: flaw in calculation of dark current?

A

alessio.beltrame

New member
I recently used BasicCCDParameters (v0.3.1) to evaluate my CMOS monochrome camera (QHY163M) and I found suspicious results. I took a look at the source code and I found a portion of the code that is not sound to me. Let's start by saying that the sensor on my camera does black point correction, so the single dark frame method (first part of the <if> instruction - Line 421) does not apply.

In the <else> clause (lines 431-438) there is the following code:

var heat1_noise = Math.sqrt( dark1StdDev[c] * dark1StdDev[c] - bias1StdDev[c] * bias1StdDev[c] );
var heat2_noise = Math.sqrt( dark2StdDev[c] * dark2StdDev[c] - bias1StdDev[c] * bias1StdDev[c] );
// "n" units of additional read noise increase sigma(heat noise) by sqrt(n),
// so too measure the darkCurrent per second:
var sbq = heat2_noise / heat1_noise;
darkCurrent[c] = gain[c] * ( sbq * sbq ) / ( self.dark2ExposureTime - self.dark1ExposureTime );

Let's pretend that heat1_noise and heat2_noise actually represent the dark noise only (this is true as a first approximation only). The variable sbq is calculated as the ratio of dark noise in the two frames, so it's not "n units of additional read noise", whatever that remark means.  Instead of sbq^2, the formula should use a value that corresponds to how much the dark signal (= Dark current * exposure time) has increased in dark2 with respect to dark1.

It is also easy to prove that, for negligible values of read noise, the formula for dark current only depends on gain and exposure time, which is clearly wrong.

Finally, let's do a dimensional analysis of the dark current equation. We expect to get a result in electrons per second, however sbq is a unitless number, so sbq^2 is also unitless; sbq^2 is multiplied by gain (e-/ADU) and divided by the dark2ExposureTime - dark1ExposureTime (seconds), so the result is measured in e-/(ADU*s) instead of e-/s.

That's why I'm currently using different methods for the measurement of dark current, which provide completely different results. BTW, I can't find the formula used for the dark current in the referenced HAIP book, let alone a formal demonstration. The other calculations of the script - FWC, gain, dynamic range, etc. - seem to be correct to me.

Can somebody please confirm that this portion of the script is actually flawed or explain to me where my line of reasoning is wrong?

Thanks in advance
Alessio
 
It looks flawed to me.

I wrote my own:
https://pixinsight.com/forum/index.php?topic=8683.0
Pick up version 6 at the end of that thread.

Mark
 
Hi Alessio and Mark, and Georg,

The BasicCCDParameters script was released under GPL license. Can we include it in the official GitHub repository and update it as you are describing here?

Mark, can we release your script as well?
 
Thanks Mark,
I looked at your script and it's based on the same method I use in my manual calculation (with the exception that I use multiple darks at once to provide a single result, while your script lists the result for each couple of frames). I tried to run the script and it works well, even if my CMOS camera is actually monochrome (it's not a DSLR): the output results for the 4 channels are almost identical, as they should be. Last notice: I use Sequence Generator Pro to capture images and the FITS header uses EXPOSURE as the keyword for exposure time. It might be a good idea to check to check for that keyword in addition to EXPTIME.

I also appreciated the fact that your script does check for errors and for pixel clipping.

Very good work
 
Juan Conejero said:
Hi Alessio and Mark, and Georg,

The BasicCCDParameters script was released under GPL license. Can we include it in the official GitHub repository and update it as you are describing here?

Mark, can we release your script as well?
Click to expand...
I have no objection. I would also be fine with giving it a different license that allows anyone to use or modify it.
Georg
 
[quote author=georg.viehoever]
[quote author=Juan Conejero]
Hi Alessio and Mark, and Georg,

The BasicCCDParameters script was released under GPL license. Can we include it in the official GitHub repository and update it as you are describing here?

Mark, can we release your script as well?
[/quote]
I have no objection. I would also be fine with giving it a different license that allows anyone to use or modify it.
Georg
[/quote]

I'm also happy for mine to be released under a similar license that allows anyone to use or modify.

Mark
 
Hi
Is an updated version of this script available?
Pi 1.8.8-7 still shows 0.3.1 on my PC
Or is there another way to characterise my camera within PI?
TM
 
The flaw in the caclculation of dark current that Alessio Beltrame detected in 2017 is still present (lines 433 - 440) in script "BasicCCDParameters", v0.3.1 which is part of the current PixInsight distribution v1.8.9-2, build 1597. The other calculations of the script: FWC, gain, dynamic range, etc. seem to be correct.

Mark Shelley's script "DSLRSensorParameters", v0.0.7, see https://pixinsight.com/forum/index.php?threads/noise-estimation-in-dark-frames.10012/ , post #17 calculates correct values of dark current.

Bernd
 
Hi Bernd,

Confirmed. But, unfortunately, we have no time to deal with these issues present in unmaintained scripts now. If anyone is willing to contribute a code fix we'll be glad to include it in the next version. Can you *confirm* that the code snippet in this thread's initial post is correct and that by simply replacing the existing code in lines 431-438 with the ones posted here is sufficient?
 
Alessio Beltrame analyzed the code for calculating the dark current in the BasicCCDParameters script. Two methods are used in this script:

First method:
Dark_current = gain * (mean(Dark) – mean(Bias)) / dark_exposure_time

This method doesn't work at all when optical black level subtraction is performed which is the case for many regular digital cameras and some CMOS sensors.

Second method:
He showed the flawed part of the second method in the "BasicCCDParameter" script, but not a corrected version.
So unfortunately the answer to your question
Juan Conejero said:
Can you *confirm* that the code snippet in this thread's initial post is correct and that by simply replacing the existing code in lines 431-438 with the ones posted here is sufficient?
Click to expand...
is no.

He wrote that he was using a different calculation which was published by him in "CMOS CAMERAS - Dark current theory, measurement & image calibration", https://pixinsight.com/doc/docs/DC_tutorial/DC_tutorial.pdf , pp. 6 - 9:

DC_tutorial_p_8.png

with

DC_tutorial_p_7.png

and RON = read out noise.

In other words: a difference image (D1 - D2) must be generated, and the standard deviation of this difference image is used in the calculation of the dark current. His calculation results in correct values also in cases when optical black level subtraction is performed.

For me, the code of the BasicCCDParameters script is confusing, I don't understand this code. However, I am not a programmer...

Perhaps user skilled in programming can take a look. Besides Mark Shelley's script "DSLRSensorParameters" v0.0.7 ( https://pixinsight.com/forum/index.php?threads/noise-estimation-in-dark-frames.10012/ , post #17) is working well. In post #4 of this thread, Alessio Beltrame confirmed this. In post #6 of this thread, Mark affirmed your question whether his script can be released in PixInsight.

Bernd
 
Last edited:
You must log in or register to reply here.
Back
Top