Non-linearity of Canon DSLR Dark Frames

[IanL]

In case you missed it, Craig Stark has posted a new article here entitled "Profiling the Long-Exposure Performance of a Canon DSLR"

Link to original article:  http://www.cloudynights.com/item.php?item_id=2786

Link to forum where he provides further comment: http://www.cloudynights.com/ubbthreads/showflat.php/Number/5299843

PDF version of the article (with readable graphs):  http://www.stark-labs.com/craig/articles/assets/CanonLinearity.pdf

The thing that jumped out at me was where he says:

The upper-left plot shows the mean signal in each of the darks taken at each of the six durations.  You can see this has a rather curious shape to it.  This should be a straight line that increases the mean intensity (ADU) with time.  But, it’s clearly not. Up until about two minutes, the mean intensity in the dark decreases with time (more dark current photons yielding less output signal), at which point it performs an about-face and increases with time.  Clearly, something is happening to the signal.

My question is, what does this mean for Canon DSLR-toting users of PI.  I am guessing that the dark-scaling routines used by PI are not (necessarily) appropriate for use with Canon DSLR processing, since the dark signal does not increase linearly over time.  (The article identifies that the weird behaviour is presumably due to some on-chip/firmware processing going on prior to creation of the RAW file, but after two minutes or so it picks up again.)

Any thoughts on how this might, or might not, affect calibration stages of a Canon DSLR workflow?

[georg.viehoever]

Hi Ian,

that's interesting material. I am currently not sure what it means for the kind of processing done by the BatchPreprocessing script and related processes in PI. The dark scaling done by PI assumes that dark signal is proportional to bias+t*darkCurrent*gain. Craig's results suggest that this is not the case for CR2s -  he sees some variation in both bias and gain.

BTW, some research that I did in http://pixinsight.com/forum/index.php?topic=4161.msg29198#msg29198 and http://pixinsight.com/forum/index.php?topic=4086.msg29086#msg29086 also suggests that something strange is going on with dark currents. My original idea had been to find a better formula for modelling dark current in Canon cameras based on darks with differerent exposure times. In the end I gave up because I saw so many strange effects - and I was not sure if a more elaborate dark scaling based on this would actually give measurably better results.

Attached in screenshots 1+2 you see mean and variance of the ADU of 120 darks at different exposure times for my Canon EOS40D. I don't see a linear relationship for both of them (Craig sees  it at least for the variance). Screenshot 3 is based on a sample of 900 pixels, showing how some pixels (hot+warm) go to high values with exposure time, while some stay level or even decrease (which they should not based on physics).

Maybe I'll pick up work again when I have more time.

Georg

[Juan Conejero]

The dark scaling done by PI assumes that dark signal is proportional to bias+t*darkCurrent*gain.

Actually not. Our dark optimization routine makes no assumptions on the relations among the intensity of the dark signal and any physical acquisition condition or sensor characteristics, including linearity. Here is precisely the strength of our method: it is a purely numerical optimization algorithm.

[georg.viehoever]

The dark scaling done by PI assumes that dark signal is proportional to bias+t*darkCurrent*gain.

Actually not. Our dark optimization routine makes no assumptions on the relations among the intensity of the dark signal and any physical acquisition condition or sensor characteristics, including linearity. Here is precisely the strength of our method: it is a purely numerical optimization algorithm.

Hmm, I am not so sure. Here is what I believe to happen in PI, assuming calibration as shown in http://pixinsight.com/forum/index.php?action=dlattach;topic=2570.0;attach=3174;image :

The ImageCalibration process first subtracts the master bias from the light (which is wrong because the bias from the bias frames may be different from those of the lights), and then subtracts some scaled version of the bias subtracted MasterDark (again wrong, 2 times: the bias is not the true bias, and consequently just scaling this bias subtracted MasterDark is wrong as well). The only thing where PI is special is in determining the scaling factor via some noise based fitting. The assumptions about bias, gain and dark current being constant in bias+t*darkCurrent*gain are part of the ImageCalibration process, and Craig shows they are wrong.

Am I getting something wrong here?
Georg

[Juan Conejero]

Am I getting something wrong here?

Yes and no. From what you say it seems---or at least that's what I think most readers may understand---that subtracting a master bias is wrong. Actually, subtracting a master bias is a necessary step to remove the additive pedestal that exists in all science and reduction frames (lights, darks, flats), and is part of any standard data reduction procedure in astronomy. I'm sure you intended to mean something different, but we have to be careful when we write these things because they may lead to misinterpretations. So I don't think PixInsight's image calibration routines are wrong, but I always are ready to be corrected.

The assumptions about bias, gain and dark current being constant in bias+t*darkCurrent*gain are part of the ImageCalibration process

The assumption about a constant sensor gain (for a given data set) is indeed true: it is an inherent part of the data acquisition and reduction models, including CCD noise models. The assumption about constant dark signal does not exist in our dark scaling routines. However, it is evident that when we are subtracting a master dark frame from a set of light or flat frames we are implicitly saying that "this master dark frame represents the (scaled) dark signal in all of the frames being calibrated". The assumption about a constant bias pedestal is similar.

the bias from the bias frames may be different from those of the lights

Then the entire data reduction process is wrong, mainly because we cannot remove all additive components from the data. For example, this invalidates flat fielding, which must be a purely multiplicative process. In reality, things are not so bad, or making astrophotography with these cameras would be impossible. It is not the image calibration or data reduction processes which are wrong; it is that DSLR cameras can't provide pure linear raw data, as expected for strict astronomical data reduction procedures. Quoting Craig's article (which is really excellent by the way):

The Canon DSLRs do very well in astrophotography. But, they’re not designed
from the ground up for this. They’re designed for a different market and their engineers make
different choices as a result. Some of the choices impact how well the camera works for
astrophotography.

The only thing where PI is special is in determining the scaling factor via some noise based fitting

Actually, this makes an important difference IMO. And it will be even more important when we release the next version of our dark optimization routine (multipoint dark optimization).

[georg.viehoever]

...

the bias from the bias frames may be different from those of the lights

Then the entire data reduction process is wrong, mainly because we cannot remove all additive components from the data. ...it is that DSLR cameras can't provide pure linear raw data, as expected for strict astronomical data reduction procedures....

That is exactly the point: DSLRs do not behave the way the reduction process assumes they work. And even PI's special dark optimization procedure cannot compensate for this.

Having said this: the difference between the real and the assumed behaviour is not that great, and so calibration usually still produces good images. During my experiments I tried to produce more elaborate models from darks with different exposure times. I used weighted least squares, both linear, 2nd order and logarithmic models, but I never was able to beat the results from the "simple" linear model used by PI and others.

I am looking forward to the multipoint optimization in PI 1.8. Will it really produce better images than the current procedure with DSLRs? Can't wait to see it .

Georg

[Juan Conejero]

Will it really produce better images than the current procedure with DSLRs?

It should do very well for both DSLRs and CCDs. Currently we use a single scaling factor for dark frame optimization. With multipoint optimization we'll approximate a nonlinear function. This will fix the main problem that we have now: undercorrection of hot pixels. Hopefully it will also improve the accuracy of dark frame subtraction in all cases, especially in complex cases such as DSLR images.

[vicent_peris]

Hi,

In IC we have implemented a first version of dark signal scaling. It is completely linear in the sense that it does simply a scaling of the dark frame.

As you say, this has a limited application in the case of hot pixels because they don't have a linear behavior. But this doesn't mean that the dark scaling algorithm is not working properly nor producing good results.

This algorithm minimizes the noise of the whole image. Think that dark noise signal is not only present in the hot pixels we see in the images; it's a component of every pixel of a uncalibrated image. With this algorithm we are minimizing the noise contribution of the dark signal in the 99%+ pixels of the image; the other 1% are hot pixels which, at the end, their noise contribution is very small.

My recommendation is, if working with DSLR, to acquire dark frames of the same exposure time and temperature. Usually CCD sensors have very few hot pixels, so they can be easily rejected during image integration.

To correct hot pixels we can implement two different approaches:

- To model the hot pixel behavior of the camera. This can be done with sample image of varying exposure time and temperatures.
- Make a multipoint scaling of the dark frame, depending on the dark signal of each pixel. This doesn't require sample images, only the master dark frame.


Best regards,
Vicent.

[Kathy_MD]

A year ago (having read Stark's 2009 paper "Three Canon DSLRs Hit The Bench" where he tested the 450D),  I changed my processing methods.

I took bias frames out of the processing and added dark-flats.  So, now I have lights, darks, flats and dark flats.  This seems to amelioratete the scaling issues done internally buy the camera.

[Ignacio]

Let me add one more piece to the mysterious in-camera canon processing. A year back I did some testing on my canon 1000D, to characterize its noise/gain profile. I noticed then, that when exposing beyond 10 seconds, maximum ADU count would drop from 4095 (what you would expect from a 12-bit camera) to 3651. This implies that some "substraction" is taking place inside the camera, similar but different to what Craig is reporting in his paper for the 450D.

This was independently observed by http://www.fredmi.fr/en/rebel-xs/2-le-canon-eos-1000d.html

Intriguing,
Ignacio

[georg.viehoever]

We now have plenty of experimental evidence that Canon is cooking its RAW images in some way. I was wondering if someone has played with alternative firmware such as MagicLatern http://magiclantern.wikia.com/wiki/Magic_Lantern_Firmware_Wiki or CHDK http://chdk.wikia.com/wiki/CHDK. Maybe those firmware guys know what's happening behind the scenes.

Georg

[vicent_peris]

From a neophyte perspective... Could we have our own firmware for these cameras?


V.

[pfile]

We now have plenty of experimental evidence that Canon is cooking its RAW images in some way. I was wondering if someone has played with alternative firmware such as MagicLatern http://magiclantern.wikia.com/wiki/Magic_Lantern_Firmware_Wiki or CHDK http://chdk.wikia.com/wiki/CHDK. Maybe those firmware guys know what's happening behind the scenes.

Georg

i have played with both of those (ML = DSLR, CHDK = P&S). both of these seem to be "overlays" or "extensions" to the camera's native firmware. i'm not sure how much they can change the behavior of the firmware that's "underneath" everything.

but anything is possible i'm sure, within reason of course.