DSLR Dark Investigation - Puzzling Result

[IanL]

As part of an investigation in to the effectiveness (or otherwise) of DSLR dark frames, I have hit something of a brick wall and am in need of some advice regarding my methodology and some puzzling findings.  As ever, I expect I have overlooked something obvious or made a stupid mistake, but here goes:

I am trying to test different methods of selecting subframes to integrate in to a master dark. In one of the tests, my aim is to evaluate the Signal to Noise ratio of the set of darks and select a subset that gives the highest SNR in the resulting master dark.  (By signal I mean the dark current, and by noise I mean the combination of read noise and dark current noise).  N.B. I am aiming to use dark fame optimisation in PI, which does seem to work well, so the objective is to give the least noisy and best match to the true dark current in the camera as the starting condition for the scaling routine to work with.

- Step 1 was to calculate the mean of each dark subframe, which should be a reasonable proxy for signal since there are no 'features' in the dark to speak of.
- Step 2 was to calculate the standard deviation of each dark sub, again this should be a reasonable proxy for noise.
- Step 3 was to calculate the SNR estimate as SNR (frame) = 20 * LOG10 ( Mean (frame) / StdDev (frame) )

I took 183 dark subs at a variety of temperatures between 6C and 25C (non-temperature controlled Canon 500D), all for the same 600 second exposure and at ISO400.   These were plotted on the graph below:



The dark blue line is the mean of each subframe (plotted on the right hand axis in ADUs).  The red line is the standard deviation (plotted in ADUs against the left hand axis).  You'll note the usual problem of Canon Dark Frames, i.e. for most of the graph the Mean reduces as the Standard Deviation increases, when we'd expect both to increase. At the far right you'll notice that both the Mean and the Standard Deviation increase rapidly.  I'm not surprised by this as this replicates Craig Stark's experiments quite nicely - the camera reduces the appearance of dark current for less bright frames but then seems to switch to a different mode, creating a 'hockey stick' curve.  (Crag found this by increasing exposure times, whereas I have replicated it by increasing sensor temperatures for the same exposure time - logically they are equivalent as more dark current should accumulate for longer exposures and/or higher sensor temperatures).

My suspicion for the hockey stick is that there is some internal look-up table and the camera is using this in conjunction with measurements from the optical black area of the sensor to apply a scaling factor to all or some parts of the image.  At some point(s) the algorithm is using a different value in the table causing a reversal in the effect on mean vs. stddev. It's probably a bit more complex than that, but it feels sort of right.

As an aside, the EXIF temperature is only a very rough guide to this relationship, you'll get a better result by looking at means and standard deviations; there is not a as close correlation between standard deviation and EXIF temperature as you might assume. The EXIF temperature is a rough guide, but there is a lot of variation for frames with the same reported temperature and a lot of overlap in mean and standard deviation ranges for different temperatures.  I am assuming the temperature probe is not that representative of the average temperature of the whole exposure.

Finally I sorted the SNRs of each subframe (see Step 3) in descending order and plotted them as the green line (this is in dB, against the left hand axis again - bear in mind that we have both ADUs and dB on one axis here - clearly not the same thing, but conveniently the results are in a similar numeric range so I can save creating a second graph so long as you keep the units for each line in mind!)

- Step 4 was to calculate the expected SNR of a master dark integrated from n frames as SNR (master) = 20 * LOG10 ( SUM ( mean(frame 1) .. mean(frame n) ) / SQRT( SUM ( Stdev(frame 1)^2 .. Stdev(frame n)^2 )  )  )

I think this is the correct calculation, and it is plotted as the purple line on the graph (again in dB against the left axis). The curve is what I'd expect to see from integrating increasing numbers of frames - i.e. a rapid increase in SNR early on, followed by a tailing off in improvement as more and more frames are added.  What was interesting is the far right of the curve; you can see from the green line that there is a rapid drop-off in SNR for the individual subs at about 179 frames.  This is due to the fact that we are passing the 'uptick' on the hockey stick curve found by Stark. Integrating these frames slightly reduces the SNR (the purple line curves down at the end).

This was the aim of the method, i.e. to use a reliable measure on the dark frame subs to decide which are adding to the overall SNR of the master dark and which are not.  So far so good, but this is where I am coming unstuck.

I next integrated master darks using 30 frames, 60, 90, 120, 150, 178 and 183 (the most I had).  The integration method was as described here (http://pixinsight.com/tutorials/master-frames/index.html) which is the standard reference I think we all use?  I didn't get the results I expected, so I tried three methods to see if pixel rejection was causing an issue:

- Method 1 was as described in the tutorial, using Pixel Rejection (1) Winsorized Sigma Clipping (Low 4.0, High 3.0) and Range Clipping (0.0 Low and 0.98 High) with Normalization set to "Scale + zero offset".
- Method 2 was the same, except Normalization was set to "No normalization".
- Method 3 was the same, except I used "No rejection" to rule out any funnies being caused by the clipping algorithms.

(N.B. For the avoidance of doubt, I am not using Normalization at all under the Image Integration Section, only under the Pixel Rejection section).

I plotted the results of all three on the chart above, as diamond, circle, and triangle markers.  The results overlay each other and aren't that clear, so I have plotted them at a larger scale here:



As you can see the SNR of 30 frames is slightly higher than that of the best single dark sub (plotted at '1' on the X axis).  After this the SNR tails off a small amount as we add more frames, which is exactly the opposite of what one would expect from the purple estimate curve in the first graph (bear in mind this was created from the means and stddevs of the actual frames).  I did a couple of extra integrations and the peak does indeed seem to be at about 30 frames - it is rising at 15 frames and falling at 35 and 45 frames (I didn't plot these extra results).  The minor differences between the three pixel rejection methods seem reasonable in themselves since the best curve (red) is the one using the most robust method to eliminate outlying hot/cold pixels.

So the question is, why am I seeing the SNR getting slightly worse as I integrate more frames, when one would expect it to get significantly better?

Bear in mind the weird pre-processing of Canon RAWs shouldn't nullify improvements in SNR by adding more frames to the stack (unless the SNR of the frames is really low compared to the rest of the set as we see at frames 179+).

So either my theory is wrong (entirely possible), I've made a stupid mistake or incorrect assumption, or something else?  Perhaps the results are just not statistically significant enough to be reliable? Maybe using means and standard deviations as estimators of signal and noise in these darks is just way off the mark? Any suggestions gratefully received.

(Please note, I'm not asking for advice as to whether I should or should not use dark frames, bias frames or magic pixie dust to calibrate my DSLR images.  The problem is that I've read so much contradictory advice and conjecture that I want to prove what does or does not work in a robust way rather than guessing or hoping that person A knows more than person B.)

[Andres.Pozo]

I think the problem is that you are stacking frames of different temperatures and that the shorter stacks have the coldest (better SNR) frames. In short stacks you get an improvement in SNR when you add more frames but after a while the hotter frames (with higher noise) start to foul up the result and the SNR decreases.

[Ignacio]

Have you taken the 10-minute darks in a row? I think what you are seeing is the heating of the sensor, combined with canon's internal compensations (probably a function of temp and exposure duration). It would help if you add temp data to the plots. My guess is that a pedestal is substracted, as a funtion of temp. So at higher temps, a larger pedestal reduces what you call "signal".

The run away numbers at the far right, I can't explain. Maybe nonlinearity at high temps?

[astropixel]

This is all very helpful. So, regulated cooling ensures consistent in-camera application, which highlights the compounding nature of increasing sensor temperature in otherwise, uncooled cameras - the dark current thing is not just a function of increasing dark current, but how Canon deal with temperature variations. No smiley for light goes on

[IanL]

To be clear, I have already addressed these points in the question:

a) All the exposures are of the same length, 600s, and at the same ISO (400).  Therefore any variation in dark current and the corresponding dark current noise is due to difference in the average temperature of the sensor through the exposure and nothing else.  I did consider the possibility of poor technique (e.g. light leaking in to the camera, odd variations in temperature, etc.) and there were some clear outliers in the original set of dark frames which were eliminated, and I am confident that the set used here are comparable and correctly made.

b) The reported (EXIF) temperature from the in-camera probe is only a rough guide to that average temperature and it does not correlate well to the true temperature.  In the interests of simplicity I have not posted that data (but will when I finally write all this up fully), but adding temperature data will not help and in fact (having spent months on and off analysing it) it will actually confuse the issue.

c) The mean brightness of the image should directly correlate to the average temperature (assuming same exposure length and ISO/gain).  For a Canon DSLR this is not the case as the mean brightness is initially reduced by on-camera processing, but at some cut-off point it starts to increase again.  This is shown by the dark blue line.

d) The standard deviation of the image should also directly correlate to the average temperature, as noise increases with signal.  The red line shows that this is indeed the case.  Thus the red line is the best indicator of the average temperature, so if you were trying to temperature match Canon dark frames, you'd measure the standard deviations and group them by reference to that.  There is some degree of correlation between standard deviation and the reported EXIF temperature, but it is not a close correlation as the EXIF temperature is taken at the start of the exposure whereas the true average temperature of the sensor varies over the length of the exposure plus in response to a variety of environmental factors (within and external to the camera).

e) The data in the graph is sorted by an approximation of signal to noise (mean / standard deviation converted to dB), i.e. the green line on the first plot. It is not sorted by time, temperature or anything else.

f) Using the means and standard deviations, it is possible to calculate the expected SNR for 2 frames, 3, 4 ... 183 frames.  That is the purple curve.

What I would expect is that the test integrations I have made with 30, 60 .. 183 frames would show a similar curve (maybe not an exact match, but something approximating it).  Instead I am getting something completely different.

So again, I am not asking about the mysteries of the Canon DSLR, on camera processing or anything else.  I have some simple to derive numeric quantities (the means and standard deviations of each frame).  I also have a basic piece of image processing theory (noise increasing as the square root of the sums of the squares of noise in the individual images, i.e. sqrt(sum(standard deviations^2), and signal increasing as the sum of the signal, i.e. sum(means).

Thus signal goes up faster than noise when you stack.  To a point, it does not matter that I am stacking frames with a lower SNR to frames with a higher SNR, the theory (and the purple line calculated using my actual data and the theory) show that the SNR will still go up (unless you add much lower SNR frames, i.e. frames 179-183).

Yes, the Canon mangles the mean value of the frame will result in a lower per frame SNR than an unadulterated version would have, but again re-read the preceding paragraph.  It is simply not relevant to the question, the SNR should still go up as I integrate more frames.

This is incontrovertible, and it is why you stack light frames, why you stack flats to make a master, darks to make a master or bias to make a master.

But when I use the PI ImageIntegration process, I do not see that increase in SNR, instead I see a decrease.  Why?

[pfile]

how about the "Weights" setting in ImageIntegration? i don't see any mention of that in your writeup…

rob

[Carlos Milovic]

Hi Ian
what parameters are you using for reading the raws? Are you disabling the black point compensation?

[Phil Leigh]

The EXIF temp is very accurate but has nothing to do with the image sensor... That is NOT what it is measuring. It is measuring the  driver circuitry for the LiveView/video mode to prevent overheating. For a non cooled camera the actual imaging sensor maybe anything up to 10c higher than the EXIF temp... And for a properly cooled (high precision cold finger) camera the sensor may be up to 10c lower than the EXIF temp.

My extensive experiments with all of this have lead me to a conclusion which may not appeal to most of you but... If you can cool your camera WELL BELOW -10c then you can simply forget all about darks which is just as well because they do not work like CCD darks... With a non-cooled camera all bets are off frankly...
YMMV.

[pfile]

i may have been the one that started that rumor about the live view drivers; i had looked at a teardown a guy did and decided that the chip that was sensitive to the cold spray was an LVDS driver for the live view screen. however looking at it again later i realized that the chip in question is the driver for the little LCD on top of the camera.

it may or may not have an embedded temperature sensor… or perhaps the temp sensor is just nearby. at any rate it was nowhere near the imaging chip. it's probably just there as a generic internal temperature and may be what they read to decide to shut down live view, but who knows.

rob

[astropixel]

You make an important point, Ian. The opposite, intuitively should be the case, one would think. Format Explorer Settings?

[IanL]

RAW settings are as follows, and are the recommended ones as far as I am aware (i.e. no black point correction, etc):



The ImageIntegration settings are as follows, and again they are the ones recommended in Vicent's tutorial (http://pixinsight.com/tutorials/master-frames/index.html):



Note that I am not using any normalisation or weighting in the Image Integration settings, as recommended.  This is the correct setting if all the darks are the same exposure and the same temperature, etc. as they should all contribute equally to the average. Clearly mine are not the same temperature, but nonetheless it still isn't relevant to the anomaly I am seeing in the SNR of the integrated images (as previously noted, the SNR should still go up unless you have incredibly noisy frames - the last few from frames 179-183 are such and and so much noise that the SNR decreases, but the other 178 should increase SNR but don't).

As an aside (again!) I will explore using weights in further experiments, it might well be valuable to weight non-temperature matched dark frames by reference to their individual SNRs like you would for light frames and of course there are plenty of different methods to test in PI! That is only worth doing if I can resolve this anomaly though.

The only question mark in the integration settings was the best way to do pixel rejection.  Vicent's tutorial is for an old version of PI and the Normalization setting under rejection isn't covered, so I did three tests using Winsorized Sigma Clipping with "Scale + zero offset" normalization, "no normalization" and also no rejection at all (in which case the master darks should just be a straight average of the stack of frames with no monkeying about).  That is the second graph and you can see the difference between methods is minimal (but 'wrong' in all cases, as SNR reduces not increases).

I think I've done everything right and have spent 50 hours over the past months double checking, which is why I am so puzzled at the result.  It must be a dumb mistake or a bad assumption on my part, but I cannot figure out what.

Whilst we're on the topic, I don't know the location of the temperature sensor or its intended function, but here is a nice graph showing the means of my dark frames plotted against EXIF temperatures:



As you can see, we have the hockey stick curve (the axes are swapped, so this time it is running from the bottom of the graph up and to the left, and then reverses up and to the right).  All other things being equal you'd expect the camera to process frames with the same amount of dark current equally, but clearly for each reported EXIF temperature there is a significant variation and overlap with higher and lower temperatures, hence I think this confirms the uselesness of EXIF temperature as a measurement of the darks (or lights for that matter).

[IanL]

At the risk of taking this off the original topic, here is a bit more on the temperature and on-camera processing stuff:

I next grouped my frames by EXIF temperature and took the median of the means of each temperature set and plotted it:



You can see the hockey stick curve with increasing temperature (as stated above there is a correlation between mean brightness and EXIF temperature, but there is a lot of variation between individual frames, plus of couse we'd expect a straight line or curve from bottom left to top right not this hockey stick).  This matches Craig's original graph quite well:



What you will note is that there is a small uptick in my graph from 12-13C, then down again at 14C-17C, where it again makes a big uptick.

Since the standard deviation is a much better measure of the actual sensor temperature, I grouped by EXIF temperature, then plotted median standard deviations against median means to get this:



You can now see there are three upticks, a small one at the left, a slightly bigger one and then the really big one.  Now I admit this might be something of a stretch (probably I should group by standard deviations rather than EXIFs as the starting point), but it does look to me like there is some algorithm or look-up table that is creating a scaling factor that make increasingly large swings up and down as the true mean brightness of the image increases?

Finally I'll present my plot of darks grouped by EXIF temperature and plotted against their median standard deviations:



In this case you can see there is a sensible relationships between temperatures and standard deviations (both increase together).  Again this replicates what Craig Stark found, and thus I think Standard Deviation is the best way to group/match frames.  The curve is slightly odd, as the theory should give a curve shaped like this and mine is inverted:

[astropixel]

I was prompted to revisit Craig Stark's paper based on this thread and approach the problem from a slightly different perspective - new thread. Your graphs are very familiar.

[Ignacio]

Interesting. I am inclined to conclude the same as before. I think the plot showing median stddev vs temp is off (ie, not linear) due to differences between actual sensor temp and what EXIF reports. I speculate that the hockey stick brightness plot, is the result of how camera internal processing adjusts a negative pedestal, as a function of exif temp, ISO and exposure duration, to somehow compensate dark current brightness, and maybe free up some A/D converter space.

Now, if one acquires darks in the same way as lights (ie, same duration/ISO, and same rest period between shots), one can use EXIF temp as reference to match lights with darks, acknowledging that the actual temp is likely off, but by the same amount in both cases. In fact, this is what I do, and it works fine, as indicated by consistent correlations during dark optimization.

Ignacio

[Phil Leigh]

Having two super-cooled, "re-purposed" DSLR's from two different manufacturers (one based on an 1100D and one on a 600D) and having spoken to them at length I can confirm:

1) in a camera that is not cooled with a lapped cold finger/peltier device, the actual surface temperature of the CMOS sensor can be anything up to 10c HIGHER than that reported by EXIF
2) There is not a perfect linear relationship between EXIF temp and actual sensor temp (for both cooled and non-cooled cameras) -it varies by model and depends on the casing materials, airflow, power source (batteries generate heat) and other factors. The rate of change of both temps (up and down) is also different.
3) the EXIF temp sensor is on the main board (exact position varies according to model) and protects the video/readout circuitry which heats up during long exposures, protracted LiveView usage and Video shooting. Long exposures alone do not generate enough heat to trigger the protection mechanism but video and LiveView both can do, especially in high ambient temps.