Yes another: "** Warning: No correlation between the master dark and target...

[Phil Leigh]

I cAn only speak for Canon DSLR darks... But they definitely do NOT scale linearly. Craig Stark and other have investigated this in-depth. There is some non-defeatable internal processing that occurs in camera... And no it is not simply a bias subtraction... Which completely messes up the linearity of the dark current. This kicks in with any exposure over 10 seconds.

Canon have so far refused to explain what they are doing.

I have abandoned DSLR darks completely and rely on bias subtraction and dithering (oh and true cooling of the sensor to -25)... I have zero dark current. Makes life soooooo much easier.

[IanL]

I'd love to be able to abandon the whole idea of DSLR darks due to the uncertainty, but only having the one camera that I can't risk breaking, cooling or other such fun it not an option at the moment. My current efforts are trying to investigate the following:

- Is it possible to match a set of darks properly to produce a reliable master dark? You certainly can't rely on the EXIF temperature, since the mean brightness doesn't correlate well to it.  So you either have to match by reference to mean brightness (may not be a good indicator of mean temperature due to the camera processing), or noise (should be a good indicator of mean temperature as it isn't affected by the processing, but noise would need to correlate to mean brightness if it is to be a reliable matching criteria - that is what I am going to look at next).

- Is it possible to match an unscaled master dark to a light frame by reference to any of the above parameters?  That would avoid the need for scaling (though it is hard to make enough matching masters due to lack of temperature regulation).  I don't know if there is a way to correlate (say) noise in a background area of the light to noise in the individual darks used for the master to create a reliable match though.

- Does dark scaling work to overcome problems of matching master darks to lights?  This is a bit of an unknown.  Based on Stark it certainly seems clear that you cannot match a short (less than 2 min) dark to a longer exposure light, or vice versa as there is a hockey stick curve with the change of direction around 100-120 seconds.  My thought was that darks and lights with exposures greater than this might scale because the dark current on Stark's graphs does appear to be linear after 2 minutes, so it may be a case that having apples and apples allows the scaling to work.  Conversely my own tests suggest that there is still something non-linear going on at 10 minutes so maybe not.

Empirically, I am pretty sure my current master darks used with scaling more of a benefit than a hindrance.  Examining several images calibrated with and without master darks it is clear that there is a pattern on the scale of 4 - 6 pixels that the dark is successfully removing (and it is obvious that the pattern is not part of the signal from the target).  It may be that I am just getting lucky, but I'd like to prove it one way or the other if poss.

[Phil Leigh]

The EXIF temperature is not the actual temperature of the sensor - that's not its purpose. The actual sensor temp can be 7-8c higher  (non cooled camera) or 7-8c colder (correctly cooled camera). Also in a non-cooled camera the sensor temp will rise during a long exposure.

Having said that, I've had success in the past using darks that are matched by EXIF temp within 3-4c and are the same duration as the lights... just need to keep away from that "optimise" button :-)

Dark scaling seems to work well with CCD images (I've processed lots for other people). The other main benefit of dark scaling is that you do not need flat darks - but with a DSLR and a good light source you can shoot an excellent flat in less than one second so the dark current is negligible anyway.

Do you use dithering? - most folks think dithering is only good for hot/cold pixel removal but I find that dithering a DSLR removes larger scale pattern noise too.

To be honest my expertise (hah!) is with cooled "DSLR's" (CentralDS, JTW etc) and the reason why I went down that road was that after using 5 different DSLR's (all good, recent models) I could not get a convincingly low noise calibration result... whereas with the cooled cameras the whole problem goes away.

I think someone who uses stock DSLR's would probably be able to help better with this. I would investigate dithering if you don't already do it. Interestingly, my dither scale correlates to 7-8 pixels which seems to be in the same ballpark as your noise...

[Juan Conejero]

The "No correlation ..." message means exactly what it says: that there is no correlation between the image being calibrated and the master dark frame. More explicitly, this means that the master dark frame cannot be applied because the result would be more noisy than the original.

Our dark frame optimization routine does not need, and does not use, any information about any acquisition condition, including temperatures and exposure times. It is a purely numerical solution. It will always converge to the best solution in the sense of minimizing noise in the calibrated image, even if such best solution consists of not applying the dark frame at all, as happens when you see the "No correlation..." message.

Another common misunderstanding about our dark frame optimization routine is to think that it assumes or requires a linear relationship between dark current and exposure time, or between dark current and temperature. This is not the case. Our routine does not assume anything at all about the shape of the noise minimization function (that's why it doesn't depend on any physical property). The only requirement is that the minimized function have a global minimum, which our routine will find.

[MikeOates]

Juan,

Thank you so much for the definitive answer to this question which has appeared here many times causing so much confusion. That explains why, even with a number of subs all with the same acquisition details the odd one will give this message.

Harry also said "Look at your darks and Bias carefully please", well I am and I will post back with my findings, which will be very enlightening!

Regards,

Mike

[bitli]

The kind of error which can happen with bias/dark are:

• The obvious: selecting the wrong files for the bias or dark.
• Calibrating twice (calibrating the darks before they are integrated, then checking calibrate again, this is a sure cause for this message.
• Any inconsistency (like not the same orientation, incompatible format and the like). If you use another program than PI to create the masters you may encounter this type of issues. Also messing up with CFA may also create incompatible bias/dark/image.
• Just a bad (too noisy) bias or dark. The optimize process is quite sensitive to the noise (more than a blind scaling). Using the SuperBias module may help having a much better bias (depending on the characteristic of the CCD) which will reflect in having a better dark after calibration.
• I suspect that having negative values when the dark is bias-subtracted (either explicitly or implicitly by checking calibrate) could also cause problems (I am just investigating it). Try to subtract the bias from the dark and see if there are many zero values, because in that case it seems that the dark is useless.  This may happen if you have a low noise camera and relatively short exposure.

-- bitli