Calibration question

[Geoff]

When I calibrate a group of files I notice the message
"Warning: no correlation between the master dark and the target frames"
What does this mean?
Also when I look at the fits header from the created dark master, I find that most of the information that was in the acquired dark frames has been lost--temperature, Observer, date, exposure etc
Geoff

[pfile]

do you have "optimize" checked on your dark? PI tries to minimize the noise in the target frame by scaling the dark. if the algorithm fails, you get that message.

sometimes this is caused by accidentally using a debayered dark, or trying to calibrate a debayered image. if you are not using an OSC, the cause can be that the dark noise is so low in both your light and dark that PI can't get a handle on it.

what kind of camera are you using?

[Geoff]

I'm using a QSI 540, which is a monochrome camera.
Geoff

[Geoff]

...and yes, I did have optimization checked.

[Juan Conejero]

"Warning: no correlation between the master dark and the target frames"
What does this mean?

This means that if your master dark frame is subtracted from the target frame (usually a flat or light frame), the result will always be worse than the original in SNR terms. "No correlation" means that there is no dark scaling factor > 0 useful to improve your target image by dark subtraction.

In our dark optimization algorithm, the dark 'signal' is treated as if it were random noise. Although we know that the thermal noise does not follow a random distribution, it does behave like random noise morphologically (i.e. pixel-to-pixel intensity variations distributed quite uniformly), and we use this property to apply a noise evaluation algorithm. The optimal dark scaling factor is the one that minimizes noise after dark subtraction.

[mschuster]

Hi Juan,

I have a question about correlation. Just for a sanity test, I calibrated a bias frame and a dark frame and received correlation values of nearly 0 and 1. I expected this (a KAF-8300).

I then tried the same experiment with a guider CCD (KAI-340). Rather than 0 and 1 I received correlation values of around 2 and 3 (for bias and dark). This was unexpected.

For both CCDs, bias and dark masters are integrations of 128 and 64 frames respectively. The darks for the 8300 were long (2400s) and short for the 340 (60s). The 8300 is cooled, the 340 is not and a lot noisier of course.

Thoughts?

Thanks,
Mike

[Juan Conejero]

Before saying anything, I need to take a look at the frames you have used. Could you upload them?

[mschuster]

Hi Juan, I uploaded them, but can't figure out how to share the folders, the share icon is disabled when I select a folder.

endor/mschuster/Calibration/QSI-683
endor/mschuster/Calibration/SBIG-ST-i

Both contain master bias, master dark, two bias frames and two dark frames.

Masters created normally (average with no normalization and weights 1, winsorized sigma clipping with no normalization, permissive sigma rejection limits).

Edit: I put a copy of everything in Forum Shared Files/mschuster/Calibration/

[Juan Conejero]

Hi Mike,

Something weird has happened with your masters because they are extremely weak. For example, these are the statistics for Dark_integration of QSI-683:

Median  = 0.0000001
Minimum = 0.0000001
MinPos  = 1061,902
Maximum = 0.0000149
MaxPos  = 2166,391

Similar values can be read for the four master frames. The dark frame optimization routine cannot work with these data.

[mschuster]

I measure the following means and medians on the masters:

QSI bias: mean 247, median 246
QSI dark: mean 271, median 267

SBIG bias: mean 1051, median 1045
SBIG dark: mean 1091, median 1066

Let me download what I uploaded and double check. I don't understand what is wrong.

Thanks,
Mike

[mschuster]

Juan, I don't understand why you are seeing those zero values. When I download I see expected values that match what I posted above. Here are statistics for QSI dark_integration.fit downloaded from both My Files and Forum Shared Files:

Mean
271.325
Median
266.750
AvgDev
8.346
StdDev
187.101
Variance
0.534
Minimum
244.672
MinPos
1061,1601
Maximum
64000.000
MaxPos
512,309

The values I see on the SBIG files also match what I expect and are not zero.

How are you generating your statistics?

[Juan Conejero]

Sorry, it was my fault. Yesterday I changed the default FITS input range to [0,65535] and I forgot to change it back to [0,1]. Now your files are correct ...

The problem with your SBIG-ST-i files is the master bias. It is way too noisy. If you simply calibratethe SBIG files with the master dark and no master bias, you get dark scaling factors of 1.0 for the dark frames and 0.7 for the bias frames. The latter value gives you a good hint: it turns out that the bias frames can be improved by subtracting most of the thermal noise from them!

The dark optimization routine works to minimize noise induced in the target frame by subtraction of the master dark frame. For this process to work correctly, the bias and dark masters must be of good quality---otherwise dark frame optimization makes no sense (actually, the whole calibration process makes no sense if the master frames are poor). In other words: ideally, the only source of noise should be thermal noise. Since the master bias is introducing a lot of read noise in the equation, the dark frame optimization routine has to under-correct for dark frame subtraction, in an attempt to compensate the excess of read noise. The result is a scaling factor much larger than one in this case.

I wouldn't say that our routine is not working well in this case, despite the surprising result. In fact, it is converging to a solution where noise is being minimized. Of course, the result is useless. However, I see nothing inherently wrong here; it is just a case where dark frame optimization is not applicable due to a poor master bias.

Thanks for posing an interesting example and data set.

[mschuster]

Hi Juan, here are the master integration gaussian noise estimates and average SNR increments:

QSI bias: 2.5e-5, 1.8
QSI dark: 6.5e-5, 1.9

SBIG bias: 3.5e-4, 1.08
SBIG dark: 5.1e-4, 1.08

The SBIG masters are 10x more noisy, but the bias/dark noise ratios seem similar and the means and medians seem reasonable.

Can you suggest how to get a better master bias? I am using 128 frames, any more than that seems silly but maybe not.

Edit: I wonder if increasing the dark exposure time would help? Here is my thinking: (dark_median - bias_median) is 5x bias sigma for QSI but only 0.6x bias sigma for SBIG. It appears there is not enough dark current to distinguish it from read noise. I will try increasing SBIG dark exposure enough to get a 5x bias sigma. Looking at measured dark current, this means about 500s exposures rather than 60s. Maybe this will solve the problem.

Thanks,
Mike

[Geoff]

"Warning: no correlation between the master dark and the target frames"
What does this mean?

This means that if your master dark frame is subtracted from the target frame (usually a flat or light frame), the result will always be worse than the original in SNR terms. "No correlation" means that there is no dark scaling factor > 0 useful to improve your target image by dark subtraction.

In our dark optimization algorithm, the dark 'signal' is treated as if it were random noise. Although we know that the thermal noise does not follow a random distribution, it does behave like random noise morphologically (i.e. pixel-to-pixel intensity variations distributed quite uniformly), and we use this property to apply a noise evaluation algorithm. The optimal dark scaling factor is the one that minimizes noise after dark subtraction.

It could have something to do with the light frames.  They were of the recent supernova in M95 and Mars was in close proximity, spreading light everywhere.  I din't get the problem with light frames of other objects
Geoff