Master Dark Correlation Warning

[llpastro]

When running ImageCalibration, I occasionally get the message "Warning: No correlation between the master dark and target frames (channel 0)".  What is this telling me and is there something I can do to prevent it?  On my current image set I got this message for 31 out of 125 images.

Larry

[Alejandro Tombolini]

Hi Larry, when Dark frame optimization is enabled (Optimize box checked), Image Calibration is trying to establish a correlation between your Master Dark and your light in order to get the best SNR in your calibrated images. The messages "No correlation between master dark and target frame" means that it have not been achieved, and the Master Dark will be subtracted from the light as it is, and the calibrated images may remain with under corrected or over corrected hot pixels.
If you have dithered your images those possible hot pixels will be rejected during ImageIntegration.
If you have darks with the same duration and temperature than lights and disable Dark frame optimization option in Image Calibration tool (Optimize box unchecked), that means that the MasterDark will be subtracted from the light directly, not been previously scaled.
Saludos, Alejandro.

[pfile]

i would argue that if your dark duration is similar to the light duration, hot pixels are more likely to be over- or under- corrected by dark optimization, because they don't behave linearly like other pixels.

if you are, say, calibrating 300s lights with an 1800s dark and the optimization fails, the overcorrected hot pixels are the least of your problems; the dark current subtracted from the light is ~6x more than what actually appears in the light. this could very well 'destroy' the calibrated image.

it's hard to say what's wrong without seeing some of the images in question, but sometimes this will happen if you have very short lights which do not have very much dark signal.

for the lights that do not generate the warning, what are the dark scaling factors? if they are really small then maybe shorter darks are in order.

rob

[vicent_peris]

Hi Larry,

This warning means that IC didn't find any match between the dark pattern in your target image and the dark pattern of the master dark. If this happens only in some of your images, it's probably because your dark signal is close to be negligible in the target image. This usually happens when you calibrate flat frames, where the signal of the dark pattern can be negligible compared to the strong flat field signal (specially considering the usually short exposure time of the flat frame). Don't worry then: it simply means that your dark signal is so weak that you don't get any benefit by subtracting it from the target image - which is, in my opinion, what matters.

If this is happening to you in the light frame set, then it means that the dark pattern contribution is close to be negligible, so IC fails to find the pattern match in some of the light just because there is a higher noise contribution from other sources, like a higher sky background. I would need to know further data:

- Are those target images light or flat frames?
- The sensor of your camera.
- The exposure time.
- The median level of your master bias and the average level of your master dark (check them in Statistics).
- The "Optimization threshold" parameter value your are using in ImageCalibration.
- Please load the master dark and apply the following steps:

• Open PixelMath, put the below equation and apply it to the master dark:
  
  iif( $T > ( (med($T) + adev($T) * k) ) , $T , 0 )
  
  Before applying the equation, change k by the value you are using in the "Optimization threshold" parameter your are using in ImageCalibration (default is 3). This equation will leave in the image only the pixels above a threshold we are setting in sigmas with the k value. All the pixels below that threshold will be set to 0.
  
  
• Open Statistics and select the processed master dark to show its statistics.
  
  
• In Statistics, please be sure that the "Unclipped" checkbox is NOT activated.
  
  
• Please give me the "count (%)" and the "count (px)" values.
  
  

Please check also the entire image set in Blink to be sure that everything is right with those images where the dark frame scaling fails.

When the scaling factor is not found, the master dark is not subtracted from the target image. As I told, it usually simply means that  your dark signal is so weak that you don't get any benefit by subtracting it from the target image.


Best regards,
Vicent.

[llpastro]

Thanks all for the responses.  Vicent, per your request, the images are light frames, the camera is an SBIG ST-10, 600 second exposures, the dark is a master 600 sec dark.  I will have to get back to you about the other items when I have time to perform the operations you suggest.

By the way Vicent, I attended your first Tucson advanced workshop and enjoyed it very much.

Larry

[jkmorse]

Vicent,

I ran into this recently as well.  Typically I always have my darks match my light iteration times (600, 900 or 1200 seconds) and never need to use the optimization switch.  But I had some blooming in a recent Ha set of NGC6888 on my SBIG 6303E so I shot a series of 450 second lights.  I tried running with both my 600 and 900 second darks using the optimization switch.  Over half the images received the warning with the 900 sec dark and 1/3 got the warning using the 600 sec dark.  Hard to believe a 900 or 600 second dark signal is too weak to register.  Also, without an anti-blooming gate, the 6303 is linear across most of its range.  Seems pretty strange.

Btw, these iterations are being shot from a mountain in New Mexico with some of the best skies in the US (solid 2 on the Bortle scale).

In any event, I will also run the tests you suggest and get back to you.

Best,

Jim

[pfile]

When the scaling factor is not found, the master dark is not subtracted from the target image. As I told, it usually simply means that  your dark signal is so weak that you don't get any benefit by subtracting it from the target image.

thanks for clarifying that, i thought that was the case but given my awful memory when someone says something different i just assume that my memory failed me.

rob

[Alejandro Tombolini]

When the scaling factor is not found, the master dark is not subtracted from the target image. As I told, it usually simply means that  your dark signal is so weak that you don't get any benefit by subtracting it from the target image.

thanks for clarifying that, i thought that was the case but given my awful memory when someone says something different i just assume that my memory failed me.
rob

Ops, sorry Rob, I thought that it were subtracted as it was. Now is clear is not subtracted.

Saludos, Alejandro.

[jkmorse]

Vicent,

Per my note above and the notes of others, there seems to be glitch in the optimize routine for calibrating with darks.  I reported above a situation where I had issues when using 900 or 600 sec darks to calibrate a 300 second set of lights.  Yesterday I was calibrating a set of 900 second lights using 900 second darks.  I had accidently checked the optimize box and, as a result, well over half of the 40 subs showed the "no correlation" warning.  When I realized my error and reran with exactly the same image set and master dark, but this time without checking optimize, every sub calibrated fine with no warning. 

That seems like a bust to me.  Please let me know what other information you need to further evaluate.  I did check every sub with blink and they were all excellent so that's not the issue.

Best,

Jim

[vicent_peris]

Hi Jim,

Thank you for the experimentation. It would be of great help to have access to your full data set. Would you please upload it somewhere?


Best regards,
Vicent.

[jkmorse]

Vicent, I will upload it tonight.  Thanks!

[jkmorse]

Vicent,

Here is the link to the files: https://www.dropbox.com/sh/2q8zjmmnl5bhbi5/AADyLB1UjlsECkoZMWGLcAWqa?dl=0
 
I have included the raw fits files. as well as my master flat and master dark.  I used the default Optimize settings, namely Optimization Threshold of 3.0000, Optimization Window of 1024 and CFA Pattern Detection set to Detect CFA.

I am working on a recent new build desktop PC running Windows 10 and have the latest build of PI operating on my system, including all recent updates (Version 01.08.04.1195 Ripley). 

Let me know if you need any other data.

Best,

Jim