Warning: No correlation between the master dark and target frames (channel 0)

[Jevans]

Hi.

New user trying to figure this one out.

Dark scaling factors:
k0 = 0.000
** Warning: No correlation between the master dark and target frames (channel 0).
Gaussian noise estimates:
s0 = 1.400e-002, n0 = 0.832 (MRS)
** Warning: Overwriting already existing file.
Writing FITS image: 32-bit floating point, 1 channel(s), 1663x1252 pixels: done


It happens both during flat and light calibration. I did read many other threads on the same topic but cannot correct the problem. I have been calibrating images long enough to understand when I'm subtracting bias twice or not at all. I have tried every combination of options in Image calibration and the message continues. My master flat comes out with the bias injected and hot pixels removed or hot pixels injected in it and bias removed, but not both.

My master dark was a simple average integration and it still contains the bias.


Thanks in advance for any help

Jayme

[pfile]

were the masters generated with pixinsight?

maybe can you post the frames somewhere? it's hard to speculate without taking a look.

rob

[Andres.Pozo]

Have you tried to use the search function? There are several tens of messages about this warning.

[Jevans]

I did read many other posts on the same topic, none of which resolved the issue.

I also read other posts which cite the theory behind how this is supposed to work, others that indicate there may be problems with this particular CCD chip's low noise and still others that indicate that hot pixels remaining after calibration is a normal tradeoff. The problem here though is not incomplete hot pixel removal, but zero adu or no hot pixel removal - over or under correction.

I also simply integrated the master bias dark and flats and followed this "simplified" workflow posted by vicent peris http://pixinsight.com/forum/index.php?topic=2564.msg17291#msg17291 which produced the same dreadful results - zero adu images.

I also tried using the matched flat darks that I captured instead of scaling the master dark so that I could rule out the optimization as an issue and the hot pixels remain - turning optimization on or off had no effect.

I am unable to post the bias/dark images online, but I can say there is nothing wrong with any calibration frames... It is either the calibration process as done by the user or the software behind it. I have successfully calibrated other target images with the same dark/bias using Nebulosity, AIP4WIN and CCDSoft with no hot pixels remaining.

Both the bias and dark frames were acquired with a QSI 583 CCD which uses the KAF8300 chip. Both bias and dark are very high quality - made up of over 200 individual frames each which were measured for temperature fluctuations, cosmic ray hits and other anomalies. All frames were measured to be the correct temp (-20C) and all medians were consistent within a few ADU.

I have programmed and tested software for over 15 years. I don't pretend to understand image processing to that extent, but if the no correlation message was due to improperly subtracting the bias,  and I used the same, single bias frame to calibrate a single set of target frames in a single operation, one would think the error condition would apply to the entire set of target images. Yet, in some of the tests I have ran the last couple of days, the message only occurs randomly. In one case, it occurred 5 out of 16 times, implying that either the software is indeed struggling to find a solution or the console messages are incorrect.

Thanks

[mschuster]

Lacking some frames to look at, how about trying some sanity checks:

Run "light - masterDark" in PixelMath with the rescale result option turned off. Check the histogram of the result. Is it reasonable? There should not be a lot of zero pixels.

Try "masterDark - masterBias + 0.5" with the rescale result option turned off. The sensor has low dark current, the result should be near 0.5 everywhere plus/minus some noise. Check the histogram.

If this all looks reasonable, double check your ImageCalibration process setup. For the Master Dark, both calibrate and optimized options should be checked.

Mike

[pfile]

dropbox is free, skydrive is there, etc. etc. you should be able to upload them somewhere.

i also have an 8300M based camera and the dark current even at -30C is appreciable in my 1800s darks. it's only the newest sony sensors that have ridiculously low dark current. the only other thing i can think of is that your darks are really short.

what software was used to capture the images?

sometimes if the FITS header gives a bias pedestal keyword, the sign of the pedestal may confuse ImageCalibration. you can check that too. it could be that the bias is destroying the dark completely due to the large pedestal; so mike's suggestions are good because they will reveal that.

rob

[Jevans]

Hello again.

I ran these tests and everything was as expected... Histograms all had normal distributions. In pixel math light - dark resulted in a
normal image with no zero values... dark - bias + .5 resulted in an image whose values ranged from .43 to about .54 across the whole image - again as expected...

I did sign up for a dropbox account. Here are the links to the masters and a calibrated light frame. All images were captured with Nebulosity 2.5.2. The master dark still contains the bias.  Note the presence of hot pixels.

Bias - https://www.dropbox.com/s/icxi9h1wzmwnfgj/NG2903%20-20C%202x2%20MASTER%20BIAS.fit
dark - https://www.dropbox.com/s/5px1l97ytw5cvsy/NG2903%20-20C%202x2%20MASTER%20DARK.fit
flat - https://www.dropbox.com/s/jqzpukyd513xbdv/NGC%202903%20-20c2x2%20blue%20flat%202014_01_04.fit
light - https://www.dropbox.com/s/qb47z1gouzjd2km/pproc_NGC%202903%20-20c%202x2%205%20min%20Blue%202014_01_04_005_005345.fit

I am still open to the possibility that I am doing something wrong, but there are only so many options on the IC panel. The resultant light was produced with the dark frame's
calibrate and optimize boxes checked.

Thanks again in advance.

[pfile]

the levels in the master bias seem really high. compared to my master bias (which is bin 1x1 and -30C, but still...) the mean is about 6x greater; 6460 16-bit ADUs vs. 1073 for mine.

i'm going to guess that nebulosity has saved the captured image as 32-bit fits. if this turns out to be true, it is a problem because there is no standard for floating point fits files. pixinsight always saves f32 as [0.0-1.0] but for instance, maxim sometimes saves a file that should be i16 as [0.0-65535.0] in floating point numbers.

if the calibration subs came in as f32 from nebulosity then PI has rescaled them to the [0.0-1.0] range. PI has to guess and so often things go wrong. the right thing to do is save all of your subs as i16, since that's how they come off the camera and there's no ambiguity there.

let me know if i'm on to something here.

rob

[Jevans]

Hi Rob.

I think the inverse may be true. I just looked at my settings in Nebulosity and found that it is set to save in 16-bit int. Although if that's the case then I'm even more baffled, since the camera captures native 16-bit as you said.

Perhaps I should set that to float32, capture some lights, darks, biases and flats tonight and see if there's a difference? Hopefully there's a solution there. Thanks again

Jayme

[pfile]

no… i16 is the proper format for raw subs. so that's good - i would not change that.

so... i am surprised about the levels. i hate to keep asking you to post things, but how about a single bias frame, and maybe a 1x1 bias frame as well so i can compare directly with mine? perhaps one sub exposure for each type; bias, dark, flat, light.

in fact you should be able to calibrate one of your lights with single subs of all the calibration frames and see if you still get the 'no correlation' message.

thanks

rob

[Jevans]

If I run the bias frames I used to create the master through AIP4WINs measurement tool, the mean values are all consistent and exactly what I have seen from my ccd from day one - right around 370 ADU (16-bit), not the 6400 as measured by PI

These bias frames also load directly into Nebulosity, CCDSoft or AIP4WIN and calibrate with no hot pixels.

I will have some time tonight so I'll capture some 1x1 binned test frames and post those.

As one more sanity check, there are readout options was well as global preferences that need to be set. I have not changed any of those settings beyond the defaults. Under Readout options, "Normalized Real Range. Resolution" is set to .1. With no manual to explain the purpose of these settings I have no idea if this has anything to do with scaling.

Thanks

[mschuster]

Yes, the mean,  median and standard deviation of your master bias are all way too high. 20 times too high mean, 200 times too high noise. This is the cause of the no correlation error. Compare the histograms of the master bias and master dark. To the first approximation they should look the same.

Double check the histograms of a single bias and a single dark. They should be similar, similar peak value and similar width. Rerun the master bias integration. The histogram of the master bias should look like that of a single bias, a similar peak value, but narrower, with less noise.

Mike

[mschuster]

Also, on the hot pixel issue. I find dithering works well to eliminate them.
Mike

[Jevans]

Hi There. Clouds prevented the capture of any more light frames last night.

The problem is getting a bit clearer, but the root cause is not... Now I have even more questions. Single bias frames look fine statistically and with the histogram... The integration of the master bias does not...

I have checked individual bias subs with AIP4WIN, Nebulosity, CCDSoft and now Pixinsight and they are all fine...  I have also stacked the master bias and calibrated subs with these programs and they are still fine. The difference here is Pixinsight.

I have ran though all of the IC tutorials and settings... The bias integration is a straight average - no weighting, no normalization, but the results keep coming out the same. I have also attempted to rerun it without using the cache, without evaluating noise, etc. and the results are the same. Unless I am still missing something  the integration algorithm for the bias frame is broken. I just reran another integration with the same results.

[Juan Conejero]

Hi Jayme,

The only way we can help you is if you upload some raw frames. You have uploaded master frames and calibrated images, which are not useful to diagnose the problem because the damage (if any) has already been done and we are unable to trace it back. We'd need just one raw frame of each class: bias, dark, flat and light (assuming that all of them have been stored in the same format).