Trouble when using ImageCalibration

[mmirot]

you guys need to try Kai's excellent calibration and align script. It needs the right fits key words but if  you use MaxIM dl on a monochrome camera this already done

http://pixinsight.com/forum/index.php?topic=3638.0

 I never subtract darks from my flats unless the noise is high from a flat longer than a minute on a cooled camera. subtract a bias

Max

[Anducal]

This script produces unfortunately the same warnings ("No correlation ...") when calibrating Lights (or flats). Meanwhile I'm at bit perplexed ...

[Cleon_Wells]

Hi Max, you may be correct, but I'm on a quest to figure out the strange behavior when integrating my cal Dark subs.
Cleon

[Juan Conejero]

Hi Andreas,

First of all welcome and thank you so much for your nice words on PixInsight.

I am downloading your zip file right now, so I'll take a deep look at your data later. Just a few comments now.

About half of the frames to calibrate fail with the warning "No correlation between the master dark and target frames".

The dark frame optimization routine cannot 'fail', in the mathematical sense. It is a minimization procedure that will always converge to a solution where the noise is minimal in your calibrated frames after dark frame subtraction. By 'noise' we are referring to dark current here, under the assumption that the dark current (or 'dark signal') consists of uncorrelated pixel-to-pixel variations. When observed through large samples, such variations have the same properties as random Gaussian noise, so we can apply our standard (multiscale) noise estimation techniques to evaluate it.

The "no correlation..." warning message indicates that the noise (in the sense explained above) will always get worse in the calibrated frame if we subtract the master dark frame, so the optimal dark scaling factor is very close to zero. Typically this may be due to two main causes:

- The dark current is very low. For example, some SONY CCD chips have very low dark current that, when the CCD is properly cooled, may draw dark frames unnecessary. In your case this seems a plausible explanation, as you are getting the same behavior in a consistent way and you already have measured nearly zero signal levels in your dark frames (i.e., comparable to bias levels).

- Incorrect (double, or none) bias subtraction. From your post this option can be discarded.

If I deactivate "optimize" there are no more problems, but I assume, the result is not optimal.

Not at all necessarily. If your dark frames virtually contain no dark current data, then a zero dark optimization factor is the correct answer.

I also tried this on windows (I'm using the Mac version), this didn't change anything. But I discovered, the results of PI running on OS X are not always exact the same like running on Win7, which seems a bit curious to me.

This has already been reported by Cleon. It is very very strange but could be due to compiler differences (GCC on Linux/OSX vs Visual Studio on Windows). I can't reproduce this problem but hopefully I will be able to do so with your data. Thank you for uploading them by the way.

[Anducal]

Hi Juan.

Thanks a lot for your detailed response.

that, when the CCD is properly cooled, may draw dark frames unnecessary.

May be this is the reason. Last night I created for a new session darkframes with a exposure time of 25 Minutes (temp = -30). The averaged pixel values are not noteworthy higher as the values of short darks or even the bias frames.
BTW: The temperature sensor is not very accurate, in real it should be about -35 degrees celsius.

But in this case shouldn't this be a very common occurrence? I use no special gear but a mass market product from Atik with a Kodak 8300 sensor, which is one of the most used sensors in astronomy.

Should this really be the reason, it would mean I can simply ignore the warnings and be happy?

It is very very strange but could be due to compiler differences

Yes I think this is the most likely reason. I'm creating portable rendering applications with the Qt framework and had similar effects in floating-point calculations caused by compiler behaviors.
The differences, I've seen in PI were small, I assume there is no real effect on the final results.

but hopefully I will be able to do so with your data.

I'm VERY curious about your results :-)

[Juan Conejero]

Hi Andreas,

After checking your data (very well organized project BTW) I think everything is clear now. Open your master dark frame (masterDark45second.fit) and your master bias (masterBias.fit). Now let's perform a manual calibration by subtracting the master bias with the PixelMath tool. The PM expression is simply:

$T - masterBias

with the rescale result option disabled (so all negative values after subtraction will be truncated to zero). Apply this PixelMath instance to masterDark45second. Now open the Statistics tool and check the track view icon. These are the relevant statistical data:

Mean     2.66e-05
Median   0.00e+00
AvgDev   2.66e-05
StdDev   1.90e-04
Minimum  0.00e+00
Maximum  4.31e-01

where the zero median denotes an overwhelming abundance of truncated pixels. Characterizing this distribution from these numbers is difficult. If we look at the histogram, a 99.97% of all pixels are concentrated under a narrow peak of width 5x10^-4, a 90.7% of the pixels have values below 1x10^-4 (6.55 ADU), and a 79% have zero values. The rest are a few hot pixels, including of course the maximum shown above. Nearly the same results are obtained with the 10-second master dark.

So the only 'problem' is with your dark frames. The dark current signal gathered in these frames is so weak that they are practically equivalent to bias frames. The ImageCalibration tool works correctly: subtracting your master dark frame (after bias subtraction) does not improve your flat and light frames in SNR terms, so the reported dark frame scaling factors close to zero are correct. In simpler words, other than a few hot pixels, you have virtually no dark current in these short-exposure frames. So the best thing you can do is disabling dark frame optimization. In this way all hot pixels will be removed during calibration. Or if you have dithered your exposures, you can just ignore dark frames because all hot pixels will be rejected from the aligned frames during integration.

[Juan Conejero]

An additional note. I understand your 'quality paranoia' (I am rather obsessive-compulsive with these things ), but working in 64-bit format is really unnecessary for calibration. The raw data are 16-bit unsigned integers. The 32-bit floating point format is more than sufficient to store the calibrated data.

[Cleon_Wells]

For what it's worth, on my quest on integration of my 14 _c Dark subs, creating a MasterDark, I  see an opposite effect between the Mac and the Win7 computers in Pixel Rejection,   Sigma NoNorm and Winsorized Sigma NoNorm clipping, see attached image. All Calibration of this one light are done on my Mac with two different MasterDarks,  on the left from the Mac , on the right from the Win7 computer.
Cleon

[mmirot]

You might want to try the latest calibrate and align script. Makes life easier with these processes.
It needs bin, filter and exposure time key word entries in the fits header to work. MaxIm does this for monocameras automatically.  Nice description and directions on the script recently posted.

http://pixinsight.com/forum/index.php?topic=3654.0

Max

[Cleon_Wells]

Max, thanks for the link. Everything seems to work when Calibrating my lights as long as my MasterDark is integrate with unCalibrated Dark sub and check the MasterDark Cal box during light Calibration.
I'm trying to figure out why there's a difference between my Mac and the Win7 computer when working with my strange _c Dark subs/low level.
Cleon

[Anducal]

Hi Juan

Thanks a lot for your detailed analysis. Now I'm understanding the "problem" completely (I hope so). I'm out on business in the moment, but tomorrow I will also do this little experiment you described for getting a better insight into this.

Now I have to process all the sessions of the last weeks, which I paused because of this warning messages. This will be a really exhausting week for me :-)

Thanks again for your efforts and thanks also to cleon and everybody who helped me on this.

Andreas

BTW1: Ok ok, I'll use 32 bit now :-)
BTW2: Should you need further testing on the Mac/Win issue, let me know, I have both systems on my mac and can easily do this.