Trouble when using ImageCalibration

[Anducal]

Hello.

I'm new here, so a few words about my person. I appologize for my broken English (I'm from Germany).

About a year ago I started in astrophotography with a cooled CCD-Kamera (ATIK 383L+) with a small refractor. After some inevitable problems, I had to solve, I got some good results. One of the steps to improve my result was switching to PI. I used AstroArt for a while, which is a fine piece of Software, but sometimes gives me not enough possibilities to control the workflow. So I discovered PI and now I'm learning to use it - which is sometimes not easy but PI is definitive worth it.

My workflow is relative simple, I'm taking L(1x1) and rgb(2x2) frames, the processing is very similar to the NGC 1808 video tutorial from Vicent Peris. After some hard learning I get relative good results now, but there is one problem I can't get rid of, when calibrating my images.

First I integrate my darks and bias frames with the settings shown in the first image (nearly the same like in the according master calibration tutorial of Vicent Peris).

Next step is the calibration of the flats (made with ef flatfield foils) as shown in image 2. And here is the problem: About half of the frames to calibrate fail with the warning "No correlation between the master dark and target frames". The frames which succeeds have often very low dark scaling factors like 0.03 ... as far as I understand the process this value should not be too far from 1.0 because I use always darks with the same expose time and temperature (-30), as the according frames.

If I deactivate "optimize" there are no more problems, but I assume, the result is not optimal.

This warning was several times discussed in these forum, but none of the found reasons seems to match here. I think there is no double substraction of bias because of wrong parameters and also my darks, bias and flats look very normal in my humble opinion.

If I don't use flats I get the same problem when calibrating my lights with master bias and master dark.

I also tried this on windows (I'm using the Mac version), this didn't change anything. But I discovered, the results of PI running on OS X are not always exact the same like running on Win7, which seems a bit curious to me.

In the moment I really have no idea what I'm doing wrong, maybe, you can help me?

Regards from cloudy Bavaria ...

anducal

[pfile]

i wonder if with such a cold sensor (-30C) and such short flat-dark exposures, that the dark noise is simply too small for PI to actually properly see the noise. i would think that simply unchecking 'optimize' and 'calibrate' on the master dark should do the right thing.

in other words, perhaps it does not make sense to try to scale such a short dark...?

[Anducal]

Hm ... interessting point.

To verify this, I did an experimental session.

To get much more noise I cooled down the Atik only to +5.

I made bias frames; also  dark/light with 005 / 020 / 060 seconds.

When trying to calibrate the Lights I discovered the same problems when using the Files for 5 seconds, the warning was shown 3 times, the images without warning showing scaling factors which were very low (0.03 and smaller).

Doing the same thing with the images for 60 seconds, there were no warnings, scaling factors were much higher (about 0.3).

The curios thing: Using the 5 seconds Frames with the 60 second masterdark also worked, with medium scaling factors (0.08).

Activating or deactiviating the bias in the ImageCalibration dialog didn't change results significant.

Now I'm really confused about the meaning of this results.

[Cleon_Wells]

Set your Probe size to around 9 pixels and tell us what your MasterBias level reads, what is your unCalibrated MasterDark level and what is the level of one of your Flat subs.
I agree with pfile that you should only subtract your MasterBias from your Flat subs before integration and you shouldn't subtract the MasterDark because the read noise in your MasterDark is  much higher then the read noise in your Flat subs, Vicent has mention this point. I'm using a DSLR and think these basic ideas work for your Atik.
Cleon

[Anducal]

/*
Set your Probe size to around 9 pixels and tell us what your MasterBias level reads, what is your unCalibrated MasterDark level and what is the level of one of your Flat subs.
*/

I did some measurements with my session data (at -30.0):

Single Bias Frame: 0.0048
Single Dark 10 seconds: 0.0047
Single Dark 45 seconds: 0.0047


Masterbias: 0.0050
MasterDark 10 seconds: 0.0047
MasterDark 45 seconds: 0.0047

Single Flat, 10 seconds: approx 0.2854
MasterFlat (created with the mentioned warnings): approx. 0.2968
approx background value in my 45 seconds single lights:  0.0072

Hm this would mean, there is nearly no dark noise in the darkframes. If bias is substracted from the darks, there is nearly nothing left over, so a scaling does not really make sense and the warning can be ignored. Is this interpretation right?

[Cleon_Wells]

I don't have any answers, just more question..8>)
I noticed that your MasterBias level increased compared to the Bias sub level.
Your Flat level is around 25% of the chip clipping point, my Flats are around 10% of the camera clip point.
You could try Calibrating a Bias, Dark, Flat and Light sub and check the Dark Scaling Factors (for the hell of it).
I've attached a screen shoot of this questionable stunt, (for the hell of it).
Cleon
PS why are you creating 64bit files?

[Anducal]

// PS why are you creating 64bit files?

Well this is my personal "quality paranoia", very useless as I suppose.


// You could try Calibrating a Bias, Dark, Flat and Light sub and check the Dark Scaling Factors

I did so, you can see the results in the images. I use the parameters of your picture (no calibration).

[Anducal]

Here you can see the images, I used.

BTW:  Merry Christmas! :-)

[Cleon_Wells]

The 0.0 Dark Scaling Factor for the Flat subs looks correct, I would only Calibrate the Flat subs with the MasterBias.
The 0.165 Dark Scaling Factor for the 45sec lumi does not seem correct if the temp and times of the Darks and the lumi are the same.
I would recreate a 32bit MasterBias using maybe 20 subs for this test, I would use  Sigma Clipping Pixel Rejection instead of Winsorized Sigma, use a High and Low window setting so you get at most 0.1% Pixel Rejection, I have a far amount of noise in my _c.fit subs from the DSLR subs so I’ll test integrate 3 subs with the High setting at 10, adjusting the Low setting for the 0.1% or less and then do the opposite  for the High setting and use these High and Low settings for integration. Calibrate 10 32bit Darks and integrate these checking the Pixel Rejection  % the same as above. Calibrate your Flat subs 32bit, using only the MasterBias and integrate the same as above. After you’ve jump through these hoops you should be able to calibrate a Lumi with close to a 1.0 Dark Scaling Factor. Happy Holidays …. 8>)
It would also help to have some one like pfile or Sanders jump in here, for I am talking a little out of my back side when it comes to these problems with a CCD camera.
Cleon

[Cleon_Wells]

Correcting a couple of points in my last post, yes you can integrate as little as 3 images, but ImageIntegration needs all your subs for the mind boggling math to work its magic for Pixel Rejection to function properly. The best way, I think , to do this is with Region of Interest, reducing the processing time for test adjustments, I use From Preview, with a small 1000 by 700 size preview  window of one of the subs.
         You don’t have to increase the High and Low rejection windows to see the individual rejection percentages because it’s on each line in Process Console window, the last two numbers on the right in the Pixel Rejection counts.

For a better understanding of ImageIntegration check this link out (a little light reading from Juan) ….  8>)

http://pixinsight.com/doc/tools/ImageIntegration/ImageIntegration.html#pixel_rejection

Cleon

[Anducal]

I did serveral tries now with most of the rejections methods (also with no rejection), things even got worser, in most cases I get the warning.

Obviously there is a essential mistake in my approach, but I can't find it.

I have a small webspace, would it be helpfull, if I provide a view zip files with let's say 5 darks/bias/flats and a sample light frame? This makes of course only sense if you have a speedy internet access.

[Cleon_Wells]

Yes, I like to learn.
I did get some Ha images the other night with the T1i with a clip on filter at iso1600,120sec  and I'm having an interesting time with creating a MasterDark, surprise 8>),
I even tried on the Windows 7 computer and see as you have, different results from the Apple, this could be from the code translator, QT working slightly different between platforms.
Cleon

[Anducal]

Here is an excerpt of my last session, 10 flats, darks and bias frames. I have also included a part of my PI project. The for the sample data used processes are on the left side. When calibrating the flats with these data, you will see a few of the mentioned warnings.

Process "masterDark10second" creates the master dark frames used for calibrating the flat.
Process "masterBias" creates the used master bias frame
Process "flatsCalibration ..." calibrates the flats with the master frames (and generates many of these warnings).

The file is really huge (> 900 MB), you should it only download when you have fast internet access and a flat rate.

www.sol-et-luna.de/sample.zip

[Cleon_Wells]

Well I down loaded your files and see the exact same results, I can't explain the very low Dark Scaling factors on your lights. In my test I did not use the MasterDark when calibrating your Flat subs.
Cleon

[Anducal]

Nevertheless this helps me a lot, if you get the same results there seem to be a "real" problem not related to wrong using of PI.

Special thanks for your efforts in doing this.

I'll write a little eMail to PI support. If there are new results, I'll let you know.

Thanks again.