Yes another: "** Warning: No correlation between the master dark and target...

[MikeOates]

Hi,

Done the obvious, looked at past threads but none the wiser.

I am having a problem doing calibration using darks and I get the
"** Warning: No correlation between the master dark and target frames (channel 0)" error.

At first I thought it was the way each file was taken, the light from Sequence Generator Pro (SGP) and the dark made before with MaximDL in the way each writes the FITS file differently. So I took some darks using SGP but the error is still there.

Looking at the resulting image I can see that the dark is not cancelling out the brighter parts of the image which it should (I presume from CCD camera circuits around the edges). These are 20 min exposures, matched in both time and temperature.

Can anyone download the two files below and see what the problem may be. I get the same result if I also use master flat and Bias, so to rule out any issues with those I just tried the calibration with the just a dark (no it's not a master, but a single frame, but that should not make any difference) The FITS header looks ok to me, but then I am not sure what I am looking for.

Dark: http://endor.uv.es/files/data/public/10e8b7.php
Light: http://endor.uv.es/files/data/public/f50d62.php

Below is the console text:

ImageCalibration: Global context

Loading master calibration frames:
Loading calibration frame image:
G:/Astronomy/_Master
Storage/814/SGP/dark/-25c/1200s/dark_1200s_bin1_20140227/raw/dark_1200sec_1x1__frame1.fit
Reading FITS: 16-bit integers, 1 channel(s), 3388x2712 pixels: done

Calibration of 1 target frames:
* Using 1 worker threads

Calibrating target frame 1 of 1
Loading target frame:
G:/Astronomy/m101/20140225/lights/m101_1200s_1x1_ha_light-25c_20140225_01.fit
Reading FITS: 16-bit integers, 1 channel(s), 3388x2712 pixels: done

Writing output file:
G:/Astronomy/m101/20140225/cal2/m101_1200s_1x1_ha_light-25c_20140225_01_c.fit
Dark scaling factors:
k0 = 0.004
** Warning: No correlation between the master dark and target frames (channel 0).
Gaussian noise estimates:
s0 = 9.061e-004, n0 = 0.909 (MRS)
* File already exists, writing to:
G:/Astronomy/m101/20140225/cal2/m101_1200s_1x1_ha_light-25c_20140225_01_c_5.fit
Writing FITS image: 32-bit floating point, 1 channel(s), 3388x2712 pixels: done

===== ImageCalibration: 1 succeeded, 0 failed, 0 aborted, 0 skipped =====
5.017 s

Thank you,

Mike

[Phil Leigh]

Mike - I downloaded your files and get the same result. However, the dark calibration appears to proceed OK and the resulting calibrated file seems as it should (to me).

Once curious thing is that the FITS Keyword "EXPOSURE" has "1200" in the dark and "1200." (note the period at the end!)

Not sure if that might impact anything...?

[MikeOates]

Phil,

Thanks for checking that out for me. Yes at first glance it may seem ok, but if you use a master dark and more lights the lighter patches are very obvious (after a stretch).

Do a strong stretch, there is a patch middle top and a pair of patches on the sides top and bottom. It only becomes an issue with long exposures and when processing to bring out the faint stuff. These glows are normal. The problem with giving an error out like this, is that makes it look like it has not done the job properly.

Odd about the decimal point, (I did not spot that & it does not show in NASA's  Fv (FITS Viewer), I hope someone else can comment on that.

Regards,

Mike

[Phil Leigh]

Yes Mike - a cursory glance it looks OK but it is over correcting badly...
In Pi, darks need to be bias subtracted to work properly - did this dark have the bias removed?

[MikeOates]

Phil,

No, the dark is not bias corrected, I usually do that during calibration and set the option for the dark to calibrated with a bias selected. But this issue was there when I did that.

The workflow being:
Register all bias, darks and flats for masters (using the correct settings for each image type), then select these in the calibration tool and enable the dark and flat calibration.

I will look at this again when I get home after work, without PI in front of me I am getting confused. 

Mike

[Phil Leigh]

Hmmm....
I normally uses the BPP script (which works fine for me) but if doing things manually I create a master bias by integrating the bias frames then calibrate each dark frame with the master bias, then integrate the calibrated files to create a master dark (the Vicent Peris method I believe).

[bianry]

I had the same problem and it got resolved by not doing a scaling (optimization) of the darks.
Don't know if it applies here though....

regards

Mats

[MikeOates]

Mats,

I just tried un-selecting Optimize for the master dark and the problem went away, i.e. no error and the image looks fine.

I have done a full calibration with darks, bias & flats, with Optimize off for the darks and that also works, even using older darks from Maxim.

But this is just a fudge, there must be something a-miss somewhere.

Thanks,

Mike

[Phil Leigh]

Hmm... unchecking "optimise" certainly makes a massive difference!
I'd love to understand how this dark scaling is supposed to work. I would have thought it would be driven by the exposure duration and temperature FITS keywords and the distribution of noise in the bias calibrated master dark... ?

[Harry page]

Hi

The comment I made in the other topic     Look at your darks and Bias carefully please 

Hi

I downloaded some bias and darks and find the bias has more noise than the dark ( tiny amount more )

Scaling works by subtracting the bias from the dark leaving only the dark current this can then be scaled by simple maths

But in your case you end up with negative numbers and this causes the problem 

In your case I would not use darks at all just use the bias for everything

Regards

Harry

[Phil Leigh]

Yes but... this dark does not need to be scaled, it is the same duration as the light and so one would hope that checking optimise (or not) would make no difference... however it clearly makes a massive difference.

[Harry page]

Hi

If you check the optimise box Pi will attempt to scale the Dark ( weather it needs it or not ) and why would you have the optimise box checked if you have the correct length darks anyway  scaling is only for when your darks do not match your light and if your camera has a linear response i.e the dark current is measurable 

If you examine the darks and bias as I have suggested you will probably see there is no difference between them or as in the other case the bias had more noise than the darks causing negative numbers and the problems.

so if you check the box the following happens

1) it will subtract the bias from the dark ( say 1 min dark )    i.e if dark is 1000 adu and bias is 990 adu we will left with 10
2) if the 10 represents 1 min of dark current and you want to use it for a 2 min dark /light it will times it by 2 giving us a fig of 20

But if the bias is 1001 we end up with a -1 fig which does not work    8)

If using a dslr ( non linear response ) never use the optimise box 

Regards

Harry

[bianry]

Mats,

I just tried un-selecting Optimize for the master dark and the problem went away, i.e. no error and the image looks fine.

I have done a full calibration with darks, bias & flats, with Optimize off for the darks and that also works, even using older darks from Maxim.

But this is just a fudge, there must be something a-miss somewhere.

Thanks,

Mike

Glad to be of service Even if I only parroted what I have been told.
Still struggling to fully understand this calibration business. Not to mention noise. Have read more about noise and the theory of noise the last days than I thought humanly possible. Still not 100% clear to me. I must be denser than I would like to admit....

regards

Mats

[IanL]

If using a dslr ( non linear response ) never use the optimise box 

Is that right Harry, and if so why?  I haven't seen any detailed analysis that suggests optimisation of DSLR darks do not work, but if there is something I'd be interested to know.

As an aside, I've just plotted the mean intensity of several sets of 10 minute darks with matching EXIF temperatures. What I have found is that the mean intensity of the darks does not correlate that well to the reported EXIF temperature (which suggests that the temperature sensor reading taken at the start of the exposure does not accurately represent the mean temperature of the whole dark exposure). There is a lot of overlap whereby sets of frames with adjacent EXIF temperatures have a range of mean intensities that overlap the next highest and lowest temperatures.

Nonetheless the mean of the mean intensities for each temperature do seem to follow a pattern (I need to measure and graph a larger set to confirm it), and the mean intensity decreases with increasing temperature.  Craig Stark's work found something similar for exposures of up to two minutes or so (i.e. intensity decreasing with time), but that mean intensity increased after that.  I am finding that even at 10 minutes something fishy is still going on.  I also need to do some noise measurements and see how they correlate to EXIF temperatures and mean intensities as Stark said that dark current noise wasn't affected and should be a much better way to figure out which darks exposures have the same mean sensor temperatures.

Point being that it is hard to make matching darks to create a master, but if the dark current meddling is based on measuring the optical black (again as suggested by Craig Stark) might one assume that a light frame of the same exposure and mean sensor temperature gets the same treatment by the camera?  Or is it more complex than that?  How would one find out?

Anyway, back to my main question. As I understood it, the dark scaling routine uses a noise evaluation process to find the best linear scaling factor for the dark frame.  If the camera processing is a purely linear function for the whole frame, surely dark frame optimisation should still work for DSLRs.  If the processing is non-linear then it might not.

I can certainly get successful correlation/scaling for my DSLR darks if they are a reasonably large stack (maybe 30 frames).  Interested to hear your thoughts.

[Harry page]

Hi

I find that dslr cameras gain different amount of dark current due mostly to the fact that they are not cooled ( and regulated )

I hear ( because I do not use one) have to collect darks at different temps and lengths and have found in practice with data sent to me that scaling did not work well

Good scaling requires a linear effect , but if you find other wise who am I to disagree 

 thanks for the input

Harry