PixInsight Forum

PixInsight => General => Topic started by: MikeOates on 2014 February 28 02:26:01

Title: Yes another: "** Warning: No correlation between the master dark and target...
Post by: MikeOates on 2014 February 28 02:26:01
Hi,

Done the obvious, looked at past threads but none the wiser.

I am having a problem doing calibration using darks and I get the
"** Warning: No correlation between the master dark and target frames (channel 0)" error.

At first I thought it was the way each file was taken, the light from Sequence Generator Pro (SGP) and the dark made before with MaximDL in the way each writes the FITS file differently. So I took some darks using SGP but the error is still there.

Looking at the resulting image I can see that the dark is not cancelling out the brighter parts of the image which it should (I presume from CCD camera circuits around the edges). These are 20 min exposures, matched in both time and temperature.

Can anyone download the two files below and see what the problem may be. I get the same result if I also use master flat and Bias, so to rule out any issues with those I just tried the calibration with the just a dark (no it's not a master, but a single frame, but that should not make any difference) The FITS header looks ok to me, but then I am not sure what I am looking for.

Dark: http://endor.uv.es/files/data/public/10e8b7.php (http://endor.uv.es/files/data/public/10e8b7.php)
Light: http://endor.uv.es/files/data/public/f50d62.php (http://endor.uv.es/files/data/public/f50d62.php)

Below is the console text:
Quote
ImageCalibration: Global context

Loading master calibration frames:
Loading calibration frame image:
G:/Astronomy/_Master
Storage/814/SGP/dark/-25c/1200s/dark_1200s_bin1_20140227/raw/dark_1200sec_1x1__frame1.fit
Reading FITS: 16-bit integers, 1 channel(s), 3388x2712 pixels: done

Calibration of 1 target frames:
* Using 1 worker threads

Calibrating target frame 1 of 1
Loading target frame:
G:/Astronomy/m101/20140225/lights/m101_1200s_1x1_ha_light-25c_20140225_01.fit
Reading FITS: 16-bit integers, 1 channel(s), 3388x2712 pixels: done

Writing output file:
G:/Astronomy/m101/20140225/cal2/m101_1200s_1x1_ha_light-25c_20140225_01_c.fit
Dark scaling factors:
k0 = 0.004
** Warning: No correlation between the master dark and target frames (channel 0).
Gaussian noise estimates:
s0 = 9.061e-004, n0 = 0.909 (MRS)
* File already exists, writing to:
G:/Astronomy/m101/20140225/cal2/m101_1200s_1x1_ha_light-25c_20140225_01_c_5.fit
Writing FITS image: 32-bit floating point, 1 channel(s), 3388x2712 pixels: done

===== ImageCalibration: 1 succeeded, 0 failed, 0 aborted, 0 skipped =====
5.017 s

Thank you,

Mike
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: Phil Leigh on 2014 February 28 03:44:07
Mike - I downloaded your files and get the same result. However, the dark calibration appears to proceed OK and the resulting calibrated file seems as it should (to me).

Once curious thing is that the FITS Keyword "EXPOSURE" has "1200" in the dark and "1200." (note the period at the end!)

Not sure if that might impact anything...?
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: MikeOates on 2014 February 28 04:11:08
Phil,

Thanks for checking that out for me. Yes at first glance it may seem ok, but if you use a master dark and more lights the lighter patches are very obvious (after a stretch).

Do a strong stretch, there is a patch middle top and a pair of patches on the sides top and bottom. It only becomes an issue with long exposures and when processing to bring out the faint stuff. These glows are normal. The problem with giving an error out like this, is that makes it look like it has not done the job properly.

Odd about the decimal point, (I did not spot that & it does not show in NASA's  Fv (FITS Viewer), I hope someone else can comment on that.

Regards,

Mike
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: Phil Leigh on 2014 February 28 04:13:11
Yes Mike - a cursory glance it looks OK but it is over correcting badly...
In Pi, darks need to be bias subtracted to work properly - did this dark have the bias removed?
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: MikeOates on 2014 February 28 04:25:49
Phil,

No, the dark is not bias corrected, I usually do that during calibration and set the option for the dark to calibrated with a bias selected. But this issue was there when I did that.

The workflow being:
Register all bias, darks and flats for masters (using the correct settings for each image type), then select these in the calibration tool and enable the dark and flat calibration.

I will look at this again when I get home after work, without PI in front of me I am getting confused.  ???

Mike
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: Phil Leigh on 2014 February 28 04:35:06
Hmmm....
I normally uses the BPP script (which works fine for me) but if doing things manually I create a master bias by integrating the bias frames then calibrate each dark frame with the master bias, then integrate the calibrated files to create a master dark (the Vicent Peris method I believe).

Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: bianry on 2014 February 28 05:12:14
I had the same problem and it got resolved by not doing a scaling (optimization) of the darks.
Don't know if it applies here though....

regards

Mats
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: MikeOates on 2014 February 28 07:02:17
Mats,

I just tried un-selecting Optimize for the master dark and the problem went away, i.e. no error and the image looks fine.

I have done a full calibration with darks, bias & flats, with Optimize off for the darks and that also works, even using older darks from Maxim.

But this is just a fudge, there must be something a-miss somewhere.

Thanks,

Mike
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: Phil Leigh on 2014 February 28 07:24:42
Hmm... unchecking "optimise" certainly makes a massive difference!
I'd love to understand how this dark scaling is supposed to work. I would have thought it would be driven by the exposure duration and temperature FITS keywords and the distribution of noise in the bias calibrated master dark... ?
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: Harry page on 2014 February 28 07:28:11
Hi

The comment I made in the other topic  :P   Look at your darks and Bias carefully please  ;D :footinmouth:

Hi

I downloaded some bias and darks and find the bias has more noise than the dark ( tiny amount more )

Scaling works by subtracting the bias from the dark leaving only the dark current this can then be scaled by simple maths

But in your case you end up with negative numbers and this causes the problem  ;)

In your case I would not use darks at all just use the bias for everything

Regards

Harry
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: Phil Leigh on 2014 February 28 07:41:03
Yes but... this dark does not need to be scaled, it is the same duration as the light and so one would hope that checking optimise (or not) would make no difference... however it clearly makes a massive difference.
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: Harry page on 2014 February 28 07:57:26
Hi

If you check the optimise box Pi will attempt to scale the Dark ( weather it needs it or not ) and why would you have the optimise box checked if you have the correct length darks anyway  :sad: scaling is only for when your darks do not match your light and if your camera has a linear response i.e the dark current is measurable  :-*

If you examine the darks and bias as I have suggested you will probably see there is no difference between them or as in the other case the bias had more noise than the darks causing negative numbers and the problems.

so if you check the box the following happens

1) it will subtract the bias from the dark ( say 1 min dark )    i.e if dark is 1000 adu and bias is 990 adu we will left with 10
2) if the 10 represents 1 min of dark current and you want to use it for a 2 min dark /light it will times it by 2 giving us a fig of 20

But if the bias is 1001 we end up with a -1 fig which does not work    8)

If using a dslr ( non linear response ) never use the optimise box  :o

Regards

Harry
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: bianry on 2014 February 28 08:07:55
Mats,

I just tried un-selecting Optimize for the master dark and the problem went away, i.e. no error and the image looks fine.

I have done a full calibration with darks, bias & flats, with Optimize off for the darks and that also works, even using older darks from Maxim.

But this is just a fudge, there must be something a-miss somewhere.

Thanks,

Mike

Glad to be of service :) Even if I only parroted what I have been told.
Still struggling to fully understand this calibration business. Not to mention noise. Have read more about noise and the theory of noise the last days than I thought humanly possible. Still not 100% clear to me. I must be denser than I would like to admit....

regards

Mats
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: IanL on 2014 February 28 09:02:20
If using a dslr ( non linear response ) never use the optimise box  :o

Is that right Harry, and if so why?  I haven't seen any detailed analysis that suggests optimisation of DSLR darks do not work, but if there is something I'd be interested to know.

As an aside, I've just plotted the mean intensity of several sets of 10 minute darks with matching EXIF temperatures. What I have found is that the mean intensity of the darks does not correlate that well to the reported EXIF temperature (which suggests that the temperature sensor reading taken at the start of the exposure does not accurately represent the mean temperature of the whole dark exposure). There is a lot of overlap whereby sets of frames with adjacent EXIF temperatures have a range of mean intensities that overlap the next highest and lowest temperatures.

Nonetheless the mean of the mean intensities for each temperature do seem to follow a pattern (I need to measure and graph a larger set to confirm it), and the mean intensity decreases with increasing temperature.  Craig Stark's work found something similar for exposures of up to two minutes or so (i.e. intensity decreasing with time), but that mean intensity increased after that.  I am finding that even at 10 minutes something fishy is still going on.  I also need to do some noise measurements and see how they correlate to EXIF temperatures and mean intensities as Stark said that dark current noise wasn't affected and should be a much better way to figure out which darks exposures have the same mean sensor temperatures.

Point being that it is hard to make matching darks to create a master, but if the dark current meddling is based on measuring the optical black (again as suggested by Craig Stark) might one assume that a light frame of the same exposure and mean sensor temperature gets the same treatment by the camera?  Or is it more complex than that?  How would one find out?

Anyway, back to my main question. As I understood it, the dark scaling routine uses a noise evaluation process to find the best linear scaling factor for the dark frame.  If the camera processing is a purely linear function for the whole frame, surely dark frame optimisation should still work for DSLRs.  If the processing is non-linear then it might not.

I can certainly get successful correlation/scaling for my DSLR darks if they are a reasonably large stack (maybe 30 frames).  Interested to hear your thoughts.
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: Harry page on 2014 February 28 09:23:53
Hi

I find that dslr cameras gain different amount of dark current due mostly to the fact that they are not cooled ( and regulated )

I hear ( because I do not use one) have to collect darks at different temps and lengths and have found in practice with data sent to me that scaling did not work well

Good scaling requires a linear effect , but if you find other wise who am I to disagree  >:D

 thanks for the input

Harry
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: Phil Leigh on 2014 February 28 09:54:13
I cAn only speak for Canon DSLR darks... But they definitely do NOT scale linearly. Craig Stark and other have investigated this in-depth. There is some non-defeatable internal processing that occurs in camera... And no it is not simply a bias subtraction... Which completely messes up the linearity of the dark current. This kicks in with any exposure over 10 seconds.

Canon have so far refused to explain what they are doing.

I have abandoned DSLR darks completely and rely on bias subtraction and dithering (oh and true cooling of the sensor to -25)... I have zero dark current. Makes life soooooo much easier.
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: IanL on 2014 March 01 01:49:37
I'd love to be able to abandon the whole idea of DSLR darks due to the uncertainty, but only having the one camera that I can't risk breaking, cooling or other such fun it not an option at the moment. My current efforts are trying to investigate the following:

- Is it possible to match a set of darks properly to produce a reliable master dark? You certainly can't rely on the EXIF temperature, since the mean brightness doesn't correlate well to it.  So you either have to match by reference to mean brightness (may not be a good indicator of mean temperature due to the camera processing), or noise (should be a good indicator of mean temperature as it isn't affected by the processing, but noise would need to correlate to mean brightness if it is to be a reliable matching criteria - that is what I am going to look at next).

- Is it possible to match an unscaled master dark to a light frame by reference to any of the above parameters?  That would avoid the need for scaling (though it is hard to make enough matching masters due to lack of temperature regulation).  I don't know if there is a way to correlate (say) noise in a background area of the light to noise in the individual darks used for the master to create a reliable match though.

- Does dark scaling work to overcome problems of matching master darks to lights?  This is a bit of an unknown.  Based on Stark it certainly seems clear that you cannot match a short (less than 2 min) dark to a longer exposure light, or vice versa as there is a hockey stick curve with the change of direction around 100-120 seconds.  My thought was that darks and lights with exposures greater than this might scale because the dark current on Stark's graphs does appear to be linear after 2 minutes, so it may be a case that having apples and apples allows the scaling to work.  Conversely my own tests suggest that there is still something non-linear going on at 10 minutes so maybe not.

Empirically, I am pretty sure my current master darks used with scaling more of a benefit than a hindrance.  Examining several images calibrated with and without master darks it is clear that there is a pattern on the scale of 4 - 6 pixels that the dark is successfully removing (and it is obvious that the pattern is not part of the signal from the target).  It may be that I am just getting lucky, but I'd like to prove it one way or the other if poss.
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: Phil Leigh on 2014 March 01 03:35:42
The EXIF temperature is not the actual temperature of the sensor - that's not its purpose. The actual sensor temp can be 7-8c higher  (non cooled camera) or 7-8c colder (correctly cooled camera). Also in a non-cooled camera the sensor temp will rise during a long exposure.

Having said that, I've had success in the past using darks that are matched by EXIF temp within 3-4c and are the same duration as the lights... just need to keep away from that "optimise" button :-)

Dark scaling seems to work well with CCD images (I've processed lots for other people). The other main benefit of dark scaling is that you do not need flat darks - but with a DSLR and a good light source you can shoot an excellent flat in less than one second so the dark current is negligible anyway.

Do you use dithering? - most folks think dithering is only good for hot/cold pixel removal but I find that dithering a DSLR removes larger scale pattern noise too.

To be honest my expertise (hah!) is with cooled "DSLR's" (CentralDS, JTW etc) and the reason why I went down that road was that after using 5 different DSLR's (all good, recent models) I could not get a convincingly low noise calibration result... whereas with the cooled cameras the whole problem goes away.

I think someone who uses stock DSLR's would probably be able to help better with this. I would investigate dithering if you don't already do it. Interestingly, my dither scale correlates to 7-8 pixels which seems to be in the same ballpark as your noise...
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: Juan Conejero on 2014 March 01 12:52:16
The "No correlation ..." message means exactly what it says: that there is no correlation between the image being calibrated and the master dark frame. More explicitly, this means that the master dark frame cannot be applied because the result would be more noisy than the original.

Our dark frame optimization routine does not need, and does not use, any information about any acquisition condition, including temperatures and exposure times. It is a purely numerical solution. It will always converge to the best solution in the sense of minimizing noise in the calibrated image, even if such best solution consists of not applying the dark frame at all, as happens when you see the "No correlation..." message.

Another common misunderstanding about our dark frame optimization routine is to think that it assumes or requires a linear relationship between dark current and exposure time, or between dark current and temperature. This is not the case. Our routine does not assume anything at all about the shape of the noise minimization function (that's why it doesn't depend on any physical property). The only requirement is that the minimized function have a global minimum, which our routine will find.
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: MikeOates on 2014 March 01 15:37:56
Juan,

Thank you so much for the definitive answer to this question which has appeared here many times causing so much confusion. That explains why, even with a number of subs all with the same acquisition details the odd one will give this message.

Harry also said "Look at your darks and Bias carefully please", well I am and I will post back with my findings, which will be very enlightening!

Regards,

Mike
Title: Re: Yes another: "** Warning: No correlation between the master dark and target...
Post by: bitli on 2014 March 02 07:08:00
The kind of error which can happen with bias/dark are:
-- bitli