No correlation between master dark and target frame

[ecoles]

I am getting an error message when running the BatchPreprossing script. The error message is in part, No correlation between master dark and target frame. I have had this same error when just doing the correction and correlation steps separately. As far as I can tell, I am selecting dark frames of the appropriate time and bin. This is probably something obvious I am doing wrong.

Please advise.

Also, I was trying to find the file that would show these errors. Is there one and if so, where would it be?

Thanks.
Eric

[ecoles]

I was doing some reading on the forum and found others with the same issue. Unchecking "Optimize dark frames" apparently solved the problem. I tried it and that worked. Not sure exactly why. Perhaps someone could explain.  Eric

[pfile]

well it's not so much that it solved the problem but cured the symptom. if you don't do dark optimization then PI does not try to find a correlation between the dark and the light. it just subtracts the dark from the light without analysis... so no message appears.

what kind of camera is it? if it's a CCD, some sony CCDs have really low dark current, hence not much dark signal. if it's an OSC, it could be indicative of a mismatch between the bayer settings and/or fits reader settings between the lights and the masters, though if you did everything in BPP from scratch just now, that's probably not the issue.

rob

[ecoles]

Rob,
I ran it without Optimize checked and it worked. Not sure why there should be any problem. The darks were taken as usual and have worked on other software with no issues. the CCD camera is the QSI 583 wsg. I checked both the bias and dark (15 min) frames. The average pixel value for the bias was around 240 and for the dark around 400. Both had a similar profile across the sensor. Does that tell us anything? If not what is it I should check?
Eric

[pfile]

well that's what i'm saying - when optimize is checked, PI tries to scale the dark until the noise in the calibrated frame is minimized. if it can't correlate the noise in the dark and the light, it throws that warning. analyzing the frames is at the heart of the scaling algorithm and if it can't analyze the frames then it can't scale properly. when optimize is not checked, then it just subtracts the dark from the light without any scaling. so no message.

with the qsi i have seen cases where, for some reason, the bias signal exceeds the dark signal. this should not happen, and i think QSI is playing games with the raw data. or, i was working with masters with different gain settings. not sure.

your pixel averages seem to indicate that they are okay. still it might be worth checking the master dark for clamped-at-zero pixels. if the master dark is uncalibrated then you'll have to manually subtract the master bias from the master dark and see if you get a result with clamped-at-zero pixels. that will definitely mess up the calibration process.

i can't remember the order that BPP does things - if it makes uncalibrated masters then you'll have to do the manual step of calibrating the master dark.

rob

[ngc1535]

Hi,

I am not certain this point is important to make or not... I have to optimize my darks to scale them before applying to real exposures. However, the BPP script also uses the dark in an attempt to calibrate flats. Since the flats are a few seconds in length and the darks are 30 minutes the warning about no correlation comes up for the FLATS but not for the main images being calibrated. So it is possible to see the warning and it is OK as long as it isn't talking about the real data (and not flats). It wasn't indicated above whether flats were being made into a flat master on the fly. You wouldn't want to just turn off the optimize dark option just because you saw the warning dealing with Flats.

Let me know if this is an on-point comment.

Thanks,
Adam

[pfile]

good point adam, this is true. if it was during flat calibration then perhaps it is not an issue, though i'm not sure what IC does when it does not find a correlation - does it just subtract the dark without modification or does it just not use the dark? i guess if it just subtracts the dark it would completely clobber your flat, so maybe it does not do that.

i wish BPP would just let me say "use master bias to calibrate flats" since my flats are usually short enough that a bias frame is adequate to calibrate them.

rob

[ngc1535]

Hi Rob,

Yes..right! That would be exactly the thing to do. If there isn't a correlation I think you are right nothing is subtracted. However, even if it chooses to do it- the math would result in values so small it may not really matter very much.

-adam

[ecoles]

Rob, Alan,

Thanks so much for your comments and analysis. I am still kind of stuck as to whether to check or un-check Optimize. I am afraid that my understanding is not advanced enough to make that decision. It would be nice if I could access the log file of the script to check what was done against your comments.

In the end I will probably consider the quality of the image generated both ways. So I ran it with and without checking Optimize. Here is what I can tell you. With STF applied the two integrated images (with and without) look almost identical. However the histograms are completely different. With Optimize checked (and the error message) the histogram is significantly shifted to the left as compared to the un-checked integration. If I do a simple stf stretch the images look the same. The histograms of the two stretched images are slightly different, but nothing significant. Processing the images in the usual way produces visually identical results.

In any case, what I need is a recommendation. Considering what you know, should I check or un-check Optimize? Awaiting your advise.

Eric

[pfile]

well, dark optimization is generally used when your dark duration does not match your light duration. the theory says that the dark current that adds up on the sensor is a linear function of time, and so it's possible to scale the dark to match the light. usually people use long darks and scale those down to match shorter lights.

some programs look at your light exposure length vs your dark exposure length and simply multiply the dark by the ratio of the two durations. pixinsight takes this a step further - it completely disregards the exposure times, and instead scales the dark iteratively, looking for a calibrated result with minimum noise. this is why it throws the warning - it's tried to scale the dark but does not see the noise in the calibrated result changing as it changes the scaling factor. hence, no correlation.

so if your dark duration matches your light duration, it's probably fine to not use dark optimization. with optimization turned on, you may see an integrated light with slightly less noise. since we're always trying to squeeze the highest SNR out of the data we have, it may make sense to use dark optimization even if your dark duration matches the lights. it's a question of final SNR in your stack. if you can't tell the difference, then i guess it's okay... but comparing the two methods with something more qualitative (like the NoiseEvaluation script) is probably a good idea.

rob

[ecoles]

Rob,

This is getting far beyond my ability to comprehend. Let me tell you what I do know.

1. I ran the noise script and both linear files look the same.
2. If you look at the histograms of the two linear files. The one run without Dark Optimization has a much broader curve and looks like it is already partially stretched. That is, the start of the peak is near zero. The linear file with Dark Optimization has a much narrower peak and is shifted well to the right on the expanded histogram. Perhaps I could upload these fits files and have someone do a more technical analysis.
3. My darks are the same duration as my lights. That's the way I was instructed. If that's not correct, let me know.
4. Once I do the stretch (MaskedStretch), the files look almost identical. The histograms are slightly different, but not much.
5. When fully processed, Pi and PS, again the images look the same. I see no difference in quality, resolution, detail, etc.

Just judging from the histograms of the linear files, I would not do the Dark Optimization. My thinking is the broader initial histogram is a better way to start. What do you think?

Eric

[pfile]

well as i mentioned just due to the peculiarities of how CCDs work, it's possible to scale darks. for instance i use 1800s darks and then scale them down to 1200s or 900s. this is just a shortcut. if you have the ability to make darks that match every light frame that you create, then that's certainly the "best" thing to do. but there's no harm in scaling/optimizing your darks if you want to save some time creating dark subs.

so what you're doing is not wrong, it's just the "strict" way to do it.

i don't know what the histogram shapes mean. a wide histogram can either be representative of varying pixel intensities in your image, or it can be representative of noise.

i think the risk here is only that if BPP is throwing the "no correlation" message while calibrating your lights, your lights may not be calibrated properly. however if your optimized dark and non-optimized dark light integrations seem the same, then i suppose all's fair.

bottom line is that if your dark and light exposure lengths match, then dark optimization is just that, an optimization, and is not strictly necessary.

rob

[ecoles]

Rob,

Thanks for all the input. I will think on this for awhile. But the results seem to relatively the same either way and the images are good. So I will go without the dark optimization, at least for now.

Cheers.

Eric

[Philippe B.]

Salut Eric


Maybe you could search "CORRELATION" on this forum and you will find the cause of your problem 
"Bitli" (for example) or "Juan", as Rob (pfile), have several answers to this problem.


Cheers
Philippe