dark frames

[k8jb]

I made a master dark using the Master Calibration Frames Tutorial and discovered it wouldn't work with the IC tool - the warning message said it couldn't find a suitable dark frame.  So I used the batch preprocessing script to cal/register/integrate using my raw darks, biases and flats and then inspected the FITS headers.

The tutorial-generated darks did not have for entries in the header that ARE in the batch script.

IMAGETYP  'Master Dark'
XBINNING  1
YBINNING  1
EXPTIME  600.

Also, shouldn't there be a temperature entry in the header or is that left up to the user?

The batch preprocessing generated darks seem to work OK otherwise.  What am I doing wrong??


John

[Cleon_Wells]

John, I always batch format convert my Dark subs, then integrate them and in IC I check "Calibrate" in the MasterDark section.
I don't know what info the fits header should have in it.
Cleon

[pfile]

i think the batchpreprocessing script does write some FITS keywords so that later it can match masters to lights.

temperature is probably interesting to have in the FITS header, but PI does not depend on knowing the temperature or duration of darks while scaling them... that's probably why it's not in there.

[k8jb]

That's what I thought - but then I started getting warnings that IC couldn't find a corresponding dark.  But it likes the darks created by batch preprocessing script (using the same original darks).

So let me rephrase the question:  do I NOT need to make darks and biases at several temperatures - I have data sets at 5deg intervals from -10 to -30C binned 1x1 and 2x2.  Are these no longer all needed and if so which one do I choose?  (I needed them for another program...)

[pfile]

you have to match the binning, so you'll need at least one master dark per bin.

scaling darks is still... scaling. it's best to use darks that match the temperature reasonably closely. dark current increases linearly with time and (i believe) exponentially with temperature. this is not to say that a scaling factor can not be found that minimizes the noise in the result when you use a given dark, but your outlier pixels (warm/hot/cold) may behave differently when the temperature and time of the dark is wildly different from the target frame. so you may need to use DefectMap or CosmeticCorrection to fix up those pixels.

but... you say that "IC could not find a corresponding dark". by this do you mean that ImageCalibration said it could find no correlation between the dark and the light? this is usually a symptom of dark current that is vanishingly low. in this case it's better not to scale the dark and just use a master dark that matches your light. or do you mean the BatchPreProcessing script threw an error saying it could not find a matching dark?

[k8jb]

"but... you say that "IC could not find a corresponding dark". by this do you mean that ImageCalibration said it could find no correlation between the dark and the light? this is usually a symptom of dark current that is vanishingly low. in this case it's better not to scale the dark and just use a master dark that matches your light. or do you mean the BatchPreProcessing script threw an error saying it could not find a matching dark?"

The IC tool said it could find no correlation between the dark and the light.  When I compare the integrated images made from image sets calibrated when the warning occurs and when I use the script-generated dark frames, I do see a difference, probably not much but it's there.   I presume when the warning is generated, no dark subtraction is done for that sub.

Tnx for the discussion,guys - I think I will re-run the process with fewer lights and capture the process window so I can see exactly what's happening.

One other thing, I could swear I read a calibration tutorial somewhere out there in cyberspace that gave a detailed discussion about how PixInsight works (not this one: Master Calibration Frames: Acquisition and Processing - the one I think I saw had the math details).  Now, of course, I can't find it...

[mschuster]

k8jb,

When you get the warning, I believe IC still does a dark subtraction using the logged dark scaling factor, which will be a number close to zero. I believe the warning simply indicates that the scaling is close to zero.

I did some reverse engineering and found these IC formulas:

A bias calibration only does this: frame - bias_master

A bias and dark calibration does this: frame - bias_master - dark_scaling_factor * (dark_master - bias_master). This is the case where the dark master was created with no bias subtraction and the optimize flag is checked. The dark_scaling_factor is the number logged.

Finally, a full bias, dark and flat calibration does this: (frame - bias_master - dark_scaling_factor * (dark_master - bias_master)) / (flat_master / flat_scaling_factor). The flat_scaling_factor is the mean of the flat_master, which is logged in the console at the beginning.

Mike

[k8jb]

Thanks Mike,

I looked a little deeper and found that if I calibrate my original images all goes as expected, no warnings, correct scale factors.  But if I run the defect correction process prior to calibration, I get the results we've been discussing - so it has something to do with defect correction process or the order I'm doing it in.  The stats of the pre-calibrated/post-defect-corrected images are a bit different  but I wouldn't expect that to be the problem and the images stats do seem to indicate dark subtraction has taken place even though it shows a 0.0 scaling factor.

Speaking of defect correction - I presume it's applied prior to image calibration since it's an inherent characteristic of the sensor and has nothing to do with the imagery.  Also, defect correction of the darks and flats - should that be done or not?


John

[lucchett]

Hi John,
Some defects can be eliminated with proper calibration.
I think it is correct to apply defect correction after calibration.
On thecontrary you should also defect correct bias and darks first, but I am not sure it could work.
Andrea