Basic Question on Calibrating Darks and Flats

[Mark de Regt]

Hi,

First question:  I am getting the idea that I should be calibrating my master dark with a master bias.  Is this correct?  If so, do I just run the master dark though "Image calibration," applying a master bias, not checking any other boxes, and calling it good?

Second question background:  I understand that, in an ideal world, I would have dark frames matching the exposure time of the flat frames.  But mine is not an ideal world, and my flats (dusk and dawn flats controlled by software) may be anywhere from one second to ten seconds in duration.

What's a guy to do about this situation?  When I just use a bias as a dark, it doesn't seem to give good results.

So now my second question: "How can I get the best master flat, given the frame-to-frame variance in exposure times of my individual flat frames?"

And, given whatever the right answer is, what boxes do I check, etc., on the "Image Calibration" process when calibrating the individual frames?

Thanks.

Mark

[bulrichl]

> First question:  I am getting the idea that I should be calibrating my master dark with a master bias.  Is this correct?  If so, do I just run the master dark though "Image calibration," applying a master bias, not checking any other boxes, and calling it good?

No, the MasterDark should NEVER be pre-calibrated. If needed (i. e. when using dark frame optimization in the calibration of the lights), the MasterDark can be calibrated during light frame calibration. The correct settings for the light frame calibration are:

a) No dark frame optimization:
Disable section 'Master Bias'
Enable section 'Master Dark', select the MasterDark, uncheck both 'Calibrate' and 'Optimize'
Enable section 'Master Flat', select the MasterFlat, uncheck 'Calibrate'

b) With dark frame optimization:
Enable section 'Master Bias', select the MasterBias
Enable section 'Master Dark', select the MasterDark, check both 'Calibrate' and 'Optimize'
Enable section 'Master Flat', select the MasterFlat, uncheck 'Calibrate'

(Details are given in my guide https://pixinsight.com/forum/index.php?topic=11968, chapter 4 and 5.)

> Second question background:  I understand that, in an ideal world, I would have dark frames matching the exposure time of the flat frames.  But mine is not an ideal world, and my flats (dusk and dawn flats controlled by software) may be anywhere from one second to ten seconds in duration.
> What's a guy to do about this situation?  When I just use a bias as a dark, it doesn't seem to give good results.

In my view it is unusual that calibrating flat frames (exposure time of only 10 s) with the MasterBias gives bad results. What camera are you using? Which are your settings for calibrating the flats and integrating the calibrated flats?

Bernd

[Mark de Regt]

Thanks, Bernd.

But I'm still confused.

1.  I have no idea what "dark frame optimization" is, or whether I should be performing it.

2.  In the guide to which you linked, under "3. Generation of the Master Calibration Frames," you say "It is advisable to prepare (according to [1]) and then check the master calibration frames."  But I have no idea to what you are referring by "[1]," and reading the entire piece did not solve my problem.

Can you shed some light on this?

Thanks.

Mark

[pfile]

hi mark, hover your mouse over the optimization checkbox in the master dark area of ImageIntegration for a brief explanation of optimization.

rob

[bulrichl]

Mark,

[1] is Vicent Peris's tutorial "Master Calibration Frames: Acquisition and Processing". The sources,  with links, are given at the bottom of the guide.

You did not indicate which camera you are using. Knowing this would facilitate an answer whether dark frame optimization might be useful for you or not. Maybe that you will have to try both approaches and compare.

Bernd

[frajanssen]

Hello,

Whenever there is information about the calibration process, I get confused.

Let's say I have the following collection: Lights, Darks (the same exposure time as the lights), Bias, Flats and Dark flats (the same exposure time as the Flats). All at the same temperature.
This is what I do:

1. Integrating of the Bias files to a Master Bias.
2. Calibration of the Darks with the Master Bias.
3. Integrating of the calibrated Darks to a Masterdark.
4. Calibration of the Dark Flats with the Master Bias.
5. Integrating of the calibrated Dark Flats to a MasterDarkFlat.
6. Calibration of the Flats with the Master Bias and the MasterDarkFlat.
7. Integrating of the calibrated Flats to a Master Flat.
8. Calibration of the Lights with the Master Bias, the MasterDark and the Master Flat.

I can do this with or without Batchpreprocessing, with or without optimization.
Is this the right way?

François Janssen

[bulrichl]

Hi François,

there are several ways to achieve a correct calibration. The one that I suggest (i. e. NOT to pre-calibrate darks, the MasterDark, Flat-Darks or the Master Flat-Dark) avoids the risk of a clipped MasterDark or a clipped MasterFlat-Dark. It is the most simple one and the one that requires the least computing time and hard disk space.

1. Integrating of the Bias files to a Master Bias.
2. Calibration of the Darks with the Master Bias.
3. Integrating of the calibrated Darks to a Masterdark.
4. Calibration of the Dark Flats with the Master Bias.
5. Integrating of the calibrated Dark Flats to a MasterDarkFlat.
6. Calibration of the Flats with the Master Bias and the MasterDarkFlat.
7. Integrating of the calibrated Flats to a Master Flat.
8. Calibration of the Lights with the Master Bias, the MasterDark and the Master Flat.

Your approach produces correct calibration results provided that your MasterDark and MasterFlat-Dark have not been clipped in the pre-calibration step. In my guide I gave examples in which clipping was a severe issue. Do you test MasterDark and MasterFlat-Dark for clipping?

There are further disadvantages of your approach:

a) precalibration of darks (and flat-darks) is a needless separate step that consumes time and
b) will produce a bunch of files that unnecessaryly consume space on the hard disk (e. g. calibrated darks and calibrated flat-darks).

These are my reasons for NEVER pre-calibrating darks, the MasterDark, flat-darks or the MasterFlat-Dark.

Bernd

[frajanssen]

Hi Bernd,
Thanks for the answer. Yes, it cost time and hard disk space.
Perhaps a simple answer: how can I test for clipping?

François

[bulrichl]

 

Perhaps a simple answer: how can I test for clipping?

PixInsight has these nice tools, HistogramTransformation and ImageStatistics. Take a look at the histogram and the statistics (option 'Unclipped' unchecked) of a pre-calibrated MasterDark or a pre-calibrated MasterFlat-Dark. In my guide (see Image 1) I have opposed the histograms of a pre-calibrated to a not pre-calibrated MasterDark (Canon EOS 600D, but this also happens with my ZWO ASI2914MC Pro). Of course clipping in the pre-calibrated MasterDark could be avoided by applying an output pedestal as well, but it is unnecessary to apply an output pedestal, when you DON'T pre-calibrate.

Bernd

[Mark de Regt]

Mark,

[1] is Vicent Peris's tutorial "Master Calibration Frames: Acquisition and Processing". The sources,  with links, are given at the bottom of the guide.

You did not indicate which camera you are using. Knowing this would facilitate an answer whether dark frame optimization might be useful for you or not. Maybe that you will have to try both approaches and compare.

Bernd

Thanks, Bernd!

I have an SBIG STX-16803, brand new.  I previously had had an STL11000M for many, many years, until it recently died.

The effects I'm seeing are: (1) a few dust donuts on the master light frame when using "Batch Preprocessing," and (2) a light column shows up as a dark column in the master light frame when using "Batch Preprocessing," even when I use the defect map (it doesn't happen when I do each step separately.

[bulrichl]

Mark,

I am no expert for defect columns since I never used a CCD camera, but I noticed posts about the topic in this forum, e. g.:
https://pixinsight.com/forum/index.php?topic=12338
https://pixinsight.com/forum/index.php?topic=12602
Perhaps some well-informed forum member could chime in.

The presence of dust donuts in the MasterLight might be a result of wrong calibration of the flat frames. Do the donuts appear positive (like in the MasterFlat = undercorrected) or negative (inverted = overcorrected)?

If the bias offset was not subtracted from the flats, donuts would be undercorrected. If the bias was subtracted twice, they would be overcorrected. (These examples shall only show the trend of the effect.) However, the problem can be due to other reasons as well.

Your remark (the issue occurs with the BatchPreprocessing script, but does not emerge when performing the steps manually) rather suggests that it may be caused by setting incorrect parameters in the script.

I am sorry, but I am afraid I cannot help you, the cause of the issue is not obvious for me.

Bernd

[frajanssen]

Bernd,

Thanks for the comment. In the following days I will test.

François