Confusion about Darks / Flats / BIas / Dark Flats

[STAstro]

It's that time of year again when I build my new Calibration Library (Since moving to Unity Gain as well), so my Calibration Frames are as follows

1. 101xDarks for Each Exposure Length (90, 150, 300 and 600 Seconds) - Used to create Master Dark for each exposure length
2. 101xFlat frames taken as same Gain/Offset as Darks and Lights for each colour filter
3. 3500+ BIAS Frames with same Gain/Offset as Flats, Darks and Lights

Now I have always done the following
Calibrated Flats against Master/SuperBIAS since my flat frames are sub 4 seconds (LRGB are all sub second) to create Master Flat for each filter
Combined all Darks to create Master Dark for each exposure length (Not subtracting BIAS)
When Calibrating Lights, only the Master Flat and Master Dark are subtracted (Since Darks contain BIAS Signal)

However, I am reading that I should be deducting the BIAS from the darks and then in the Light frame calibration I should be deducting the Master/SuperBIAS from the lights, so the question is, should I only be doing this is I am scaling my Darks?

TIA
Simon

[Niall Saunders]

Hi Simon,

It is my opinion (and method of choice) to acquiire Lights, Darks, Flats and FlatDarks at the same Gain/Offset values, and to always have a set of Darks to match my Lights (in terms of exposure time) - and a similarly matched set of Flats and FlatDarks at an exposure time that gives me suitable ADU range for calibration. In 'my case' you have to also keep in mind that I run a cooled imager, and that I set that to -20ºC all year round amd from year to year.

If these constraints can be achieved the I believe that there is no case that can be argued for requiring Bias Frames, or a SuperBias.

In the 'real world', just how many different exposure times does the average astro-imager actually require? I use two - 300s and 600s, and that means I need a library of Darks based on those two exposure times. I have a third folder for my FlatDarks, the expsure times for which have remained constant for quite some time now (howerver, I do examine my MasterFlatDark, monitoring the statistics of the ADU levels quite closely, and I can almost argue the case for just not wasting time acquiring FlatDarks at all as the ADU field is so low, and so  flat across the image).

But, because I can gather Calibration Frames so easily, I tend to do this anyway - even Bias Frames (I usually leave the camera unattended in the observatory collecting Dark Frames as this stage does take time - especially as I like to acquire around 100 exposures of each type.). For the Bias Frames, I only create a MasterBias so that I can see if anything is changing un the imager as the camera ages over time (over 8 years now) - I don't use the MasterBias for any Calibration stages.

Does this help give you a way of examining your current processing methodology? Remember, my method suits 'me', others may agree with my framework, others will (hopefully) scorn and ridicule my efforts and suggestions - which allows me to take another step in the learning curve.

[STAstro]

Hi Nial

Thanks for your reply, I also use a cooled imager and also set it to -20 so that I can image all year round too, however the QHY183M does not have a mechanical shutter, so in essence the dark frames are dark flats for the exposure time of 90s, 150s, 300s and 600s.  Because my flat frames for LRGB are all sub second exposures the need to subtract dark flats from them is not required, also you could argue that my narrowband flats are all sub 4 second, so again the need to subtract dark flats from them could be argued also.

My main question is around BIAS frames and where these should be subtracted from, as I said I only subtract them from the Flat frames at the moment and leave the BIAS signal in the Darks so they are being subtracted fromt he lights, however I came across this page on DSS website giving three ways to correctly calibrate light frames and it states at the bottom "Any other combination leads to improper calibration so you should stick with one of these 3 possibilities." which lead me to believe that my processingmethod is not correct.

http://deepskystacker.free.fr/english/theory.htm#CalibrationProcess

The nearest I come to it is Alternative Method 1, however in that method the BIAS is subtracted from the darks and then the darks subtracted from the flats, but surely this is then scaling the darks to compensate for the shorter exposure on the flats, because it is also stating that the BIAS needs to be subtracted from the lights (Obviously because it has been removed from the darks and flats)

I understand that you can't have BIAS in the Darks or Flats if you intend to subtract it from the lights, I also understand that you should not have BIAS in both the Darks and Flats at the same time as you would be subtracting the BIAS twice in pre-processing, so which method is correct?

1. My current method - Subtract BIAS from Flats, then deduct darks and flats only from Lights
2. Subtract BIAS from Darks and Flats and then Subtract Darks, Flats and BIAS from Light frames

Regards
Simon

[Niall Saunders]

Hi Simon,

First, I have never liked the diagrams used on this web page - they are extremely confusing. In many ways, a simple set of frame names, and formulas explaining their sources,  is probably easier to understand (again, it suits my way of thinking:-

Offsets : A camera could be confogured to a gain of 1.000, with an offset of 0.000, but the actual number of photons collected in an exposure may not generate a usable range of ADU values unless the gain is increases. Also, as the ADU value returned by the camera wil have been to various sources and types of noise, it is usually highly recommended that the offset level is increased to avoid ADU values trying to exist in the negative number domain.

Biases : This is how the result of Gain and Offset values actually manifest themselves although there are actually no such things as Offset or Gain Frames. They could be considered to be a PedestalElrctronSignal.

Darks : These frames contain the ThermalElectronSignal (and associated noise) 'sitting on top of' the BiasElectronSignal (and associated noise) - the whole signal having, ideally, been acquired under the same conditions as the Lights.

Lights : These frames contain the PhotonSignal (and associated noise) 'sitting on top of' the ThermalElectronSignal (and associated noise) - the whole signal having been acquired during the image capture process.

This means that we can now write the following pseudo-formula :-
Lights = LightPhotonSignal + Dark
and, if required, this can then be expanded to :-
Lights = LightPhotonSignal + DarkThermalSignal + Bias
Usually we will maximise the SNR by averaging a large subset of Darks to make a MasterDark
If the MasterDark is subtracted from each Light, we end up with :-
DarkCalLights = LightPhotonSignal

Now, the mathematics of the section concerning Lights and Light Darks can be repeated for Flats and FlatDarks to give the following pseudo-formula :-
Flats = FlatPhotonSignal + FlatDark
and, if required, this can thenbe expanded to :-
Flats = FlatPhotonSignal + FlatDarkThermalSignal + Bias
Again, we will maximise the SNR by averaging a large subset of FlatDarks to make a MasterFlatDark
If the MasterFlatDark is subtracted from each Flat, we end up with :-
DarkCalFlats = FlatPhotonSignal
Again, we will maximise the SNR by averaging a large subset of DarkCalFlats to make a MasterDarkCalFlat

Finally, in order to eliminate dust donuts and other optical abberations, we have to divide the DarkCalLights by the MasterDarkCalFlat, giving :-
FullCalLights = PhotonSignal

So, as you can see, at no point in the formulas listed above do we explicitly have to collect or utilise Biases at all. This is because each time we perform the <subract> operation we are subtracting one set of data from another - where both sets of data have (statistically speaking) the same bias signals. In other words the bias signals cancel each other out.

The only time this doesn't apply is if you are not using a 'level playing field' - you may be exposing Lights for a time that differs from those used when you caotured your Darks (or Flats, when you captured your FlatDarks), or you may not have had decent (or any) thermal control of the imaging chip. In these cases, it is possible to partially recover decent SNRs by applying Biases - but I believe that it is better to avoid mathemagic approaches in favour of taking the time to do things right first time round.

Whilst I don't disagree with the highlighted comment that you took from the DSS website, you have to be very careful as to how the diagrams are interpreted. Remember, they are not clear (IMHO) and can easily lead to confusion (mostly because lines may appear to overlap, but whether they do, or not, is open to question. Similarly vertical text doesn't help , and neither does the inconsistent use of 'action icons. So much so that I can't even really be bothered about which of the three categories 'my approach' most closely follows !

You closed with:

1. My current method - Subtract BIAS from Flats, then deduct darks and flats only from Lights
2. Subtract BIAS from Darks and Flats and then Subtract Darks, Flats and BIAS from Light frames

I suggest the following instead :-
1. Make a MasterDark from your Darks
2. Subtract your new MasterDark from your Lights and make new DarkCalLights
3. Make a MasterFlatDark from your FlatDarks
4. Subtract your new MasterFlatDark from your Flats and make new DarkCalFlats
5. Make a new MasterDarkCalFlat from your DarkCalFlats
6. Divide your new DarkCalLights by your new MasterDarkCalFlat and make new FlatDarkCalLights
7. Pre-process your new FlatDarkCalLights as per your normal steps to amke a final MasterLight, ready for Post-processing

Once again, I hope that you (and others) can pick out the picture that I painted using, at the evry least, "a thousand words" 

[STAstro]

Thanks again Niall

So one final question, in your opinion, when is it neccesary to create flat darks?  Since my LRGB Flats are all sub-second, and my narrowband ones are <4 seconds, are flat darks really neccesary? 

Since my Lights and Darks will be taken at the same exposure time, and my Lights, Darks and Flats all taken at -20C I don't have any need to perform scaling on the darks

Simon

[Niall Saunders]

Hi Simon,

Whether or not you choose to use Darks (of any type) depends on the statistical quality of your basic dark frames. If you don't have any hot/cold pixels, and have a very tight spread of thermal data around some median value, then you could just consider the dark frame to be 'equivalant' to that median value - and that median value should be equivalent to the 'Offset' parameter (multiplied be something related to you 'Gain' parameter) - in reality the madian value would be the same as the median value of a MasterBias created from a set of Biases for your imager.

However, there is usually a larger distribution ('noise') in a set of Darks than there would be in a set of Biases, so it makes sense to just take a little bit of extra time to acquire the arks as well. After all, the total time taken to acquire the same number of short-exposure Darks isn't really much longer (if at all) than the time taken to acquire the Flats themselves - and if you are finding that you typically always expose your Flats for the same times, then a single acquisition of FlatDarks might last you a very long time indeed (mine are over three years old, and show no signs of changing as the camera 'ages').

Does that  help?

[STAstro]

Thanks Niall

Helps a lot