Bias subtraction from dark frames??!

[TheMadOne]

Lo all,

New to PixInsight, running through the trial right now to give it a good evaluation before committing any $$$ to it.  So far it has proven some nice points along with some minor annoyances.  Have already seen some decent improvements in working with some of my prior processed & "published" files from subs through to final.  I have a question on what I am seeing posted here & elsewhere on processing the calibration frames.  I am seeing people stating to subtract the Bias calibration master from the Dark frames to generate your final master dark.  This is where I have questions.  I have been using Iris along with Nebulosity up to now to do my preprocessing, calibration frame generations & such.  In both instances, Jim Soloman's Astrophotography CookBook ( http://www.saratogaskies.com/articles/cookbook/index.html ) and Craig Stark's nebulosity http://www.stark-labs.com/help/nebulositytutorials/tutorials.html  Recommend NOT subtracting the Bias from the Darks as this would actually introduce noise into your image as the bias is already there in your darks to be calibrated out of your final lights.  From what I have read from Jim, Craig & others is the bias should be used for calibrating the flats as they also contain bias which is reintroduced to the ligh frames during calibration if the flats have not been calibrated.

Basically what I am reading into these guides (Jim & Craig) subtracting the bias from the darks means it is not there in the master dark for subtraction from the lights so it is "left" in the lights by the dark calibration.  Bias is subtracted from the calibrated flats which would reintroduce the bias to the lights after the darks if not subtracted.  As I read it if the master dark leaves behind the bias (as it has been removed from the darks by the bias subtraction), it is passed right through the rest of the processing as the flats which are applied after the dark subtraction also have the bias removed to avoid reintroduction of noise during the flat step.

Whew!!!!!  That was a mouthful, hope my ?? makes some kind of sense.  So, am I reading it right here, youzzz guys are stating to subtract the bias from the dark frames?  It looks like the BatchPreProcessing script is subtracting bias or calibrating the flats for use, but leaving the darks as combined & used as a master dark without bias subtraction which appears to be the way others are stating to handle the whole calibrations routine.

Thanks for looking!  CS!

Mark Jordan
http://www.thestardeckobservatory.com

[naavis]

I think subtracting the master bias frame from the master dark frame is only necessary when you need to scale the dark current, because only the dark signal is scaled, but the bias signal stays constant.

[mschuster]

+1. Mark, I am not sure you can do this with the preprocessing script, but it is easy to do with the ImageCalibration process. Disable the Master Bias pane, enable the Master Dark pane and disable both the Master Dark calibration and optimize checkboxes. This is what I do for my lights (and flats too with a flat-dark master). I don't use biases at all in my workflow.
Mike

[martin_magnan]

Hi

I have found that the bias signal is the most important calibration file and it must be properly characterized by using at least 100 frames. I use  200 files for my master_bias. The more you use the  better. Yes the bias signal is theoricaly  included together with the dark signal in the dark files but using only a few tens of files for the master_dark provide a poor characterization of the bias signal.

Take a look at the pictures below
 It is a master_bias file from a Canon 40D at 100 ISO with 10, 20 40 80 and 160 frames

the whole frame scaled at 25%



and the portion of the master at 100%



it is clear that using a master_bias with less than 80 files will introduce a lot of noise in the calibrated frames including the so called horizontal banding that, I think, is actually introduced by poor bias characterization.

Other observations made when using a good master_bias

1- the dark will be more effective at removing hot and cold pixels from the light frames.
2- when stacking, one can use higher sigma values to rejected outliner and thus obtaining a better SNR.

Martin

[niteman1946]

Interesting stuff.

Mark

[Carlos Milovic]

Hi Martin

Vicent and I have being researching this topic for a while. We found that for certain detectors (both CCD and CMOS) there is vertical pattern that dominates the bias noise. Thus, we created the SuperBias process to emulate a master bias of hundreds of frames from just a few dozens. It works remarkable well in those cases.
Anyway, there are some chips where the vertical pattern has being controlled by the internal electronics, and so it is no longer dominant. This is the case of my Canon T2i. Random noise is greater than any obvious pattern even for more than a thousand frames. So, in my case, I just use 50-100 bias.

Mark, bias frames are always needed if the dark current needs to be scaled (which is the recommended way to go). Also, you need them for the flats.
If you are concerned about a double subtraction of the bias in the pipeline, the answer is that the results are slightly different even if darks are not scaled. The reason is that bias are subtracted to dark frames before integration. Thus, in those noisy pixels where dark-bias would yield negative results, the number is truncated to zero. Then, averaging those calibrated frames may lead to slight different results from calibrating the stright average of the dark frames.

[TheMadOne]

Thanks for the info!

Yes I totally agree the Bias current needs to be addressed during the preprocessing steps, just a little confusing info on at what point the corrections should be applied.  In the past I have done most of my preprocessing steps manually, apply master dark: take the dark calibrated light sub-frames, apply master uncorrected flat, then take the dark + flat calibrated light sub-frames & apply the Bias corrections........ After that register/align lights, review-preview light frames kick any with remaining issues, combine-integrate the light frames for my base aligned/combined image to star to work on.  What  is the "order of application" for the calibration frames when using the BatchProcessing Script?  I have used it a few times & really like it.... makes short work  of the whole preproccessing thing!  I have been going through some of my saved data files from old imaging sessions & redoing them to compare results.

Martin I am using the same Canon 40D for most of my wider scale stuff, but with the Hutech modified IR/UV filter.  I just acquired an Atik 314L+ Mono to do my smaller scale targets with, plan to dabble in some NarrowBand with it too.  Just managed to acquire my Ha, OIII, & SII filters for  it too as well as an included Hb filter with the deal.  So this whole processing thing is getting more & more attention from me lately................... Now, if I could just get the skies here to cooperate on a night I'm not on shift it would be great! 

I really do most of my imaging for the "fun of it" don't get too tied up in the science end, but it is nice to get some good data, process it through to the best result & be able look through it for anything new or interesting in the final images of whatever targets I have had the opportunity to get some shots of.  Been eyeballing the night skies with a few breaks here & there for years, more as a hobby, got into the imaging aspect just a few years ago.....the sickness has taken hold & I'm hooked on it now!


Mark Jordan
http://www.thestardeckobservatory.com

[Carlos Milovic]

The right way to do the calibration is:
( Img - bias - k*(dark-bias) )/(flat - bias)

[chris_todd]

The right way to do the calibration is:
( Img - bias - k*(dark-bias) )/(flat - bias)

May I assume k is the dark scaling factor?

[mschuster]

The right way to do the calibration is:
( Img - bias - k*(dark-bias) )/(flat - bias)

The ImageCalibration process has tool tips, but unfortunately the formulas it uses are not documented.

I have done a bit of reverse engineering, and IMO Carlos your formula is not what ImageCalibration does.

If you run ImageCalibration with a master flat specified you will see a "master flat scaling factor" listed. This appears to be simply the mean of the master flat.

With master dark calibrate and optimize both checked, and master flat optimize not checked, the formula is more like

(light - bias - k * (dark - bias)) * (mean(flat) / flat)

k is the dark scaling factor listed.

If on the other hand master flat calibrate is checked, then flat is also bias-subtracted and dark-scaled by a similar formula. The tool tip documents this.

For my subs the dark scaling factor k is usually 1 +/- 0.001 or so. Note that dark scaling factor is an approximation itself and its value is noisy to some extent. IMO it is unclear whether the +/- 0.001 is due to noise or is actually real. If you think k is very close to 1 (I do) then the expression (light - bias - k * (dark - bias)) simplifies to (light - dark). Bias drops out. This is why I do not use bias for my light calibrations.

My flats on the other hand are 24 second exposures through a Ha filter with a panel. I need the long exposure to get a reasonable 50% DN count or more. IMO, although dark current on my camera is very low, simply subtracting a bias is not enough, rather I want to use basically a flat-dark, a dark exposed to 24 seconds. But flat-dark scaling the flat ends up also with a near 1 scale factor again so I don't bother scaling.

So I end up with something that does this:

flatDarkMaster = integrate(flatDark)
flatMaster = integrate(flat - flatDarkMaster)
darkMaster = integrate(dark)
calibratedLight = (light - darkMaster) * (mean(flatMaster) / flatMaster)

No bias at all as I said before.

Finally, note that "dark scaling" itself is an approximation. Dark current per unit time varies per pixel. Juan I think has mentioned that he plans on improving the dark scaling algorithm. If so this is yet again a reason to investigate and see what works best for you.

Mike

[Carlos Milovic]

Yes, flats are normalized. This is a constant number, to achieve a mean of 1.0 in the flat. Other people use the minimum value, so it is always grater than 1.0 and others the maximum... so it is smaller than 1.0.
You just have to be consistent and use the same method. The scaling factor itself is not important.

Flat frames should always be subtracted by the bias frame, or they just do not represent accurately the multiplicative effects of uneven illumination of the sensor and pixel responses.
As you say, darks may also used with flat frames. Fortunatelly, they depend on the exposure time. So, if the time is short enough, and you are near of half the full well of the sensor (so, you are in the middle of the linear zone), darks are not needed in most cases.
As said before, dark frames included the bias information too. So, if you are taking flatdarks, technically you don't need them. Nevertheless, since bias are so cheap to take, having a few dozens of them is easier than dozens of flatdarks, so the bias current is better represented by a regular master bias.

[martin_magnan]

Hi

Yes Vincent I'm aware of your superbias process but AFAIK it is not available on MacOS. It is one of your post about Superbias that convinced me to do more bias frame to get a well characterized master_bias.

I have yet to own or use a sensor witch does not show a vertical pattern in the bias signal. A few weeks ago one of my friend sent me his master_bias and I was surprised to see no vertical banding.

The right way to do the calibration is:
( Img - bias - k*(dark-bias) )/(flat - bias)

This formula show why the bias is very important. It appear 3 times in the equation. If the master_bias is noisy then this noise will be injected 3 times in the calibrated frames and it become fixed noise. In my opinion, the use of noisy bias  is responsible for oblique streak pattern found in the background of many images and for the famous "Canon horizontal banding". Look at the pictures I posted above. You will see an horizontal banding in the bias with few frames but it disappear when using more frames. If the master_bias show horizontal banding then it will be injected in the calibrated frames.

f you think k is very close to 1 (I do) then the expression (light - bias - k * (dark - bias)) simplifies to (light - dark). Bias drops out

This is true if noise is near zero (High SNR) or if your master_dark and master_bias are made with the same number of files. But since it is very easy to get a high SNR master_bias with 200-300 individual files and not so easy with 5 or 10 or 20 minutes dark files, the easiest way to properly calibrate your light frames is by using a good master_bias. Try it.

Martin