PixInsight Forum

PixInsight => General => Topic started by: Juan Conejero on 2015 August 14 03:28:18

Title: Dark Frame Optimization Algorithm
Post by: Juan Conejero on 2015 August 14 03:28:18
We have seen some discussions recently on other forums about dark frame correction and dark frame optimization. Since some of these discussions and analyses involve our image calibration tools and algorithms, and mainly because I've read some uninformed opinions and judgements, I think it is pertinent to stick here a copy of a post I wrote back in 2012 (http://pixinsight.com/forum/index.php?topic=4161.msg29156#msg29156), where I describe our current dark optimization algorithm, how it works, and its main limitations.

That said, an improved dark frame optimization routine is currently in the implementation/test phase and will be released in the coming Fall/Winter for PixInsight 1.8.4. The new algorithm is more robust to read noise in the master dark frame and improves on the hot pixel undercorrection problem.


Let's state the problem first. We may describe an uncalibrated raw light frame with this very simple equation:

I = I0 + D + B

where I is the uncalibrated raw light frame, D is the dark current, and B is the bias pedestal. All terms in this equation are vectors (or matrices) because we are describing a process for a whole image at the pixel level. I0 is the dark-and-bias-subtracted raw light frame (we know that the next calibration step would be flat fielding, but this is unimportant to this discussion). Our goal here is to obtain I0.

As we have stated it above, this is a trivial problem: just subtract good master bias and dark frames from the uncalibrated light frame, and the result will be a good approximation to I0 (the better the masters, the better the approximation). However, the actual problem is more complex because the dark current D is highly time- and temperature-dependent, so a perfect match between the dark current signals in I and D is only a theoretical goal. A better (but still far from perfect) approximation to the actual problem is more like this:

I = I0 + k*D + B

where k is a scaling factor that attempts to account for effective exposure time and temperature differences between the light frame and the master dark frame. The process of finding a good approximation to the scaling factor k is what we call dark frame optimization.

Note that I have said that this is far from a perfect solution. It is wrong mainly because the dark current does not vary linearly with exposure time and temperature for the whole numeric range of the data, so a single multiplicative factor cannot be very accurate, especially when time and temperature differences between I and D are large. Dark optimization, however, is always better than nothing. From our experience, dark optimization is beneficial even if exposure times and temperatures are matched between light and dark frames. For this reason the corresponding parameter is enabled by default in our ImageCalibration tool.

Now that we have described the problem, let's describe our solution. Algorithmically this is known as an optimization problem: find the value of a parameter that minimizes (or maximizes) the value of a function. For example, think in terms of the economical cost of a production process. We have to produce an item A that depends on a factor r, which leads to the problem: find the value of r that minimizes the cost of producing A. In real cases r usually represents a complex set of parameters and constraints, such as environmental factors, availability of prime matters, production systems, etc., so that the most difficult part of the solution often consists of identifying a significant set of parameters or model to define a suitable cost function. This leads, for example, to the subject of linear programming (http://en.wikipedia.org/wiki/Linear_programming) and, more generally, to mathematical optimization (https://en.wikipedia.org/wiki/Mathematical_optimization).

The dark frame optimization problem is a relatively simple case of function optimization with a single parameter. The first step is to define the cost function that we want to minimize. To explain why and how we have selected a particular function we need some visual analysis. Take a look at the following two images:

(http://forum-images.pixinsight.com/20120423/A.png) (http://forum-images.pixinsight.com/20120423/B.png)

One of them is a crop of a bias-subtracted master dark frame, the other is not. Difficult to say which is which, isn't it? The image to the right is a mix of synthetically generated uniform noise and impulsional noise (salt and pepper noise) with 0.5% probability. It has been generated with the NoiseGenerator tool in PixInsight. With this comparison I am trying to show that a master dark frame looks very similar to random noise: essentially, a master dark frame is composed of pixel-to-pixel intensity variations whose spatial distribution is rather uniform, plus a relatively small amount of hot pixels whose distribution and typical values are similar to impulsional noise. Of course, we know that the thermal noise is a fixed pattern characteristic of a given sensor, so it is not random because it is predictable. However, morphologically a master dark frame is virtually indistinguishable from random noise. It is not random noise, but it behaves like that, and we are going to take advantage of this simple property to implement a purely numerical solution.

So our cost function is just a noise evaluation function. We already have implemented a powerful multiscale noise evaluation algorithm in several PixInsight tools, such as ImageIntegration, to implement a noise-based image weighting algorithm (Jean-Luc Starck and Fionn Murtagh, Automatic Noise Estimation from the Multiresolution Support, Publications of the Royal Astronomical Society of the Pacific, vol. 110, February 1998, pp. 193-199). In this case, however, we have implemented a simpler and very efficient method (Jean-Luc Starck and Fionn Murtagh, Astronomical Image and Data Analysis, Springer-Verlag, 2002, pp. 37-38) known as k-sigma noise estimation:

- Compute a single-layer wavelet transform of the image. This transform consists of the finest wavelet scale w1 plus a residual cJ, which we simply discard. We use the standard B3 spline wavelet scaling function as a separable low-pass filter.

- Iterate a k-sigma process as follows: At each iteration n > 0, denote by d(n) the subset of pixels in w1 such that |d(n-1)ij| < k*sigma(n-1), where sigma(n-1) is the standard deviation of the current subset of pixels (from the previous iteration). In our implementation we have d(0)=w1, k=3, and this process is iterated until no significant difference is achieved between two consecutive iterations. For dark frame optimization, we iterate up to 1% accuracy and normally 3 or 5 iterations are sufficient.

- The final noise estimate is the standard deviation of the resulting subset of pixels. Note that this is an unscaled noise estimate of Gaussian white noise, since it depends on the wavelet scaling function used. However, since we don't want to compare noise estimates among different images, but only to minimize the noise in a particular image after dark subtraction, an unscaled estimate is perfectly suitable.

Note that the use of a wavelet transform is actually irrelevant in this process; we use it just because it is very fast and we have it readily available on the PixInsight/PCL platform. A simple convolution with a low-pass filter followed by a subtraction from the original image would be equivalent.

This noise evaluation algorithm is robust and 'cannot fail'  The rest of our algorithm is a routine to find the minimum of a unimodal function. For the sake of robustness, I haven't implemented anything fancy here: just an adaptation of the classical golden section search (http://en.wikipedia.org/wiki/Golden_section_search) algorithm. Perhaps not the fastest alternative, but golden section search is absolutely robust, which is what I want here.

Summarizing, the algorithm can be described at a high level as follows:

- Find a range [k0,k1] of dark scaling factors that bracket the minimum of the noise evaluation function. This is the initial bracketing phase.

- Iterate the golden section search algorithm to find the minimum of the noise evaluation function. At each iteration, compute the calibrated raw frame as:

I0 = I - k*D - B

and evaluate the noise in I0 with the k-sigma iterative method.

- Iterate until the minimum of the noise evaluation function is determined up to a prescribed accuracy (1/1000 fractional accuracy in our implementation).

- The final value of the dark scaling factor is the value of k from the last iteration.

From our tests, this algorithm has proven extremely robust and accurate. However it has two known problems:

- It is very sensitive to bad quality calibration frames. In particular, if the master bias and/or the master dark frames include significant amounts of read noise, the optimization algorithm will find a value of k that overcorrects thermal noise to compensate for the additive random noise component. In extreme cases, this may lead to dark holes as a result of overcorrected hot pixels. However, if the master calibration frames are bad quality ones, the whole image reduction process makes no sense, actually...

- It implicitly assumes that the dark optimization function is constant for the whole range of numerical data values, since it consists of a simple multiplicative factor. This is not the case in practice, especially for high intensity values. As a result of this simplistic optimization model, the algorithm tends to undercorrect hot pixels. This is a very minor issue, however, since hot pixels are easily rejected during integration (because you dither your images, don't you?) and can be removed also with cosmetic correction techniques.

In a future version of the dark optimization algorithm we'll compute different optimization factors for separate intensity ranges in a spline fashion. This should fix the hot pixel undercorrection problem.
Title: Re: Dark Frame Optimization Algorithm
Post by: k8jb on 2016 August 26 08:39:55
I was just about to ask another question re: dark frames and you answered most of my concerns.  You note that the dark noise components are not linear with time and temperature. So my question is this: is there a reasonable way to pick a time/temperature for your typical images (i.e., due to light pollution, I'm usually limited to under 2 or 3 minutes max binned 1x1 and certainly less binned 2x2). And my working temperature range is -10C (hot summer night) to -25C (winter night).  I have a set of dark frames every 5 degC from -30C to -10C at 3 minutes exposure so I presume I can check the linearity of the dark noise vs temperature, then maybe do the same thing at different exposure times, right?  Would that be of any value?  Or would a simple 3 minutes at -15C (kind of in my middle temperature range) work about as well as anything?  I only image for my own enjoyment so extreme precision is not required.

I understand you can't give me a definitive answer, just point me in the right direction.
Title: Re: Dark Frame Optimization Algorithm
Post by: savery on 2016 October 10 04:24:11
Dear Juan,

When creating darks and bias files with MaximDL, the default pedestal setting is 100ADU.
If these files are then used in Pixinsight to calibrate lights using the default settings, the program removes the pedestal as a first step.  When the resulting calibrated files are then measured with subframeselector, most of the measured values are zero, i.e. the calibrations are unusable.

I just tried to process a set of images of the Bubble Nebula.  I had 10 red, 10 blue, 10 green and 10 Halpha.  After much head-scratching and hair-pulling, I finally worked out that if I increase the output pedestal as 100 ADU, the reds and greens are correctly calibrated.  To calibrate the blue, I needed to increase the output pedestal to 200, whilst to calibrate Halpha, I needed a pedestal of 300 ADU.

Is this behaviour expected?  I ask because I am unable to find any documentation regarding increasing the pedestal in this way.  The nearest I got was your message from 2010 where you imply that only in very special circumstances it may be necessary to add a small pedestal in the range 50 to 200 ADU "especially in master calibration frames".  Perhaps what I am doing is exactly what you recommended, but I am puzzled as to why the need to do this adjustment is not better documented.  I spent hours and hours trying to work out why the lights were not being properly calibrated.  In fact until recently, I gave up and just skipped the calibration altogether.  Even the new excellent book "Inside Pixinsight" by Warren Keller is silent on this question.

Perhaps this issue is peculiar to my imaging setup for some reason, but I don't think so because I had a very similar problem when using a different camera, a different scope and different imaging software (Astroart).

Sorry if I am raising an issue that everyone knows about except me!

Best regards,

Title: Re: Dark Frame Optimization Algorithm
Post by: nickg on 2016 October 13 16:36:49

I have the same issue. I have never been able to use Bias frames for a successful calibration in PI. I get zero background values that render the calibration useless. Dark and flat calibration work fine. If anyone could shed some light on this behavior that would be great.

Title: Re: Dark Frame Optimization Algorithm
Post by: schulzep on 2017 June 04 09:59:46
I use modded Canon XSi and T3i cameras.  I am using the trial PI to look at some data processed by ImagesPlus (IP) and PS3. I did not dither, but I took all the lights, darks and flat, within a four hour window. Flats were taken using an EL plate.  IP stacked 15 in the final image whereas PI stacked only 6.

In PI I received many errors about incompatibility of darks during calibration. The same happened with the flats. Then only six lights were stacked 

My question is what to do?  If the darks and flats were taken one after another as rapidly as possible, what more can a person do to obtain darks and flats which won't cause errors?  In a non-cooled DSLR, the temperature will naturally rise. If this is the problem, one might just forget darks and go to dithering.

Finally, does the lack of good darks and flats have anything to do with the rejection of lights in PI?
Title: Re: Dark Frame Optimization Algorithm
Post by: RickS on 2017 June 04 20:33:52
In PI I received many errors about incompatibility of darks during calibration. The same happened with the flats.

By errors, do you mean "Warning: No correlation between dark and target frames" messages?  These are common when calibrating flats with darks that have much longer duration.  This is just a warning, not a fatal error.  It is telling you that the result of dark calibration was going to be noisier than the original sub, so it skipped the dark calibration.

Then only six lights were stacked 

My question is what to do?  If the darks and flats were taken one after another as rapidly as possible, what more can a person do to obtain darks and flats which won't cause errors?  In a non-cooled DSLR, the temperature will naturally rise. If this is the problem, one might just forget darks and go to dithering.

Finally, does the lack of good darks and flats have anything to do with the rejection of lights in PI?

If some of your lights were rejected then it is more likely that image registration failed.  Check the output on the process console.  StarAlignment prints a summary with the number of lights successfully registered and the number of failures.  You may need to tweak the registration parameters if this is the case.  I'd also recommend you Blink check the calibrated lights to see if they all look OK.

I'd also strongly recommend dithering whether you take darks or not.  Personally, I always do both!

Title: Re: Dark Frame Optimization Algorithm
Post by: M Covington on 2018 January 01 20:13:39
That algorithm is a very good idea because I know from experience with DSLRs that it is almost impossible to take dark frames at the same temperature as the images.  (Or images at the same temperature as each other!) Some cameras record the temperature and tell us how much it is shifting; others leave us guessing.
Title: Re: Dark Frame Optimization Algorithm
Post by: M Covington on 2018 January 04 17:39:26
I notice that when I use dark frame optimization, I get calibrated images with higher amplitude than when I don't.   Does this mean the optimization checkbox (in Batch Preprocessing) also does some scaling of the calibrated images?
Title: Re: Dark Frame Optimization Algorithm
Post by: PHannah on 2018 November 27 11:41:57
Juan, thank you for this very interesting article. It has answered a question that has been troubling me, namely: why is there no EXPTIME keyword in the FITS header of my master darks? I thought this was needed in order to scale the master dark to the same exposure as the light frames (or flats) being calibrated.

Now I see that PI uses a much more sophisticated algorithm that takes into account temperature and non-linear variations also. So it doesn't need to know the exposure time of the master dark.

Is that correct or am I missing something?

Title: Re: Dark Frame Optimization Algorithm
Post by: Juan Conejero on 2018 December 18 02:06:57
Hi Peter,

Sorry for the late answer, I didn't see your post before.

So it doesn't need to know the exposure time of the master dark.

That's correct. Our dark frame optimization algorithm is purely numeric. It does not use any metadata or acquisition properties stored in the images.