Hi Matts,
how minimizing the noise in the resulting calibrated Light Frame secures that we have removed the correct Dark Signal.
Because:
(i) Uncertainties in both the light frame and the master dark frame, as well as the signal in the light frame, remain constant during the whole dark scaling process. Since they remain constant, we can simply ignore all of them them during the whole process.
(ii) Dark current, as recorded in the master dark frame, consists exclusively of
small-scale structures. By small-scale structures, we refer to pixel-to-pixel variations. We use a wavelet transform to isolate these variations in the first wavelet layer, as image structures at the 1-pixel scale.
(iii) A necessary precondition is that the uncertainty in the measurement of the dark current is negligible in the master dark frame.
(iv) Another necessary precondition is that the light frame and the master dark frame are
correlated. This means that both frames share the same dark signal, although scaled by a factor k > 0. Our algorithm tries to find the value of k.
Note that our algorithm is purely numeric. It is not based on any physical properties of the sensor. In fact, our algorithm ignores temperatures and exposure times completely, because it does not need to know them.
Note also that we treat dark signal as both signal and noise, at different stages of the algorithm. When you integrate a large number of dark frames into a master dark frame, dark current
is signal. By averaging n dark frames, you reduce the uncertainty in its measurement by Sqrt(n). However, when our dark scaling algorithm evaluates noise in the light frame after dark subtraction, the same dark current
is noise. This is the difficult part because of the duality in the interpretation of the same part of the data. I'll try to explain this in more detail.
Following your notation, we have:
Master Dark Frame = DF = k*D + Nd
Light Frame = LF = S + D + Ns
where D is the dark signal, S is the image signal, Nd is the noise in the master dark frame and Ns is the noise in the light frame. Both Nd and Ns are supposed to be randomly valued and uniformly distributed. Our task is to find a good approximation to the true dark scaling factor k > 0.
Precondition (iii) says that the uncertainty in the measurement of D is negligible, that is:
|D| >> |Nd|
so we can assume:
Nd = 0
without changing anything important. This means that the quality of the master dark frame (in SNR terms) is very important, a fact that Vicent has stressed sufficiently in his tutorial. Note that this is also extremely important in our numerical procedure because, as Nd is uncertain by nature, we have no way to remove it, so each time we try out a scaling factor k we'll be multiplying also k*Nd. If Nd and D are comparable in magnitude, they may easily become undistinguished numerically, and our process will have no way to converge to a good (that is, certain) value of k.
Precondition (ii) tells us that in the morphological and statistical senses, D is very similar to Ns. Both features are composed of small-scale structures. Ns is random and can be assumed to follow a Gaussian distribution, as usually happens with large data sets having a strong central tendency. D also has a strong central tendency, since most dark current variations are quite similar. We know that D is not a random variable, but observed over a large portion of the image, its distribution is basically uniform. Uniform and random here, while not the same thing, can be treated in the same way because its properties are essentially the same for our strictly numerical purposes.
Now let's try out a value of k, call it k
i:
LF
i = LF - 1/k
i * DF
We have said that we can neglect Nd, so we have:
LF
i = S + D - k*D/k
i + Ns
The above expression is evaluated iteratively to feed a minimization algorithm. At each iteration, the algorithm computes the standard deviation of the noise in LF
i. When the algorithm brackets a minimum within a very small interval (< 0.005 in the current implementation), the current value of k
i is returned as the true value of k.
As S and Ns remain unchanged during the whole process, what we are minimizing is actually the standard deviation of:
D - k*D/k
i + Ns
Note that our multiscale noise evaluation algorithm is able to isolate S from the rest of terms, and it performs that task in a very robust way.
As I've said above, here we are treating D as if it were pure noise. As we approximate k
i = k, the standard deviation of the noise in LF
i reaches a minimum, which is approximately equal to the standard deviation of Ns. As Carlos has pointed out, the function to be minimized is strictly continuous and has a single minimum. An important fact that must also be pointed out, is that our algorithm makes no assumption as for the linearity of the dark signal as a function of temperature, exposure time, or any other physical acquisition condition. In fact, the dark signal could exhibit a wildly nonlinear behavior, and our algorithm would still find the optimum scaling factor k.
Hope this helps you and all users to understand how our algorithm works. It is not perfect, and we indeed have some ideas to improve it, but we think it works remarkably well.
