Image Calibration - the details ?

[Niall Saunders]

First, let me re-post Juan's message in full, and then I want to break it down, to see if I can understand things.

Here is his original response :-

Because:

(i) Uncertainties in both the light frame and the master dark frame, as well as the signal in the light frame, remain constant during the whole dark scaling process. Since they remain constant, we can simply ignore all of them them during the whole process.

(ii) Dark current, as recorded in the master dark frame, consists exclusively of small-scale structures. By small-scale structures, we refer to pixel-to-pixel variations. We use a wavelet transform to isolate these variations in the first wavelet layer, as image structures at the 1-pixel scale.

(iii) A necessary precondition is that the uncertainty in the measurement of the dark current is negligible in the master dark frame.

(iv) Another necessary precondition is that the light frame and the master dark frame are correlated. This means that both frames share the same dark signal, although scaled by a factor k > 0. Our algorithm tries to find the value of k.

Note that our algorithm is purely numeric. It is not based on any physical properties of the sensor. In fact, our algorithm ignores temperatures and exposure times completely, because it does not need to know them.

Note also that we treat dark signal as both signal and noise, at different stages of the algorithm. When you integrate a large number of dark frames into a master dark frame, dark current is signal. By averaging n dark frames, you reduce the uncertainty in its measurement by Sqrt(n). However, when our dark scaling algorithm evaluates noise in the light frame after dark subtraction, the same dark current is noise. This is the difficult part because of the duality in the interpretation of the same part of the data. I'll try to explain this in more detail.

Following your notation, we have:

Master Dark Frame = DF = k*D + Nd
Light Frame = LF = S + D + Ns

where D is the dark signal, S is the image signal, Nd is the noise in the master dark frame and Ns is the noise in the light frame. Both Nd and Ns are supposed to be randomly valued and uniformly distributed. Our task is to find a good approximation to the true dark scaling factor k > 0.

Precondition (iii) says that the uncertainty in the measurement of D is negligible, that is:

|D| >> |Nd|

so we can assume:

Nd = 0

without changing anything important. This means that the quality of the master dark frame (in SNR terms) is very important, a fact that Vicent has stressed sufficiently in his tutorial. Note that this is also extremely important in our numerical procedure because, as Nd is uncertain by nature, we have no way to remove it, so each time we try out a scaling factor k we'll be multiplying also k*Nd. If Nd and D are comparable in magnitude, they may easily become undistinguished numerically, and our process will have no way to converge to a good (that is, certain) value of k.

Precondition (ii) tells us that in the morphological and statistical senses, D is very similar to Ns. Both features are composed of small-scale structures. Ns is random and can be assumed to follow a Gaussian distribution, as usually happens with large data sets having a strong central tendency. D also has a strong central tendency, since most dark current variations are quite similar. We know that D is not a random variable, but observed over a large portion of the image, its distribution is basically uniform. Uniform and random here, while not the same thing, can be treated in the same way because its properties are essentially the same for our strictly numerical purposes.

Now let's try out a value of k, call it ki:

LFi = LF - 1/ki * DF

We have said that we can neglect Nd, so we have:

LFi = S + D - k*D/ki + Ns

The above expression is evaluated iteratively to feed a minimization algorithm. At each iteration, the algorithm computes the standard deviation of the noise in LFi. When the algorithm brackets a minimum within a very small interval (< 0.005 in the current implementation), the current value of ki is returned as the true value of k.

As S and Ns remain unchanged during the whole process, what we are minimizing is actually the standard deviation of:

D - k*D/ki + Ns

Note that our multiscale noise evaluation algorithm is able to isolate S from the rest of terms, and it performs that task in a very robust way.

As I've said above, here we are treating D as if it were pure noise. As we approximate ki = k, the standard deviation of the noise in LFi reaches a minimum, which is approximately equal to the standard deviation of Ns. As Carlos has pointed out, the function to be minimized is strictly continuous and has a single minimum. An important fact that must also be pointed out, is that our algorithm makes no assumption as for the linearity of the dark signal as a function of temperature, exposure time, or any other physical acquisition condition. In fact, the dark signal could exhibit a wildly nonlinear behavior, and our algorithm would still find the optimum scaling factor k.

Hope this helps you and all users to understand how our algorithm works. It is not perfect, and we indeed have some ideas to improve it, but we think it works remarkably well.

Now, I will follow on with my 'interpretation' of the responses, with questions inserted where I start to 'lose the plot'

Cheers,

[Niall Saunders]

(i) Uncertainties in both the light frame and the master dark frame, as well as the signal in the light frame, remain constant during the whole dark scaling process. Since they remain constant, we can simply ignore all of them them during the whole process.

Here, I am assuming that any image, i, can be considered to be a combination of signal, s, and uncertainty, u (or 'noise').

This allows us to write
i = s + u

Because, at this early stage of the calibration process, we have NOT yet stacked our Lights, then we are only talking about a SINGLE (individual, raw) Light frame, and therefore our task, or aim, is to eliminate the u component such that we have
i = s
which is great, assuming that we can establish some representation of the 'noise uncertainty', u

Now, at this point we can 'ignore' the whole issue of Flat frames - they can actually just be considered as a 'special instance' of Lights - the sole purpose of which is to identify optical anomalies in a given imaging train. In other words, any process we choose to apply to our Lights (using, perhaps, Darks and Biases) can be generically applied to our Flats.

At this point, however, I do not understand the "Since they remain constant, we can simply ignore all of them " statement


(ii) Dark current, as recorded in the master dark frame, consists exclusively of small-scale structures. By small-scale structures, we refer to pixel-to-pixel variations. We use a wavelet transform to isolate these variations in the first wavelet layer, as image structures at the 1-pixel scale

This statement is representative of the fact that 'dark noise' (the dark current that creates 'uncertainty' in our desired image, i) has an effect on a PIXEL-by-PIXEL basis. So, a pixel that may be highly susceptible to dark current (e.g. a 'hot' pixel) has no specific ability to infuence the behaviour of its nearest neighbours.

This is why investigations using a 1-pixel size Wavelet approach allows local variations to be assessed.


(iii) A necessary precondition is that the uncertainty in the measurement of the dark current is negligible in the master dark frame

Presumably this situation is created by acquiring sufficient individual Dark frames, and then integrating these in a statistically robust manner to generate a MasterDark where the u component has been minimised.

It would seem therefore that, although the overall concept being proposed within ImageCalibration allows a MasterDark to be more 'widely applicable' (i.e. it is no longer quite so important to match Lights and Darks in both temperature and exposure time), there is a fundamental requirement to have at least taken sufficient individual Dark frames to ensure that the MasterDark is a 'good' one.

What is not clear here is whether it is better to create a MasterDark based on a set of LONG exposures, or whether short exposures would create the same end result. What is also not specifically stated, although I am assuming that it is implied (for statistical 'robustness'), is that the series of individual Darks themselves MUST be temperature and time correlated. Can someone clarify this?


(iv) Another necessary precondition is that the light frame and the master dark frame are correlated. This means that both frames share the same dark signal, although scaled by a factor k > 0. Our algorithm tries to find the value of k

I am confused by this statement almost straight away. Initially I assumed that this referred to the notion of 'temperature correlation', whereby all calibration fames that are to be used with each other ARE correlated in terms of temperature. If this is NOT the case - as is 'hinted at' towards the end of Juan's reply - then why bother with closed-loop TEC cooling of CCDs in the first place?


OK - let me throw these thoughts out for discussion, before I try and break down the remainder of Juan's reply in my mind.

Cheers,

[Juan Conejero]

At this point, however, I do not understand the "Since they remain constant, we can simply ignore all of them" statement

I refer to the fact that those components of the problem —signal, other noise in the light frame— are invariants during the dark scaling process. Since they don't change, our algorithm just hasn't to handle them in any special way.

At each iteration, we just vary the dark scaling factor k, subtract k*D from the light frame, and evaluate the standard deviation of the noise from the first wavelet layer of the result. "Noise" here is actually composed of (scaled) "dark noise" and "other noise sources in the light frame", but since the latter term remains constant, we just ignore it. When k represents the true scaling ratio between dark signal in the master dark frame and the light frame, the dispersion of the noise in the dark-subtracted result reaches a minimum. We stop when we are able to bracket a minimum in the noise evaluation function within a sufficiently narrow interval, and take the k that achieves that minimum as the output of the algorithm.

(the dark current that creates 'uncertainty' in our desired image, i)

(Note: I have edited this paragraph after my initial answer to explain it better)

The "pattern" of pixel-to-pixel dark current variations on the CCD is not random. So it is not an uncertain quantity and hence does not create uncertainty in the image —that's why we take dark frames, because in this way we can obtain an estimate of the mean dark current pattern. Speaking in strict terms the dark signal is an artifact that we obviously want to remove, since it represents nonuniform variations that do not belong to the image. We treat dark current as if it were noise in our algorithm, but this is only an algorithmic trick that allows us to implement an accurate method to optimize the dark scaling factor, adapted to each target image.

What is not clear here is whether it is better to create a MasterDark based on a set of LONG exposures, or whether short exposures would create the same end result.

You want a very high signal-to-noise ratio in your master dark frame. This is especially true with the scaling algorithm implemented in our ImageCalibration, for the reasons I explained in my previous post (precondition 3). So long exposures are better, but you don't want saturated or wildly varying hot pixels in your individual dark frames, so this fixes an upper limit.

What is also not specifically stated, although I am assuming that it is implied (for statistical 'robustness'), is that the series of individual Darks themselves MUST be temperature and time correlated.

That's absolutely correct. A set of individual dark frames to be integrated into a master dark frame should be coherent.

Initially I assumed that this referred to the notion of 'temperature correlation', whereby all calibration fames that are to be used with each other ARE correlated in terms of temperature.

I was talking of correlation in purely abstract terms. The dark signal is the information that is shared by the light frame and the master dark frame, although there is a scaling factor multiplying it in the master dark, which we want to know. This notion of shared information is what we are referring to when we speak of correlation.

then why bother with closed-loop TEC cooling of CCDs in the first place?

Our algorithms and implementations try to solve existing problems in a very accurate and efficient way, but they are not intended to invalidate or replace good acquisition techniques, by no means.

You actually gain nothing by acquiring light and calibration frames at different temperatures. Our scaling algorithm may be able to let you work —within certain limits— with data sets acquired under varying conditions; however, by acquiring them under similar, controlled conditions, you get more coherent data and hence you help our algorithms to do their job better and more accurately.

[Carlos Milovic]

Yes, that's the second law of image processing: always get the best raw data possible

(the first is that in most cases problems are related to the light, but we cannot change the illumination of our objects )

[mattssporre]

Niall,

In (i) I think you make one error in the following statement

(i) Uncertainties in both the light frame and the master dark frame, as well as the signal in the light frame, remain constant during the whole dark scaling process. Since they remain constant, we can simply ignore all of them them during the whole process.

Here, I am assuming that any image, i, can be considered to be a combination of signal, s, and uncertainty, u (or 'noise').

This allows us to write
i = s + u

Because, at this early stage of the calibration process, we have NOT yet stacked our Lights, then we are only talking about a SINGLE (individual, raw) Light frame, and therefore our task, or aim, is to eliminate the u component such that we have
i = s
which is great, assuming that we can establish some representation of the 'noise uncertainty', u

We can never remove the uncertainty, u, by working with only one light frame. The only way we can do that is by stacking a lot of light frames. The uncertainty in the stacked light frame will be u(stacked) = Sqrt(u1*u1 + ... + un*un).

The reason why it is possible to remove the darks signal by subtraction (and not by averaging = stacking) is just that it is not uncertainty. Thus you need to add the dark signal in your equation as well as the uncertainty in the dark signal.

i =s + us + d + ud

Since d is a master dark frame (stack of many frames) we can assume that ud << d and make the approximation

i = s +d + us

BR
Matts

[mattssporre]

Hi again,

I think that some of the misunderstanding here is due to belief that the dark signal is the same thing as noise (in the statistical sense). But it is not. It is more like the noise from the traffic outside when you really want to listen to Mozart and you can not hear the music properly because the traffic is so darn loud 

This is the reason why I always use the word "uncertainty" instead of noise. In our situation the difference between uncertainty and dark signal manifests itself in the following way. We can make the uncertainty as small as we want by stacking a lot of frames (averaging), but we can never remove the darks signal by stacking! The dark signal is a REAL signal. Unwanted but real.

The only way we can remove the dark signal is by subtracting it. Vice versa we can not get rid by uncertainty by subtracting only one frame.

If we know the temperature and the exposure length we can just measure the dark signal and subtract it (being careful so it does not add too much uncertainty to the image by stacking many dark signal measurements to construct a master dark).

If we do not know the exp time and temp of the master dark frame nor the light frame (the PI algorithm does not care), we need to estimate the true dark signal by scaling our dark frame with a factor k and minimise d(of the light frame) – k * d (master dark).

My understanding of what Juan said is that this difference can be treated statistically.

BR
Matts

BR
Matts

[vicent_peris]

Hi,

even if you have light and dark frame with the same lenght and temp, my recommendation is activate the dark frame scaling as it will correct small temp fluctuations.

OTOH, don't care about acquiring your dark frames at the same temp and with the same lenght. As dark signal pixel to pixel proportions are constant, there's no problem if you integrate a two min dark with a ten min one. What makes sense is not knowing how much dark signal has a pixel, but the proportion of its dark signal to the one of the adjacent pixels. That's because dark signal is a fixed pattern that will have all your images acquired with a determined camera and a determined camera configuration.

Regards,
Vicent.

[Carlos Milovic]

Is that true with Canon's amp-glow? I'm not that sure...

[mattssporre]

Vicent,

I think your statement

Hi,

OTOH, don't care about acquiring your dark frames at the same temp and with the same lenght. As dark signal pixel to pixel proportions are constant, there's no problem if you integrate a two min dark with a ten min one. What makes sense is not knowing how much dark signal has a pixel, but the proportion of its dark signal to the one of the adjacent pixels. That's because dark signal is a fixed pattern that will have all your images acquired with a determined camera and a determined camera configuration.

Regards,
Vicent.

needs to be taken with a grain of salt. If I would use this for really large differences (e.g. dark frame temp of +10C and light frame of -15C) I would like to understand the algorithm in detail .

I am not at all sure all CCDs/DSLRs have this property (having the same proportional of dark signal compared to the other pixels). I remember to have seen a graph (only exposure time differences) and the scaling was linear only for moderate deviations from the used exposure time.

BR
Matts
Btw the user interface to write this stuff in keeps jumping up and down when the reply gets larger than the square. Really annoying. Is there a better way?

[vicent_peris]

I am not at all sure all CCDs/DSLRs have this property (having the same proportional of dark signal compared to the other pixels). I remember to have seen a graph (only exposure time differences) and the scaling was linear only for moderate deviations from the used exposure time.

Of course always is better to acquire all the frames under the same environmental conditions. My words were assuming that dark signal increases linearly with time.

In the next implementation of the dark scaling algorithm we will make further approachings to correct this non linearity, specially of the real hot pixels.

V.

[Carlos Milovic]

Yes, for larger differences the correlation is not linear. What I don't know if it is a function of temperature and time only or for intensities too (I mean, even if there is not a linear relation of temp and exp. time, still always exists a constant number witch makes de correlation, or it becomes a function of pixel intensities).

PS: Try Firefox, or use the "Reply" buttom instead of the quick-reply.

[Simon Hicks]

Matts,

I agree with everything you say about confusion of terms and its a very valid point that cannot be stressed often enough.

However, 'uncertainty' seems a problematic one as well. I would stick with Dark Signal and Dark Noise.....one is a signal and one is a random noise element on that signal. We might be very certain or uncertain about either of these depending on how much info we have, hence the problem with the term 'uncertainty' instead of 'noise'.

Cheers
         Simon

[Niall Saunders]

Btw the user interface to write this stuff in keeps jumping up and down when the reply gets larger than the square. Really annoying. Is there a better way?

Try running your Internet Explorer in 'compatibility mode', by clicking the small icon at the end of the address box - or upgrade to Firefox

Have a look at http://pixinsight.com/forum/index.php?topic=1809.msg10874#msg10874

Cheers,

[mattssporre]

Niall,

Try running your Internet Explorer in 'compatibility mode', by clicking the small icon at the end of the address box - or upgrade to Firefox

Yes! now it works much better  .

BR
Matts