Author Topic: New tutorial: Master Calibration Frames: Acquisition and Processing  (Read 35430 times)

Offline Yuriy Toropin

  • PixInsight Addict
  • ***
  • Posts: 209
    • View Profile
;) Back to business -

Juan or Vicent,
or anybody else involved in the magic,

can you please comment on what does this coefficient means
   Writing output file: X:/Photo/Processing/2010_04_15_M13/0 - Flat Calib/fl-d_Luminance_12_c.fit
   Dark scaling factors:
   k0 = -0.389

and can this information be used for obtaining better calibration frames, i.e. to reject some of individual Flat Field samples before integrating them in the Master Flat?

Value of this coefficient varies widely [-0.389,0.234] during calibration of the series from 32 individual flat field shots...

Thanks a lot,
    Yuriy

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6782
    • View Profile
    • http://pixinsight.com/
Hi Yuriy,

Quote
k0 = -0.389

This is the scaling factor by which the master dark frame has been multiplied before subtracting it from your target frame (a flat frame in your example).

The dark optimization routine has found that by multiplying the master dark by -0.389, the noise is minimized in this particular target frame after dark subtraction. Note that since the scaling factor is negative, the master dark frame is being added in this case, instead of subtracted. This denotes lack of correlation between thermal noise in the target frame and the master dark frame. We have forced negative dark scaling factors in artificial tests conducted to test ImageCalibration. Negative factors can be due to:

- Wrong calibration frames. For example, calibration frames and target frames acquired with different sensors, misplaced frames due to geometrical transformations prior to calibration, or lack of linearity.

- Presence of other significant sources of noise besides thermal noise. For example, if we add a significant amount of uniform random noise to the master dark frame (to simulate read noise), the optimization factor tends to zero (on average), because the random noise cannot be correlated with the target frames.

- Malfunction of the dark optimization routine. This is very unlikely because the routine implements extremely robust minimization and noise evaluation algorithms (robustness has been enforced even at the cost of significant performance loss). But we can never rule out this possibility completely.

Could you upload some of these frames and the master dark you are using? This looks like a very interesting test set.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
    • View Profile
Thanks for that Juan - that may wll be useful information in the future.

Cheers,
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline Yuriy Toropin

  • PixInsight Addict
  • ***
  • Posts: 209
    • View Profile
Hi Juan,
I've prepared 82Mb ZIP archive for upload.
Please, check you personal email at pixinisht com and give me instructions on where to put it.

BTW, would you prefer ZIP of RAR will be also Ok, it's smaller, ~71Mb.

Regards,
   Yuriy

Offline Yuriy Toropin

  • PixInsight Addict
  • ***
  • Posts: 209
    • View Profile
ZIP archive uploaded.
I used Master Bias and Master Dark (for 3 sec exposures) for calibration of Luminance flat field files, obtained at exposures ~0.2 sec,
everything is in archive together with the screenshot.
To be honest, Dark calibration could be, probably, skipped in this case, dark signal is low on KAF8300 chip at -25oC, but there are still some hot pixels visible.

Anyway, will be interesting to understand what the issue was with calibration of this set for the master flat field.

Also, probably, ImageCalibration should receive some setting to reject such outliers (with negative coefficients), on decision of user?

Thanks a lot,
   Yiriy

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
    • View Profile
    • http://www.harrysastroshed.com
Hi Tasos

Quote from Fabio At AstroArt

Hello Harry,
all 16 bit FITS files are 16 bit signed. (-32k .. 32k)
Unsigned files do not exist.

It's only possible to REPRESENT unsigned numbers,
in the range 0..64k using the BZERO trick.

http://fits.gsfc.nasa.gov/standard30/htmlfiles/fits_standard30index.html

See 5.2 and 5.3.

Hope it means something to you

Harry
Harry Page

Offline Alexander

  • Newcomer
  • Posts: 9
    • View Profile
Thankyou Vincent and Juan

Went through tutorial with my Lights Darks Bias Flat frames that i gathered over the last week, had same problem with bias subs, so then applied Juan,s RGB/K expression: $T - 0.5...and everything worked through ok, I only have a Meade Dsi colour camera but PI has a way to improve any image,with tutorials like these,so looking forward to more. :)

 Alexander

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6782
    • View Profile
    • http://pixinsight.com/
Hi Yuriy,

Thanks for uploading your calibration frames.

I have tested ImageCalibration with your files, and my results are similar to yours. With the default 1024x1024 optimization window I have obtained the following dark scaling factors:

-0.287,
+0.135,
-0.068,
+0.029.

This shows that there is no correlation between your master dark frame and the flat frames. The +0.135 factor, which is the only one that might be questionable, is IMO well within the uncertainty limits of function minimization.

With full frame optimization (optimization window = 0), the scaling factors are respectively:

+0.013,
+0.023,
-0.019,
+0.028,

showing that there is definitely no correlation at all between your flat frames and the master dark frame. We can conclude that you should not apply this master dark frame to your flats.

The dark optimization routines implemented in ImageCalibration work on the basis that the dark frame is correlated to the target frames with respect to thermal noise. Assuming negligible read noise and neglecting other sources of noise in the images, consider the following expressions:

F = S + NT
D = k*NT

where F is a target frame (being calibrated), S represents the signal in the target frame, NT is the thermal noise in the target frame, and D is the dark frame. Dark frame optimization tries to find the scaling factor k. It works by computing the dark-subtracted target frame F':

F' = F - 1/k * D

such that the standard deviation of the noise in F' is minimal as a function of k.

When there is no shared NT between F and D, the function minimization algorithm tends to generate a zero scaling factor on average, that is, as the average of a large set of computations on an uncorrelated data set. Computed scaling factors are not always exactly zero because there are other sources of noise in the images, which have been neglected in the expressions above. Note that this neglected noise is precisely what the algorithm evaluates in order to find k. This uncertainty can yield odd results eventually, such as negative scaling factors.

The next version of ImageCalibration will issue a warning when this happens. The advantage of our implementation is that you can always know what happens with your data. This is just what we want to achieve with PixInsight.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline vicent_peris

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 986
    • View Profile
    • http://www.astrofoto.es/
Quote
The conceptual error, in my opinion, is in the fact that a CCD does not produce negative values in the range [-32768,0[, so storing physical pixel values as signed integers does not make any sense.
Indeed, and AA4 reports values in the range of 0 to 891 for the bias frame. It is beyond me why it decided to save flats, darks, and lights as 16bit unsigned and bias as signed 16bit.

Quote
So Astroart does calculations in 15 bit, as does Iris??
No, it just has a peculiar way of saving data.

BR
Tasos

No. In Iris, you can manipulate the image between 0 and 32768. :) It does 15 bit calculation... at least some time ago...

V.

Offline harist

  • Newcomer
  • Posts: 46
    • View Profile
Hi Harry,

When BZERO is inserted by AA4 then the files are OK in PI (flats, lights, darks). What I do not understand is why this is not the case with bias frames (no BZERO inserted there by AA4).

On another matter when I calibrated using FLATS (with very high values I have to admit), PI could not align the calibrated files. Following your video tutorial I applied the "shift" trick to destroy the color info and everything was OK. I suppose a future version of IC could have this option for an OSC MasterFlat frame (Discard color info).

Tasos

Offline Yuriy Toropin

  • PixInsight Addict
  • ***
  • Posts: 209
    • View Profile
...
showing that there is definitely no correlation at all between your flat frames and the master dark frame. We can conclude that you should not apply this master dark frame to your flats.
...
The next version of ImageCalibration will issue a warning when this happens. The advantage of our implementation is that you can always know what happens with your data. This is just what we want to achieve with PixInsight.
Thanks a lot for checking this and for clarifications,
just another example that "classical" approach to calibration will fail in some cases!

Warning will be helpfull, but even now log in Processing Console gives enough insights on what's going on.

Thanks a lot again, Juan, I (and, IMHO, the whole PI community) really appreciate all those pieces of really valuable info on different aspects of processing you're providing on the forum!

Offline mattssporre

  • Newcomer
  • Posts: 20
    • View Profile
    • Spur Photography

The dark optimization routines implemented in ImageCalibration work on the basis that the dark frame is correlated to the target frames with respect to thermal noise. Assuming negligible read noise and neglecting other sources of noise in the images, consider the following expressions:

F = S + NT
D = k*NT

where F is a target frame (being calibrated), S represents the signal in the target frame, NT is the thermal noise in the target frame, and D is the dark frame. Dark frame optimization tries to find the scaling factor k. It works by computing the dark-subtracted target frame F':

F' = F - 1/k * D

such that the standard deviation of the noise in F' is minimal as a function of k.

When there is no shared NT between F and D, the function minimization algorithm tends to generate a zero scaling factor on average, that is, as the average of a large set of computations on an uncorrelated data set. Computed scaling factors are not always exactly zero because there are other sources of noise in the images, which have been neglected in the expressions above. Note that this neglected noise is precisely what the algorithm evaluates in order to find k. This uncertainty can yield odd results eventually, such as negative scaling factors.

The next version of ImageCalibration will issue a warning when this happens. The advantage of our implementation is that you can always know what happens with your data. This is just what we want to achieve with PixInsight.


Juan,

I am a little confused here ???. My notation

D = Dark signal
Nd = Noise in the dark frame (or uncertaintaty in the obtained dark frame which we use to measure the dark signal).

Dark Frame = D + Nd

S = the light signal, the one that we try to extract.
Ns = uncertainty in the obtained light frame (also referred to as noise)

Light Frame = S + D + Ns

I should also include the offset signal (bias) and vignette factors but they are not important for what I want to ask.

If we have the exact time and temp for the Dark Frame we can subtract the Dark Frame from the Light Frame and get rid of the Dark signal, D, but the dark noise will not go away.

Light Frame – Dark Frame = S + Sqrt[Nd*Nd + Ns*Ns]

Now if the Dark Frame and the Light Frame does not match in time and/or temperature you have to scale the Dark Frame in some way. Assume the light frame is taken at temperature T0 and with exposure t0. Also assume that the corresponding time and temp for the dark frame is t1 and T1

Light Frame – Dark Frame = S(t0,T0) + D(t0,T0) – k*D(t1,T1) + Sqrt[Nd*Nd + Ns*Ns]

We need to find k, but what I hear you say is that you find this scaling by trying to minimize the noise in the resulting Light Frame – Dark Frame, i.e. by minimising Sqrt[Nd*Nd+Ns*Ns].

But how does this secure that we actually find a k such that D(t0,T0) – k*D(t1,T1) = 0?

BR
Matts
PS sorry for this long and technical premiere post at this forum, but I may have misunderstood something and then there is a chance to learn something  :)

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2173
  • Join the dark side... we have cookies
    • View Profile
    • http://www.astrophoto.cl
Hi Matt.

If you take lots of dark frames, the final master dark will have a negligible amount of noise, so, so the sake of simplification, one may discard it. Once you have only noise from the light frame, at the end it is unidentifiable from the signal, so, again for simplicity, you may assume that everything is signal. Also, there is no correlation between dark current and noise or signal, so at the end it is the same thing.

In the real world, of course, things are not that pretty, but even so if you minimize the noise (or standard deviation of small samples, if you like), it is a better result than a theoretical linear escalation.
Minimizing noise guarantees to find a value for k if there is some correlation between light and dark frames (the dark current itself should be the correlation). This happens because the subtraction should follow a function such at +-infinity the noise goes to +infinity. Since it is a continuous function, there should be at least one local minimum that is a solution to the problem.

[Juan, feel free to overdrive anything I said :D]
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline mattssporre

  • Newcomer
  • Posts: 20
    • View Profile
    • Spur Photography
Carlos,

I understand that the noise (or rather uncertainty in the signal) gets smaller and smaller when adding more and more frames (dark frames in this case) :P.

The question I have is related to Juans statement that PI try to scale/optimize a master dark frame (to fit the Light Frame we would like to remove the Dark Signal from) by MINIMIZING the NOISE in the resulting calibrated Light Frame.

I just do not understand  ??? how minimizing the noise in the resulting calibrated Light Frame secures that we have removed the correct Dark Signal.

BR
Matts

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6782
    • View Profile
    • http://pixinsight.com/
Hi Matts,

Quote
how minimizing the noise in the resulting calibrated Light Frame secures that we have removed the correct Dark Signal.

Because:

(i) Uncertainties in both the light frame and the master dark frame, as well as the signal in the light frame, remain constant during the whole dark scaling process. Since they remain constant, we can simply ignore all of them them during the whole process.

(ii) Dark current, as recorded in the master dark frame, consists exclusively of small-scale structures. By small-scale structures, we refer to pixel-to-pixel variations. We use a wavelet transform to isolate these variations in the first wavelet layer, as image structures at the 1-pixel scale.

(iii) A necessary precondition is that the uncertainty in the measurement of the dark current is negligible in the master dark frame.

(iv) Another necessary precondition is that the light frame and the master dark frame are correlated. This means that both frames share the same dark signal, although scaled by a factor k > 0. Our algorithm tries to find the value of k.

Note that our algorithm is purely numeric. It is not based on any physical properties of the sensor. In fact, our algorithm ignores temperatures and exposure times completely, because it does not need to know them.

Note also that we treat dark signal as both signal and noise, at different stages of the algorithm. When you integrate a large number of dark frames into a master dark frame, dark current is signal. By averaging n dark frames, you reduce the uncertainty in its measurement by Sqrt(n). However, when our dark scaling algorithm evaluates noise in the light frame after dark subtraction, the same dark current is noise. This is the difficult part because of the duality in the interpretation of the same part of the data. I'll try to explain this in more detail.

Following your notation, we have:

Master Dark Frame = DF = k*D + Nd
Light Frame = LF = S + D + Ns

where D is the dark signal, S is the image signal, Nd is the noise in the master dark frame and Ns is the noise in the light frame. Both Nd and Ns are supposed to be randomly valued and uniformly distributed. Our task is to find a good approximation to the true dark scaling factor k > 0.

Precondition (iii) says that the uncertainty in the measurement of D is negligible, that is:

|D| >> |Nd|

so we can assume:

Nd = 0

without changing anything important. This means that the quality of the master dark frame (in SNR terms) is very important, a fact that Vicent has stressed sufficiently in his tutorial. Note that this is also extremely important in our numerical procedure because, as Nd is uncertain by nature, we have no way to remove it, so each time we try out a scaling factor k we'll be multiplying also k*Nd. If Nd and D are comparable in magnitude, they may easily become undistinguished numerically, and our process will have no way to converge to a good (that is, certain) value of k.

Precondition (ii) tells us that in the morphological and statistical senses, D is very similar to Ns. Both features are composed of small-scale structures. Ns is random and can be assumed to follow a Gaussian distribution, as usually happens with large data sets having a strong central tendency. D also has a strong central tendency, since most dark current variations are quite similar. We know that D is not a random variable, but observed over a large portion of the image, its distribution is basically uniform. Uniform and random here, while not the same thing, can be treated in the same way because its properties are essentially the same for our strictly numerical purposes.

Now let's try out a value of k, call it ki:

LFi = LF - 1/ki * DF

We have said that we can neglect Nd, so we have:

LFi = S + D - k*D/ki + Ns

The above expression is evaluated iteratively to feed a minimization algorithm. At each iteration, the algorithm computes the standard deviation of the noise in LFi. When the algorithm brackets a minimum within a very small interval (< 0.005 in the current implementation), the current value of ki is returned as the true value of k.

As S and Ns remain unchanged during the whole process, what we are minimizing is actually the standard deviation of:

D - k*D/ki + Ns

Note that our multiscale noise evaluation algorithm is able to isolate S from the rest of terms, and it performs that task in a very robust way.

As I've said above, here we are treating D as if it were pure noise. As we approximate ki = k, the standard deviation of the noise in LFi reaches a minimum, which is approximately equal to the standard deviation of Ns. As Carlos has pointed out, the function to be minimized is strictly continuous and has a single minimum. An important fact that must also be pointed out, is that our algorithm makes no assumption as for the linearity of the dark signal as a function of temperature, exposure time, or any other physical acquisition condition. In fact, the dark signal could exhibit a wildly nonlinear behavior, and our algorithm would still find the optimum scaling factor k.

Hope this helps you and all users to understand how our algorithm works. It is not perfect, and we indeed have some ideas to improve it, but we think it works remarkably well.
« Last Edit: 2010 April 19 08:16:42 by Juan Conejero »
Juan Conejero
PixInsight Development Team
http://pixinsight.com/