Author Topic: For beginners: Guide to PI's ImageCalibration  (Read 2798 times)

Offline bulrichl

  • PixInsight Addict
  • ***
  • Posts: 275
    • View Profile
For beginners: Guide to PI's ImageCalibration
« on: 2017 December 27 19:03:48 »
Guide to PixInsight's ImageCalibration (revised on 26th February 2018)

I observe that many PixInsight newcomers struggle with ImageCalibration and don't get on. Similar questions are put in the forum again and again. Whereas finally the questioner may have gotten the desired solution, unfortunately there is often no feedback given. Then the thread is open-ended and comes to nothing. Such threads are not helpful at all for beginners.

So this guide was generated with the goal in mind to provide a general help about the usage of PixInsight's ImageCalibration for novices. Please don't answer to this post. When you feel that an important point is missing or there is something wrong, please send me a private message. If reasonable I will supplement or correct my description.

Please, keep in mind: PI's ImageCalibration is very powerful and flexible, but it does not execute many checks whether your settings are reasonable. Some settings will even yield wrong results. The old wisdom applies here: "garbage in, garbage out". You in person are responsible for the right settings. There is no reason to put a bug report about ImageCalibration when your calibration result looks strange - just take a look at your settings.

The goal of well calibrated light frames can be achieved in different ways. That is the reason why you may read different recommendations for the preparation of master calibration frames and the calibration procedure. Since there is a wealth of different cameras, some of these recommendations may work well for one configuration and fail for another. It was my goal to describe a procedure that (hopefully) will work generally. Therefore, my recommendations may differ from approaches recommended elsewhere.

In my description I intentionally do not mention Overscan calibration because this is a specialty of some dedicated astro cameras. I guess that people who use such cameras know what they are doing.

--------------------------------------------------

1 General Settings
1.1 Digital Single Lens (DSLR) cameras
If you use a DSLR camera which is able to save the images in a proprietary raw format (e.g. Canon's CR2, Nikon's NEF or Sony's ARW [and other formats]), set your camera to use the raw format. In this case, in PI's Format Explorer set DSLR_RAW (respectively newly: RAW) to 'Pure raw'. In the case of images of a DSLR camera the Color Filter Array (CFA) pattern (e.g. RGGB or GBRG) has been witten to the file header and is available for the Debayer process.

1.2 One Shot Color (OSC) cameras
The camera driver provides an image in FITS format. You should check once that the image that is displaed in PI is true sided. If not, change the setting 'Coordinate origin' in PI's Format Explorer|FITS. The default is 'Upper left corner (Up-bottom)'.
However, watch out: if you change this setting, all calibration files will have to be newly generated with the very same setting as well in order that the calibration will yield correct results.
The CFA pattern is not saved in the FITS file, so you have to explicitely specify it when debayering (see camera handbook).

1.3 DSLR and OSC cameras: using of CFA images for calibration
If you use a DSLR or a OSC camera you want to perform the entire calibration process with monochrome CFA images. Only after the calibration process (including CosmeticCorrection) is completed, the calibrated light frames are debayered (to yield RGB images), then registered and finally integrated.

1.4 Monochrome cameras
If you use a monochrome camera, separate flat frames (and MasterFlats) have to be generated for each filter.

1.5 Acquisition software and file format
Use the same software for the acquisition of light frames and calibration frames, and the same file format for all frames.


2 Conditions for the Acquisition of Calibration Frames
2.1 Temperature
For cameras without temperature control: try to take the dark frames at the same ambient temperature as the light frames. For cameras with temperature control: use the same set value for all frames.

2.2 Camera settings
Use the same camera settings for the acquisition of dark frames that were used for the light frames. Relevant settings for DSLR cameras are: ISO and exposure time, for astro cameras: gain, (if applicable) offset, and exposure time.
Also, for flat frames and the frames used for the calibration of the flat frames, identical settings should be used.

2.3 Unchanged light path for the acquisition of flat frames
For flat frame acquisition it is all-important to have an unaltered light path, i. e. the same flattener or reducer, the same camera orientation (rotating angle) and focus position as when taking the light frames. Best is not to change anything and take the flat frames directly before or after the light frames. With refractors it is usually possible to use one MasterFlat for some longer time.


3 Generation of the Master Calibration Frames
It is advisable to prepare (according to [1]) and then check the master calibration frames before the light frame calibration is executed.

3.1 MasterDark and MasterBias
It is assumed that the MasterDark and MasterBias are prepared according to [1].

3.2 MasterFlat
It is assumed, that the MasterFlat is prepared by calibrating the flat frames and integrating the calibrated flat frames according to [1].
Depending on equipment and the used method of flat frame acquisition, some people consider it favorable not to use the option of 'Dark frame optimization' in the calibration of the flat frames. Instead of the MasterDark with dark frame optimization they will use either
- a MasterFlat-Dark without dark frame optimization or
- a MasterBias.
In case of long exposure time of the flat frames (e. g. with narrow band filters) or a camera with high noise or with "amplifier glow", a MasterFlat-Dark may be favorable. In case of short exposure time of the flat frames and a camera with low noise, without "amplifier glow", a MasterBias will do well. Just try, which approach is best in your case.


4 Potential pitfalls in calibration
4.1 Correct subtraction of the bias level
In [3], Juan Conejero states:

Quote
Most [dark frame optimization] problems [(such as 'no correlation')] happen because the bias level ist not correctly subtracted from one or more calibration frames; usually from the master dark frame.

I agree, but would not confine this statement to the usage of dark frame optimization - it holds generally. In the same thread, Juan specifies the three possibilities of subtracting bias correctly:

Quote
(1) You have simply integrated the individual dark frames to generate the master dark frame. In this case the master dark frame does have a bias pedestal. This is the most usual procedure, and also the one shown in Vicent's tutorial.

(2) After (1), you have calibrated the master dark frame with ImageCalibration to subtract the master bias frame. In this case the master dark frame does not have a bias pedestal.

(3) You have calibrated the individual dark frames by subtracting the master bias frame from each of them, before integrating them to generate the master dark frame. In this case the master dark frame does not have a bias pedestal, as in (2). This procedure is atypical.


Possibility (3), which Juan denotes as 'atypical', is the cumbersome version of (2): being mathematically equivalent to (2), it will produce identical results. There is just some (unnecessary) arithmethic involved.

With both (2) and (3), the subtraction of bias is executed in a preliminary step. I will denote this procedure as 'Pre-calibration'. This approach suffers from a serious drawback with modern DSLR and CMOS cameras: in the calibrated MasterDark severe clipping (truncating of negative values) occurs. The cause of this is explained in the following sections.

4.2 "Dark Current Suppression" and its impact for calibration
Modern DSLR or CMOS cameras have a dark current suppression mechanism (in the hardware) that subtracts dark current. Thus dark frames and bias frames have similar average intensities, almost independent of the exposure time of the dark frames! Subtraction of the MasterBias from the dark frames (or from the MasterDark) results in negative values for a good portion of the pixels. If this subtraction is carried out in a preliminary step, all negative values will be truncated (clipped), and these data are lost. I experienced this situation both with a calibrated MasterDark of my Canon EOS 600D (= Rebel T3i) and as of late with a calibrated MasterDark of my brand-new ZWO ASI294MC Pro. Image 1 shows a screen dump of the histograms: on the left side the not pre-calibrated MasterDark, on the right side the pre-calibrated MasterDark (here: of a Canon EOS 600D = Rebel T3i), in which about half of the peak is truncated (set to zero). If such a pre-calibrated MasterDark is used for light frame calibration, only about half of the pixels in the light frame are calibrated correctly. The rest of the pixels (in this example: the other half) is not corrected for dark current at all; only the bias is subtracted here! The consequence of using such a pre-calibrated MasterDark is higher noise in the calibrated lights.

4.3 Truncation of negative values during a calibration process
Why are negative values simply truncated?
In [4], reply #2, Juan Conejero explained in detail the necessity of truncation of negative values in the calibration process. In short: for the calibration process a coherent data set must be used. However, the truncation of negative values is NOT applied to intermediate results of a calibration step, but rather
Quote
Truncation to the [0,1] range is carried out as the very last step of the calibration task for each frame, i.e. after overscan, bias, dark, flat and pedestal correction.

In [4], reply #2, Juan Conejero suggested the addition of a pedestal in order to avoid the issue of data loss by truncation of negative values during calibration. However, this is unnecessary. As you will see in section 5.1, there is no need to use pedestals when you
1) don't precalibrate neither dark frames nor the MasterDark and
2) calibrate the master dark (if necessary) only during light frame calibration.

4.4 Pre-calibration of the MasterDark?
This means: When the MasterDark is pre-calibrated, severe clipping occurs in the calibrated MasterDark. However, when using PI's option of calibrating the MasterDark only during light frame calibration, negative values in the intermediate result (the calibrated MasterDark) will not be clipped.


The bottom line from sections 4.1 to 4.4 therefore is:
NEVER pre-calibrate your dark frames, neither dark subframes nor the MasterDark. If necessary, use instead PixInsight's option of calibrating the MasterDark during calibration of the light frames.

--------------------------------------------------
There are tutorials that suggest to pre-calibrate dark frames or the MasterDark AND don't even give a reference to truncation of negative values (even the otherwise excellent tutorial [5] does). This is bad advice. For modern DSLR or CMOS cameras which apply dark current suppression such a calibration approach will result in a lower SNR of the calibrated light frames. Fortunately better instructions can be found, e.g. [6] and [7].
--------------------------------------------------


5 Light Frame Calibration with PI's ImageCalibration
Select the ImageCalibration process and load the light frames by 'Add Files'.

5.1 Dark frame optimization
Once you decided to not pre-calibrate the MasterDark, things become quite simple. In fact, for light frame calibration there are only two settings left, that will lead to a correct subtraction of the bias level. These are the two cases which have to be differentiated:

Case 1: Calibration WITH dark frame optimization
The general form of the applied calculation is represented in equation {1}:

Code: [Select]
Cal1 = ((LightFrame - MasterBias) - k0 * (MasterDark - MasterBias)) / MasterFlat * s0   (WITH dark frame optimization) {1}
where k0: dark scaling factor,
      s0: master flat scaling factor (= Mean of the MasterFlat)

Conclusion: The only problematic term (concerning negative values) in equation {1} is: (MasterDark - MasterBias). When the calibration of the MasterDark is executed only during light frame calibration, the term (MasterDark - MasterBias) is an intermediate result, accordingly negative values will not be truncated. For the calibration of the light frames with dark frame optimization, use the settings in Image 2, right side.

Settings:
a) A MasterBias is needed, so check section 'Master Bias' and select the MasterBias.
b) Check section 'Master Dark' and select the MasterDark. Check both options 'Calibrate' and 'Optimize'.

Note that in dark frame optimization PI uses neither temperature nor exposure time to evaluate k0. PI optimizes k0 for lowest noise in the resulting calibrated light frame [2].


Case 2: Calibration WITHOUT dark frame optimization
Without dark frame optimization, k0 equals 1.0, therefore MasterBias is canceled out from equation {1}. Thus equation {1} is simplified to equation {2}:

Code: [Select]
Cal2 = (LightFrame - MasterDark) / MasterFlat * s0                      (NO dark frame optimization) {2}
Conclusion: The term that could be problematic concerning negative values, (MasterDark - MasterBias), does not appear in equation {2}. There is no MasterBias at all. For a light frame calibration without dark frame optimization a MasterBias is not needed, because the MasterDark is not calibrated at all. Accordingly, a truncation of negative values cannot occur. For the calibration of light frames use the settings in Image 2, left side.

Settings:
a) Neither bias frames nor a MasterBias is needed, so uncheck section 'Master Bias'.
b) Check section 'Master Dark' and select the MasterDark. Uncheck both options 'Calibrate' and 'Optimize'.


Whether it is favorable to use dark frame optimization or not depends on the camera. Using dark frame optimization for a camera without temperature control may greatly improve the calibration result. This is because temperature deviations between dark and light frame acquisition are unavoidable in this case. For a camera with temperature control the benefit will be much lower, possibly even not detectable.

However, if the camera shows "amplifier glow", this might not be calibrated out completely with dark frame optimization enabled. In cases of a not temperature controlled camera with "amplifier glow" it's worthwhile to accurately compare the results of both settings.

5.2 Section 'Master flat'
Settings:
Check the 'Master flat' section and select your MasterFlat. Since the flat frames are already (according to [1]) calibrated and then integrated to a MasterFlat, uncheck 'Calibrate' in the 'Master flat' section.

5.3 ImageCalibration's output to Process Console
After having adjusted the necessary settings for light frame calibration, the process can be executed. During or after the execution, observe the output to the process console. The output may look like the following example (extract):

--------------------------------------------------
Applying bias correction: master dark frame ...

Dark frame optimization thresholds:
Td0 = 0.00179381 (733282 px = 4.068%)
Computing master flat scaling factors ...
s0 = 0.093233                           (1) <==
...
Writing output file: F:/Astro/171217/Calibrated/Light_360SecISO800_092718_c.xisf
Dark scaling factors:
k0 = 0.985                              (2) <==
Gaussian noise estimates:
s0 = 1.687e-03, n0 = 0.911 (MRS)
...

--------------------------------------------------

(1):
The master flat scaling factor s0 is the mean of the MasterFlat.

(2) (Only when dark frame optimization is enabled):
k0 is the dark scaling factor, optimized for lowest noise in the resulting calibrated light frame.

5.4 Approved settings for light frame calibration (examples from personal experience)
Canon EOS 600D = Rebel T3i (no temperature control, no "amplifier glow"):
Using this DSLR without temperature control, the result of the light frame calibration was greatly improved by using dark frame optimization. Because the 600D doesn't show "amplifier glow" in the dark frames, the MasterBias was used for flat frame calibration.

- Take dark frames, bias frames and flat frames,
- integrate the dark frames to a not pre-calibrated MasterDark,
- integrate the bias frames to a MasterBias (and, if applicable, make superbias),
- calibrate the flat frames with the MasterBias (or superbias) and integrate to a MasterFlat,
- calibrate the light frames using the settings shown above in Image 2, right side.


ZWO ASI294MC Pro (with temperature control, with "amplifier glow"):
Using this CMOS camera with temperature control, I did not notice an improvement of the light frame calibration result by using dark frame optimization. The "amplifier glow" tended not to calibrate out completely with dark frame optimization enabled. So I decided not to use it. The "amplifier glow" as well was the reason for using a MasterFlat-Dark for flat frame calibration.

- Take dark frames, flat-dark frames and flat frames,
- integrate the dark frames to a not pre-calibrated MasterDark,
- integrate the flat-dark frames to a not pre-calibrated MasterFlat-Dark,
- calibrate the flat frames with the MasterFlat-Dark and integrate to a MasterFlat,
- calibrate the light frames using the settings shown above in Image 2, left side.


6 After Light Frame Calibration
6.1 CosmeticCorrection
The calibration will leave some hot pixels in the calibrated light frames. This is normal because hot pixels don't behave ideally. These remaining hot pixels must be corrected with the CosmeticCorrection process directly after completed calibration process.

6.2 Debayer
In case of a DSLR or an OSC camera, the calibrated light frames have to be debayered now, thus yielding RGB images. For OSC cameras, the CFA pattern has to be specified explicitely.

6.3 Registration
The images are then registered by the StarAlignment process.

6.4 Integration
Finally, the registered images are integrated by the ImageIntegration process.

Thereby, data reduction has been completed.

==================================================

By now I have searched in the PixInsight forum and found the following interesting threads ([8] and [9]) that I was not aware of previously. So I discovered that my recommendations are anything but new:

The forum user Ignacio used and suggested the approach with dark frame optimization (Case 1, see [8] and [9]), and the forum user astropixel used and suggested the approach without dark frame optimization, referred to as 'bias_in_the_dark (BITD) method' (Case 2, see [8]). This all was written already in 2014!

==================================================


--------------------------------------------------

Sources:

[1] Tutorial by Vicent Peris: "Master Calibration Frames: Acquisition and Processing"
https://www.pixinsight.com/tutorials/master-frames/

[2] Juan Conejero: "Dark Frame Optimization Algorithm"
https://pixinsight.com/forum/index.php?topic=8839.0

[3] Juan Conejero in: "Warning: No correlation between the master dark and target frames", Replies #2 and #4
https://pixinsight.com/forum/index.php?topic=3286.0

[4] "Image Calibration and negative values (KAF 8300)", https://pixinsight.com/forum/index.php?topic=6822.0

[5] Kayron Mercieca, "2. Generating a Master Superbias and a Master Dark" and "4. Calibrating Lights and Correcting Hot and Cold Pixels"
http://www.lightvortexastronomy.com/tutorial-pre-processing-calibrating-and-stacking-images-in-pixinsight.html#Section2

[6] David Ault, "PixInsight Manual Image Calibration, Registration and Stacking"
http://trappedphotons.com/blog/?p=693

[7] Warren A. Keller, "Inside PixInsight", Springer (2016)

[8] "Preprocessing Canon DSLR frames - a different approach", https://pixinsight.com/forum/index.php?topic=7006.0

[9] "Bias frames - pixel rejection - temperature regulated DSLR", https://pixinsight.com/forum/index.php?topic=6980.0

--------------------------------------------------
« Last Edit: 2018 February 26 15:16:22 by bulrichl »

Offline Alfredo Beltran

  • PixInsight Addict
  • ***
  • Posts: 107
    • View Profile
Re: For beginners: Guide to PI's ImageCalibration
« Reply #1 on: 2018 October 10 17:43:34 »
Very good guide for beginners and for experienced users as well. Thanks for posting it
iOptron CEM60
Celestron 925 EdgeHD Hyperstar
MicroTouch Focuser
Canon T1i modified by Brent Oliver
Bogota, Colombia
http://www.flickr.com/photos/alfredo_beltran/

Offline russp

  • Newcomer
  • Posts: 15
    • View Profile
Re: For beginners: Guide to PI's ImageCalibration
« Reply #2 on: 2018 October 20 08:28:11 »
Great guide - very useful, can I ask what debayer pattern you're using for your ASI294 ? I've been using RGGB but I'm not entirely sure it's correct as it's giving me really noisy blotchy backgrounds even after DBE.