Guide to PixInsight's ImageCalibration (revised on 31st January 2020)
I observe that many PixInsight newcomers struggle with ImageCalibration and don't get on. Similar questions are put in the forum again and again. Whereas finally the questioner may have gotten the desired solution, unfortunately there is often no feedback given. Then the thread is open-ended and comes to nothing. Such threads are not helpful at all for beginners.
So this guide was generated with the goal in mind to provide a general help about the usage of PixInsight's ImageCalibration for novices. Please don't answer to this post. When you feel that an important point is missing or there is something wrong, please send me a private message. If reasonable I will supplement or correct my description.
Please, keep in mind: PI's ImageCalibration is very powerful and flexible, but it does not execute many checks whether your settings are reasonable. Some settings will even yield wrong results. The old wisdom applies here: "garbage in, garbage out". You in person are responsible for the right settings. There is no reason to put a bug report about ImageCalibration when your calibration result looks strange - just take a look at your settings.
The goal of well calibrated light frames can be achieved in different ways. That is the reason why you may read different recommendations for the preparation of master calibration frames and the calibration procedure. Since there is a wealth of different cameras, some of these recommendations may work well for one configuration and fail for another. It was my goal to describe a procedure that (hopefully) will work generally. Therefore, my recommendations may differ from approaches recommended elsewhere.
In my description I intentionally do not mention Overscan calibration because this is a specialty of some dedicated astro cameras. I guess that people who use such cameras know what they are doing.
--------------------------------------------------
0 Specific properties of sensors and cameras used for astrophotographyBoth regular digital cameras (e.g. Digital Single-Lens Reflex (DSLR) cameras or Digital Single Lens Mirrorless (DSLM) cameras) and dedicated astro cameras using Charge-Coupled Device (CCD) or Complementary Metal-Oxide-Semiconductor (CMOS) sensors can be used for astrophotography.
It is important to differentiate between monochrome and One Shot Color (OSC) sensors. In OSC sensors, each individual photosite of the sensor is equipped with a color filter. Usually 3 (sometimes 4) different colors are used for the color filters. Each photosite is exposed only to the light transmitted through its color filter. The different colors are arranged in a periodic pattern on the sensor, called Color Filter Array (CFA) pattern, e.g. a Bayer pattern. The raw bayered/mosaiced data of an OSC camera are not color images. The color in CFA data has to be reconstructed by an interpolation algorithm called "Debayering" or "Demosaicing" which transforms the CFA data to RGB images. In contrast, the photosites of a monochrome sensor are not equipped with color filters.
Regular digital cameras normally are OSC cameras. There are dedicated astro cameras in monochrome and in OSC versions.
In order to reduce thermal noise, dedicated astro cameras normally are equipped with a cooling system with temperature control whereas regular digital cameras normally are not cooled.
1 General Settings1.1 Special recommendations for OSC cameras
When using an OSC camera (be it a regular digital camera or a dedicated astro camera) you generally want to perform the entire calibration process with raw CFA data. Only after the calibration process (including CosmeticCorrection) is completed, the calibrated light frames are debayered (to yield RGB images), then registered and finally integrated.
1.1.1 Regular digital cameras
If you use a regular digital camera which is able to save the data in a proprietary raw format (e.g. Canon's CR2, Nikon's NEF or Sony's ARW [and other formats]), set your camera to use this raw format.
The capturing software normally lets you choose whether the files will be saved to disk in the proprietary raw format or in FITS format (or both of them). The FITS format contains some useful metadata that are not stored in the proprietary raw format (e.g. the name and the coordinates of the object, the focus position, etc.). However, the proprietary raw format results in somewhat smaller file size. Dependent on the chosen file format there is a difference in the data: the proprietary raw format contains intensity values in the bit depth of the analog digital converter (ADC), e.g. for a 14-bit ADC in the range of 0 to 2^14 - 1 = 16383. In contrast, the intensity data in a FITS file are scaled to 16 bit, i.e. in the range of 0 to 2^16 - 1 = 65535. This is the reason that the use of different software for the acquisition of light frames and calibration frames or the use of different file formats for light frames and calibration frames may lead to wrong results. So follow the advice given in 1.2.
If you decide to let the files be saved to disk in the proprietary raw format, set the Raw Format Preferences in PixInsight (Format Explorer, double click on 'RAW') to 'Pure Raw'. The proprietary raw format contains the CFA pattern. When opening the file in PixInsight, the raw image decoding software 'LibRaw' that is used by PixInsight's RAW module detects the CFA pattern and makes it available for the Debayer process.
If you decide to let the files be saved to disk in FITS format, see 1.1.2.
1.1.2 Dedicated astro cameras in OSC version
The camera driver provides the image data in FITS format. The CFA pattern is not necessarily saved in the FITS header. Some acquisition software writes the non-standard FITS keywords 'BAYERPAT', 'XBAYROFF' and 'YBAYROFF' which are supported by PixInsight as well. If the FITS keyword 'BAYERPAT' is not written to the FITS header, you have to explicitely specify the CFA pattern when executing the Debayer process (see camera handbook).
1.2 Acquisition software and file format
Always use the same software for the acquisition of light frames and calibration frames, and the same file format for all frames.
2 Conditions for the Acquisition of Calibration Frames2.1 Temperature
For cameras without cooling system: try to take the dark frames at the same ambient temperature as the light frames. For cameras with cooling system: use the same set value for all frames.
2.2 Camera settings
For the acquisition of light frames and ALL sorts of calibration frames, the same ISO (in case of a regular digital camera) or gain and offset (in case of a dedicated astro camera) must be used. Dark frames should also match the exposure time of the light frames, and flat-darks should match the exposure time of the flat frames.
2.3 Unchanged light path for the acquisition of flat frames
For flat frame acquisition it is all-important to have an unaltered light path, i. e. the same flattener or reducer, the same camera orientation (rotating angle) and focus position as when taking the light frames. Best is not to change anything and take the flat frames directly before or after the light frames. With refractors it is usually possible to use one MasterFlat for some longer time.
3 Generation of the Master Calibration FramesIt is advisable to prepare (according to [1]) the master calibration frames and check them before the light frame calibration is executed.
3.1 MasterDark and MasterBias
It is assumed that MasterDark and MasterBias are prepared according to [1].
3.2 MasterFlat
It is assumed, that the MasterFlat is prepared by calibrating the flat frames and integrating the calibrated flat frames according to [1].
Depending on equipment and the used method of flat frame acquisition, some people consider it favorable not to use the option of 'Dark frame optimization' in the calibration of the flat frames. Instead of the MasterDark with dark frame optimization they will use either
- a MasterFlat-Dark without dark frame optimization or
- a MasterBias.
In case of long exposure time of the flat frames (e. g. with narrow band filters) or a camera with high noise or with "amplifier glow", a MasterFlat-Dark may be favorable. In case of short exposure time of the flat frames and a camera with low noise, without "amplifier glow", a MasterBias will do well. Just try, which approach is best in your case.
If you use a monochrome camera, separate flat frames (and MasterFlats) have to be generated for each filter.
4 Potential pitfalls in calibration4.1 Correct subtraction of the bias level
In [3], Juan Conejero states:
Most [dark frame optimization] problems [(such as 'no correlation')] happen because the bias level ist not correctly subtracted from one or more calibration frames; usually from the master dark frame.
I agree, but would not confine this statement to the usage of dark frame optimization - it holds generally. In the same thread, Juan specifies the three possibilities of subtracting bias correctly:
(1) You have simply integrated the individual dark frames to generate the master dark frame. In this case the master dark frame does have a bias pedestal. This is the most usual procedure, and also the one shown in Vicent's tutorial.
(2) After (1), you have calibrated the master dark frame with ImageCalibration to subtract the master bias frame. In this case the master dark frame does not have a bias pedestal.
(3) You have calibrated the individual dark frames by subtracting the master bias frame from each of them, before integrating them to generate the master dark frame. In this case the master dark frame does not have a bias pedestal, as in (2). This procedure is atypical.
Possibility (3), which Juan denotes as 'atypical', is the cumbersome version of (2): being mathematically equivalent to (2), it will produce identical results. There is just some (unnecessary) arithmethic involved.
With both (2) and (3), the subtraction of bias is executed in a preliminary step. I will denote this procedure as 'Pre-calibration'. This approach suffers from a serious drawback with modern DSLR and CMOS cameras: in the calibrated MasterDark severe clipping (truncating of negative values) occurs. The cause of this is explained in the following sections.
4.2 "Dark Current Suppression" and its impact for calibration
Modern DSLR or CMOS cameras have a dark current suppression mechanism (in the hardware) that subtracts dark current. Thus dark frames and bias frames have similar average intensities, almost independent of the exposure time of the dark frames! Subtraction of the MasterBias from the dark frames (or from the MasterDark) results in negative values for a good portion of the pixels. If this subtraction is carried out in a preliminary step, all negative values will be truncated (clipped), and
these data are lost. I experienced this situation both with a calibrated MasterDark of my Canon EOS 600D (= Rebel T3i) and as of late with a calibrated MasterDark of my brand-new ZWO ASI294MC Pro.
Image 1 shows a screen dump of the histograms: on the left side the not pre-calibrated MasterDark, on the right side the pre-calibrated MasterDark (here: of a Canon EOS 600D = Rebel T3i), in which about half of the peak is truncated (set to zero). If such a pre-calibrated MasterDark is used for light frame calibration, only about half of the pixels in the light frame are calibrated correctly. The rest of the pixels (in this example: the other half) is not corrected for dark current at all; only the constant bias offset is subtracted here! The consequence of using such a clipped, pre-calibrated MasterDark is higher noise in the calibrated lights.
4.3 Truncation of negative values during a calibration process
Why are negative values simply truncated?
In [4], reply #2, Juan Conejero explained in detail the necessity of truncation of negative values in the calibration process. In short: for the calibration process a
coherent data set must be used. However, the truncation of negative values is NOT applied to
intermediate results of a calibration step, but rather
Truncation to the [0,1] range is carried out as the very last step of the calibration task for each frame, i.e. after overscan, bias, dark, flat and pedestal correction.
In [4], reply #2, Juan Conejero suggested the addition of a pedestal in order to avoid the issue of data loss by truncation of negative values during calibration. However, this is unnecessary. As you will see in section 5.1, there is no need to use pedestals when you
1) don't precalibrate neither dark frames nor the MasterDark and
2) calibrate the master dark (if necessary) only during light frame calibration.
4.4 Pre-calibration of the MasterDark?
This means: When the MasterDark is pre-calibrated, severe clipping occurs in the calibrated MasterDark. However, when using PI's option of calibrating the MasterDark only during light frame calibration, negative values in the intermediate result (the calibrated MasterDark) will not be clipped.
The bottom line from sections 4.1 to 4.4 therefore is:
NEVER pre-calibrate your dark frames, neither dark subframes nor the MasterDark. If necessary, use instead PixInsight's option of calibrating the MasterDark during calibration of the light frames.--------------------------------------------------
There are tutorials that suggest to pre-calibrate dark frames or the MasterDark AND don't even give a reference to truncation of negative values (even the otherwise excellent tutorial [5] does). This is bad advice. For modern DSLR or CMOS cameras which apply dark current suppression such a calibration approach will result in a lower SNR of the calibrated light frames. Fortunately better instructions can be found, e.g. [6] and [7].
--------------------------------------------------
5 Light Frame Calibration with PI's ImageCalibrationSelect the ImageCalibration process and load the light frames by 'Add Files'.
5.1 Dark frame optimization
Once you decided to not pre-calibrate the MasterDark, things become quite simple. In fact, for light frame calibration there are only two settings left, that will lead to a correct subtraction of the bias level. These are the two cases which have to be differentiated:
Case 1: Calibration WITH dark frame optimizationThe general form of the applied calculation is represented in equation {1}:
Cal1 = ((LightFrame - MasterBias) - k0 * (MasterDark - MasterBias)) / MasterFlat * s0 (WITH dark frame optimization) {1}
where k0: dark scaling factor,
s0: master flat scaling factor (= Mean of the MasterFlat)Conclusion: The only problematic term (concerning negative values) in equation {1} is: (MasterDark - MasterBias). When the calibration of the MasterDark is executed only during light frame calibration, the term (MasterDark - MasterBias) is an
intermediate result, accordingly negative values will not be truncated. For the calibration of the light frames with dark frame optimization, use the settings in
Image 2, right side.
Settings:
a) A MasterBias is needed, so check section 'Master Bias' and select the MasterBias.
b) Check section 'Master Dark' and select the MasterDark. Check both options 'Calibrate' and 'Optimize'.
Attention: The calibrated MasterDark is an intermediate result which is not saved to disk!
Note that in dark frame optimization PI uses neither temperature nor exposure time to evaluate k0. PI optimizes k0 for lowest noise in the resulting calibrated light frame [2].
Case 2: Calibration WITHOUT dark frame optimizationWithout dark frame optimization, k0 equals 1.0, therefore
MasterBias is canceled out from equation {1}. Thus equation {1} is simplified to equation {2}:
Cal2 = (LightFrame - MasterDark) / MasterFlat * s0 (NO dark frame optimization) {2}Conclusion: The term that could be problematic concerning negative values, (MasterDark - MasterBias), does not appear in equation {2}. There is no MasterBias at all. For a light frame calibration without dark frame optimization a MasterBias is not needed, because the MasterDark is not calibrated at all. Accordingly, a truncation of negative values cannot occur. For the calibration of light frames use the settings in
Image 2, left side.
Settings:
a) Neither bias frames nor a MasterBias is needed, so uncheck section 'Master Bias'.
b) Check section 'Master Dark' and select the MasterDark. Uncheck both options 'Calibrate' and 'Optimize'.
Whether it is favorable to use dark frame optimization or not depends on the camera. Using dark frame optimization for a camera without temperature control may greatly improve the calibration result. This is because temperature deviations between dark and light frame acquisition are unavoidable in this case. For a camera with temperature control the benefit will be much lower, possibly even not detectable.
However, if the camera shows "amplifier glow", this might not be calibrated out completely with dark frame optimization enabled. In cases of a not temperature controlled camera with "amplifier glow" it's worthwhile to accurately compare the results of both settings.
5.2 Section 'Master flat'
Settings:
Check the 'Master flat' section and select your MasterFlat. Since the flat frames are already (according to [1]) calibrated and then integrated to a MasterFlat, uncheck 'Calibrate' in the 'Master flat' section.
5.3 ImageCalibration's output to Process Console
After having adjusted the necessary settings for light frame calibration, the process can be executed. During or after the execution, observe the output to the process console. The output may look like the following example (extract):
--------------------------------------------------
Applying bias correction: master dark frame ...
Dark frame optimization thresholds:
Td0 = 0.00179381 (733282 px = 4.068%)
Computing master flat scaling factors ...
s0 = 0.093233 (1) <==
...
Writing output file: F:/Astro/171217/Calibrated/Light_360SecISO800_092718_c.xisf
Dark scaling factors:
k0 = 0.985 (2) <==
Gaussian noise estimates:
s0 = 1.687e-03, n0 = 0.911 (MRS)
...--------------------------------------------------
(1):
The master flat scaling factor s0 is the mean of the MasterFlat.
(2) (Only when dark frame optimization is enabled):
k0 is the dark scaling factor, optimized for lowest noise in the resulting calibrated light frame.
5.4 Checking of the calibration result, Output pedestal
After following the above suggestions, you should inspect some of the resulting calibrated light frames
- with HistogramTransformation:
check the histogram (adjust horizontal zoom that the histogram region around intensity 0 can be viewed, this might be the case at a horizontal zoom of about 50) and
- with ImageStatistics (option 'Unclipped' disabled):
check count (%), it should be very near 100 %. The fraction of clipped pixels equals (100 % - count (%)).
The important question is: are the calibrated light frames clipped? If this still is the case, you will have to apply an output pedestal in the ImageCalibration process (in the 'Output Files' section of ImageCalibration).
5.5 Approved settings for light frame calibration (examples from personal experience)
Canon EOS 600D = Rebel T3i (no temperature control, no "amplifier glow"):Using this DSLR without temperature control, the result of the light frame calibration was greatly improved by using dark frame optimization. Because the 600D doesn't show "amplifier glow" in the dark frames, the MasterBias was used for flat frame calibration.
- Take dark frames, bias frames and flat frames,
- integrate the dark frames to a not pre-calibrated MasterDark,
- integrate the bias frames to a MasterBias (and, if applicable, make superbias),
- calibrate the flat frames with the MasterBias (or superbias) and integrate to a MasterFlat,
- calibrate the light frames using the settings shown above in
Image 2, right side.
ZWO ASI294MC Pro (with temperature control, with "amplifier glow"):Using this CMOS camera with temperature control, I did not notice an improvement of the light frame calibration result by using dark frame optimization. The "amplifier glow" tended not to calibrate out completely with dark frame optimization enabled. So I decided not to use it. The "amplifier glow" as well was the reason for using a MasterFlat-Dark for flat frame calibration.
- Take dark frames, flat-dark frames and flat frames,
- integrate the dark frames to a not pre-calibrated MasterDark,
- integrate the flat-dark frames to a not pre-calibrated MasterFlat-Dark,
- calibrate the flat frames with the MasterFlat-Dark and integrate to a MasterFlat,
- calibrate the light frames using the settings shown above in
Image 2, left side.
6 After Light Frame Calibration6.1 CosmeticCorrection
The calibration will leave some hot pixels in the calibrated light frames. This is normal because hot pixels don't behave ideally. These remaining hot pixels must be corrected with the CosmeticCorrection process directly after completed calibration process.
6.2 Debayer
In case of an OSC camera, the calibrated light frames have to be debayered now, thus yielding RGB images. If the files are written to disk in FITS format, the CFA pattern might have to be specified explicitely (see 1.1.2).
6.3 Registration
The images are then registered by the StarAlignment process.
6.4 Integration
Finally, the registered images are integrated by the ImageIntegration process.
--------------------------------------------------
Note for OSC cameras:
The debayering is an interpolation algorithm which may introduce certain kinds of artifacts that may show up even in the integration result. If you capture a sufficiently large number of light frames and dither enough between light frames, it is worth trying to apply a method called "CFA drizzle". This is the approach recommended by Juan Conejero [10]. It has two important advantages over debayer interpolation: the resolution of the integration result will be higher and interpolation artifacts are avoided.
When using CFA drizzle, the workflow after calibration of the light frames must be modified slightly:
Steps 6.1 Cosmetic Correction and 6.2 Debayer remain the same. In steps 6.3 Registration and 6.4 Integration, the option 'Generate drizzle data' has to be checked. Before Image Integration is applied, the Drizzle files (extension: xdrz) have to be added, additionally to the registrated files. Subsequently an additional step, 6.5 Drizzle Integration, is necessary. The option 'Enable CFA drizzle' has to be checked here.
--------------------------------------------------
Thereby, data reduction has been completed.
==================================================
By now I have searched in the PixInsight forum and found the following interesting threads ([8] and [9]) that I was not aware of previously. So I discovered that my recommendations are anything but new:
The forum user Ignacio used and suggested the approach with dark frame optimization (Case 1, see [8] and [9]), and the forum user astropixel used and suggested the approach without dark frame optimization, referred to as 'bias_in_the_dark (BITD) method' (Case 2, see [8]). This all was written already in 2014!
==================================================
--------------------------------------------------
Sources:
[1] Tutorial by Vicent Peris: "Master Calibration Frames: Acquisition and Processing"
https://www.pixinsight.com/tutorials/master-frames/[2] Juan Conejero: "Dark Frame Optimization Algorithm"
https://pixinsight.com/forum/index.php?topic=8839.0[3] Juan Conejero in: "Warning: No correlation between the master dark and target frames", Replies #2 and #4
https://pixinsight.com/forum/index.php?topic=3286.0[4] "Image Calibration and negative values (KAF 8300)",
https://pixinsight.com/forum/index.php?topic=6822.0[5] Kayron Mercieca, "2. Generating a Master Superbias and a Master Dark" and "4. Calibrating Lights and Correcting Hot and Cold Pixels"
http://www.lightvortexastronomy.com/tutorial-pre-processing-calibrating-and-stacking-images-in-pixinsight.html#Section2[6] David Ault, "PixInsight Manual Image Calibration, Registration and Stacking"
http://trappedphotons.com/blog/?p=693[7] Warren A. Keller, "Inside PixInsight", Springer (2016)
[8] "Preprocessing Canon DSLR frames - a different approach",
https://pixinsight.com/forum/index.php?topic=7006.0[9] "Bias frames - pixel rejection - temperature regulated DSLR",
https://pixinsight.com/forum/index.php?topic=6980.0[10] "Bayer drizzle instead of de-Bayering with OSC",
https://pixinsight.com/forum/index.php?topic=13504.msg81540#msg81540 and
"Additional Interpolation Methods in Debayer Module",
https://pixinsight.com/forum/index.php?topic=13549.msg81718#msg81718--------------------------------------------------
