For beginners: Guide to PI's ImageCalibration

bulrichl

PTeam Member
Nov 2, 2016
819
56
La Palma, Canary Islands
Guide to PixInsight's ImageCalibration (revised on September 02, 2020)

Motivation for writing this text
I observe that many PixInsight newcomers struggle with ImageCalibration and don't get on. Similar questions are put in the forum again and again. Whereas finally the questioner may have gotten the desired solution, unfortunately there is often no feedback given. Then the thread is open-ended and comes to nothing. Such threads are not helpful at all for beginners.

So this guide was written with the goal in mind to provide a general introduction to the usage of PixInsight's ImageCalibration for novices. Please don't answer to this post. When you feel that an important point is missing or there is something wrong or unclear, please send me a private message. If reasonable I will supplement or correct my description.

Please, keep in mind: PI's ImageCalibration process is powerful and flexible, but it does not execute many checks whether your settings are reasonable. Some settings will inevitably yield wrong results. The old wisdom applies here: "garbage in, garbage out". You in person are responsible for the right settings. There is no reason to put a bug report about ImageCalibration when your calibration result looks strange - just check whether the conditions for the acquisition of the calibration frames were appropriate, whether the master calibration files were prepared correctly and take a critical look at your settings for the calibration of the light frames.

The goal of correctly calibrated light frames can be achieved in different ways. That is the reason why you may read different recommendations for the preparation of master calibration files and the calibration procedure. Since there is a wealth of different cameras, some of these recommendations may work well for one configuration and fail for another. It was my goal to describe a procedure that (hopefully) will work generally. Therefore, my recommendations may differ from approaches recommended elsewhere.

In my description I intentionally do not mention Overscan correction because this is a specialty of some dedicated astro cameras. I guess that people who use such cameras know what they are doing.

References and endnotes are specified in square brackets. The references and endnotes are compiled at the bottom of this document.

--------------------------------------------------

1 Setup and Properties of Digital Cameras used for Astrophotography
1.1 General setup of sensors used in digital cameras
Sensors of digital cameras are structured as a 2D array of photosites. The photosites convert incident light to photocurrent which is integrated to electrical charge. After exposure, the generated electrical charge of each photosite is converted to voltage in the readout process. Each voltage is amplified and then converted to a digital number by a A/D converter, generating a 2D array of pixel values.

1.2 CCD and CMOS sensors
Charge-Coupled Device (CCD) sensors and Complementary Metal-Oxide-Semiconductor (CMOS) sensors are commonly utilized for digital cameras. Both kinds of sensors are quantum detectors based on the semiconductor material silicon which is sensitive to light of wavelengths in the range of about 300 to 1000 nm (UV/VIS and NIR). The differences between these technologies are related to the readout process: in a CCD sensor, a vertical and horizontal transport of electrical charges is effected, then the electrical charges are output serially at one location on the sensor. Conversion to voltage, amplification and A/D conversion are performed outside the sensor, in the camera electronics. In a CMOS sensor, electrical charge to voltage conversion is performed on each individual photosite, and amplification and A/D conversion are performed on the sensor as well. These differences result in different properties of the raw data, and this has to be taken into account in image calibration. The bottom line is: the right approach for image calibration also depends critically on the used sensor and digital camera model.

1.3 Monochrome and OSC sensors
It is important to differentiate between monochrome and One Shot Color (OSC) sensors. In OSC sensors, each individual photosite of the sensor is equipped with a color filter. Usually 3 (or sometimes 4) different colors are used for the color filters. Each photosite is exposed only to the light transmitted through its color filter. The different colors are arranged in a periodic pattern on the sensor, called Color Filter Array (CFA) pattern, e.g. a Bayer pattern or Fujifilm's X-Trans pattern. The raw bayered/mosaiced data of an OSC camera are not color images. The color in CFA data has to be reconstructed by an interpolation algorithm called "Debayering" or "Demosaicing". In this process missing color information is computed for each pixel thus generating RGB (color) images. In contrast, the photosites of a monochrome sensor are not equipped with color filters.

1.4 Properties of digital cameras used for astrophotography
In principal both regular digital cameras (e.g. Digital Single-Lens Reflex (DSLR) or Digital Single Lens Mirrorless (DSLM) cameras) and dedicated astro cameras can be used for astrophotography. When choosing a regular digital camera for astrophotography one should take care that the camera can be set to save the data in raw format. However, there are some regular digital cameras that manipulate the data even when the raw format is set (e.g. applying black point correction, applying of spatial filters for hot pixel removal, or the "star eater" issue). Such cameras are ill-suited for astrophotography.

Regular digital cameras normally are OSC cameras. There are dedicated astro cameras in monochrome and in OSC versions.

In order to reduce thermal noise, dedicated astro cameras usually are equipped with a cooling system with temperature control whereas regular digital cameras normally are not cooled.


2 General Settings
2.1 Special recommendation for OSC cameras
When using an OSC camera (be it a regular digital camera or a dedicated astro camera) you generally want to perform the entire calibration process with raw CFA data. Raw CFA data are classified as 'Gray' in the 'Information' tool bar of PixInsight and are displayed as grayscale images. Only after the calibration process (including CosmeticCorrection) is completed, the calibrated light frames have to be debayered (to generate RGB images). Subsequently they are registered and finally integrated.

2.2 File formats
If you use a regular digital camera which is able to save the data in a proprietary raw format (e.g. Canon's CR2 or CR3, Nikon's NEF, Sony's ARW or Fujifilm's RAF format), set your camera to use this raw format.

The acquisition software normally lets you choose whether the data coming from the camera will be saved to disk in the proprietary raw format or in FITS format (or both of them). The FITS format contains some useful metadata that are not stored in the proprietary raw format (e.g. the name and the coordinates of the object, the focus position, etc.). Due to data compression, the proprietary raw format results in somewhat smaller file size. Dependent on the chosen file format, there is a difference in the data: the proprietary raw format contains intensity values in the bit depth of the analog digital converter (ADC), e.g. for a 14-bit ADC in the range of 0 to 2^14 - 1 = 16383. In contrast, the same data in a FITS file are scaled to 16 bit, i.e. they are multiplied by factor 4, the range is 0 to 2^16 - 1 = 65535. This is one important reason that the use of different file formats for one project (light and calibration frames) may lead to severe issues in image calibration. So please follow the advice given in section 2.3.

If you decide to let the data coming from the camera be saved to disk in FITS format, see section 2.2.2.

2.2.1 Proprietary raw format of regular digital cameras
If you decide to let the data coming from the camera be saved to disk in the proprietary raw format, ascertain that the RAW Format Preferences in PixInsight (Format Explorer, double click on 'RAW') is set to 'Pure Raw'. The proprietary raw format contains the CFA mosaic pattern. When opening the file in PixInsight, the raw image decoding software 'LibRaw' that is used by PixInsight's RAW format support module detects the CFA mosaic pattern and makes it available for the ImageCalibration and Debayer processes.

From version 1.5.5 on, the RAW format support module comes with the new options 'Force focal length' and 'Force aperture'. When enabled, either no metadata will be generated for focal length and aperture respectively (when the default value of 0 is left) or the inputted values will be used. This is useful when the frames were captured with a telescope, and the camera of course is not able to detect the correct values of focal length and aperture. In this case, these options should be enabled by the user in order to avoid meaningless metadata (e.g. a default focal length of 50 mm).

2.2.2 FITS format
The camera driver or the acquisition software provides the image data in FITS format. The data are scaled to 16 bit (range 0 to 65535).

The FITS header does not necessarily contain the CFA mosaic pattern for OSC cameras. Some acquisition software writes the non-standard FITS keywords 'BAYERPAT', 'XBAYROFF', 'YBAYROFF' and 'ROWORDER' which are supported by PixInsight as well ('ROWORDER' is new in PixInsight release 1.8.8-6, see the mouse-over text in the FITS Format Preferences). If the FITS keyword 'BAYERPAT' is not written to the FITS header, you have to explicitely specify the correct CFA mosaic pattern (see camera handbook) when executing the ImageCalibration or the Debayer process.

2.3 Camera driver and acquisition software
Some camera manufacturers provide two camera drivers: one 'native' driver and one ASCOM driver. For the acquisition of frames for one project (light and calibration frames) always use the same camera driver, the same acquisition software and the same file format. It is not guaranteed that different drivers, different acquisition software or different file formats will produce compatible results (e.g. regarding scaling of intensity values or width and height of the frames). Such incompatibilities will invariably cause image calibration to either produce wrong results or even fail completely.

If you use an OSC camera of make ZWO, caution is advised when the 'native' camera driver is used: the ZWO SDK enables the user in the acquisition software to control settings that influence the white balance of a displayed color image. This is achieved by two parameters, WB_R and WB_B, data range 1 to 100, the default values are WB_R = 52 and WB_B = 95. The intensities of the red channel will be multiplied by WB_R/50 and the intensities of the blue channel by WB_B/50. Unfortunately the results of this multiplication are also written to disk in the FITS file. So it is important to set the values of both parameters to 50 and subsequently apply 'Save Config'; only in this way, the real raw intensities will be saved to disk in the FITS files, see [1]. Since the data coming from the camera are saved in FITS files which usually have the number format 'unsigned 16-bit integer', otherwise rounding errors and clipping of high values will arise. Such a complication is generally avoided when the ASCOM camera driver is used instead of the native driver.


3 Why do we perform Image Calibration?
3.1 Temporal noise
Definition according to [2], lecture 6:
Temporal noise is the temporal variation in pixel output values under constant illumination due to device noise, supply and substrate noise, and quantization effects.
Temporal noise in the light frames cannot be reduced by image calibration; the image calibration will even introduce a slight amount of additional temporal noise from the master calibration files into the calibrated subframes.

3.2 Fixed pattern nonuniformity (FPN)
Definition according to [2], lecture 7:
Fixed Pattern Noise (FPN), also called nonuniformity, is the spatial variation in pixel output values under uniform illumination due to device and interconnect parameter variations (mismatches) across the sensor.
FPN is generated by imperfections of the sensor: the individual photosites of a sensor do not behave ideally. There are pixel-to-pixel variations in bias offset, dark current and light sensitivity. Some CMOS sensors also show a pronounced artifact called "amplifier glow".

The effect of FPN may be negligible in daylight photography, but it is crucial in low light photography, particularly in astrophotography. Whereas temporal noise can be reduced by capturing and integrating more frames, FPN cannot be reduced in this way. On the contrary, FPN has to be removed as far as possible, or it will become visible once the image is stretched decently. This will emerge even more clearly in deeper exposed images.

Sometimes it is claimed that a correct image calibration (with dark frames) is unnecessary when dithering (= shifting the pointing of the telescope slightly in random directions) between light frames was applied. This statement is wrong. If at all possible, dithering between light frames AND a correct image calibration should be applied.

3.3 Vignetting and shadowing effects
To make things worse, imperfections of the optics affect the light that hits the photosites: vignetting and shadowing effects caused by dust particles in the light path are common to every telescope.

3.4 Motivation for performing a correct image calibration
All above mentioned effects cause reproducible detractions of the raw data which can be removed to a large extent by correctly performed image calibration. Residual FPN plus residual vignetting / shadowing effects will set a limit beyond that an image cannot be stretched further. So the motivation to perform a correct image calibration is: to reduce FPN, vignetting and shadowing effects as far as possible. Since there are additive and multiplicative corrections involved, calibration steps have to be executed in the reverse order of the occurence of the disruptive effects (also see: [3]).


4 Types of Calibration Frames
4.1 Bias frames
Bias frames are captured with the sensor in complete darkness, at the shortest exposure time that the camera can provide which is achieved by setting an exposure time of 0 s. The bias signal contains only the constant bias offset and the fixed pattern generated in the readout process.

Bias frames are needed when dark frame scaling (in PixInsight: dark frame optimization) shall be applied (see section 8.1). Bias frames are also needed for the calibration of shortly exposed flat frames which don't contain a significant amount of dark signal.

Please note: The bias frames of cameras with a Panasonic MN 34230 sensor (e.g. ZWO ASI1600, QHY163 or Atik Horizon) show a varying gradient across the frame and an inconsistent bias level when exposure times < 0.2 s are used [4]. With such a camera, it is not advisable to use bias frames at all. However, it is not valid to generalize this recommendation for all CMOS sensors: other CMOS sensors usually don't show this anomaly of an inconsistent bias level. The only other exception that I am aware of is the Sony IMX294 sensor.

4.2 Dark frames
Dark frames are captured with the sensor in complete darkness, at an exposure time that matches the frames they are intended to calibrate (the target frames). In the special case when dark frame optimization (see section 8.1) shall be applied for the calibration of the target frames, the exposure time of the dark frames shall be greater or equal the exposure time of the target frames. Dark frames contain bias signal plus dark signal. The dark signal consists of the term (dark current * exposure time), the fixed pattern generated thereby and "amplifier glow".

4.3 Flat-darks
Flat-darks are dark frames for the calibration of flat frames. They are captured with the sensor in complete darkness, at the same exposure time as the flat frames.

If the flat frames contain a non-negligible amount of dark signal, flat-darks have to be used instead of bias frames for the calibration of flat frames. This case may apply e.g. when flat frames for narrowband filters are captured, resulting in a long exposure time.

The question whether it is advisable to use bias frames or flat-darks for the calibration of flat frames was discussed in [5]. Jon Rista's contributions to this thread are particularly worth reading. The bottom line is: flat-darks are only needed if there is non-trivial dark signal in the flat frames. In posts #160, #163 and #169 a simple test is described how to verify whether a non-trivial dark signal is contained in the flat frames. The result depends critically on the used sensor and the conditions for flat field acquisition. If the test result is negative, the additional effort for capturing matched flat-darks would be waste, and the flat frames should be calibrated with a MasterBias instead. Sole exception: cameras with a Panasonic MN 34230 sensor (bias level instability, see 4.1).

4.4 Flat frames
Flat frames are captured through the telescope or lens, and it is essential that the field is as uniformly illuminated as possible. Flat frames contain bias signal, the information about vignetting / shadowing effects and the pixel-to-pixel variation of light sensitivity. In certain cases, there may be also a non-negligible amount of dark signal, see 4.3. Flat frames have to be calibrated before they are integrated to the MasterFlat which then is applied to the dark-calibrated light frames during image calibration.

Flat frames are needed for the correction of vignetting, of shadowing effects caused by dust particles and of the fixed pattern that is generated by different light sensitivity of individual photosites of a sensor. This step of the calibration process is called "flat field correction".


5 Conditions for the Acquisition of Calibration Frames
5.1 Temperature
For cameras without cooling system: try to take the dark frames at the same ambient temperature as the light frames. For cameras with cooling system: use the same set value for all frames.

5.2 Camera settings
For the acquisition of light frames and ALL sorts of calibration frames, the same ISO (in case of a regular digital camera) or gain and offset (in case of a dedicated astro camera) must be used.

In the special case when dark frame optimization shall be applied (see section 8.1), the exposure time of the dark frames shall be greater or equal the exposure time of the light frames. When dark frame optimization is not to be applied, dark frames have to be captured at the same exposure time as the light frames. The exposure time of flat-darks should match the exposure time of the corresponding flat frames.

5.3 Dark frames, bias frames and flat-darks: avoiding of light leaks
Light leaks can lead to complete failure of image calibration. So any light leak has to be carefully avoided. PixInsight's processes ImageStatistics and HistogramTransformation are well-suited for spot-checking some frames. In order to detect a light leak in your equipment, it is advisable to take a few dark frames with constant exposure time in a bright place with changing external illumination. Compare the statistics and histograms of these dark frames: differences point to the existence of a light leak which has to be localized and remedied. The Blink process is particularly helpful to analyze the statistics of a whole series of calibration frames. Load the series into Blink, click on the bar graph icon ('Series analysis report'), check the option 'Write text file', select the output folder and confirm with 'OK'. The statistics of the whole series is then saved to the text file "Statistics.txt" for further inspection.

Probable candidates for light leaks are all mechanically moving parts in the light path: focuser, camera rotator, filter slider or filter wheel. Lens caps made of plastic are not necessarily nontransparent for IR light. If your lens cap is not made of metal, wrap additional aluminum foil around it and secure it with a rubber band.

It goes without saying that dark frames, bias frames and flat-darks that shall be used for the generation of master calibration files always shall be captured in a dark place.

5.4 Flat frames: unchanged light path for the acquisition
For flat frame acquisition it is all-important to have an unaltered light path, i.e. the same flattener or reducer, the same camera orientation (rotating angle) and focus position as when taking the light frames. If you use a monochrome camera, separate flat frames have to be captured for each filter. Best is not to change anything and take the flat frames directly before or after the light frames. With refractors it is usually possible to use one MasterFlat for some longer time.

5.5 Flat frames: determining the appropriate exposure time
Flat frames have to be captured through the telescope or lens applying an uniformly illuminated field as possible. Too short exposure time for flat frames can cause nonuniform illumination, e.g. when the light source flickers (depending on the power source, the illuminant and the use of a dimmer) or when a mechanical shutter of a dedicated astro camera is used. So check with your equipment whether there is a lower limit for flat frame exposure time in terms of uniform illumination.

Illumination level and exposure time for flat frames have to be controlled in order that the peak in the histogram is in the region of linear response of the sensor. Normally this is the case at about half of the saturation intensity. At first determine the maximum intensity value (in ADU) in the histogram of an overexposed frame (= saturation intensity). Divide this value by factor 2 and take the result as the approximate target value for the mean of a flat frame.

For flat frames of OSC cameras it is important, 1. that no values are clipped in the upper range (this is to be avoided by all means!) and 2. that in the lower range the weakest color channel has sufficient signal. The best way to judge flat frames is to inspect some of them by taking a look at the statistics and the histogram.


6 Generation of Master Calibration Files
It is advisable to prepare the master calibration files and check them (by inspecting them visually and taking a look at image statistics and histograms) before the light frame calibration is executed. In order to minimize additional noise that is introduced by image calibration, the number of calibration frames that is used to prepare a master calibration file matters. The noise reduction ratio of the additionally added noise is proportional to the square root of the number of the calibration frames.

6.1 MasterBias (MB), MasterDark (MD) and MasterFlatDark (MFD)
MB, MD and MFD shall be prepared according to [6]: simple integration of the dark frames, bias frames and flat-darks with the following parameters:

Section 'Image Integration': Combination 'Average', Normalization 'No normalization', Weights 'Don't care (all weights = 1)'
Section 'Pixel Rejection (1)': Rejection algorithm 'Winsorized Sigma Clipping', Normalization 'No normalization'
With the settings in section 'Pixel Rejection (2)', a fine tuning is achievable in order that only unwanted signal (e.g. caused by "cosmic ray artifacts") is rejected.

When the process is completed, the integration result, the MB, MD and MFD respectively, must be saved to disk.

6.2 MasterFlat (MF)
The procedure of preparing the MF involves two steps:
- calibration of the individual flat frames and
- integration of the calibrated flat frames to the MF.
If you use a monochrome camera, separate MFs have to be generated for each filter.

6.2.1 Calibration of the flat frames
The optimal approach for the calibration of flat frames depends on the used sensor and digital camera model.

For CCDs, the procedure that was recommended initially in Vicent's tutorial [6] may work well: calibration of the flat frames with the regular MD and MB using dark frame optimization. This means that the following parameters shall be set:

Enable section 'Master Bias', select the MB, disable option 'Calibrate'.
Enable section 'Master Dark', select the MD, enable both options 'Calibrate' and 'Optimize'.
Disable section 'Master Flat'.

However, for CMOS sensors that exhibit "amplifier glow", the usage of the regular MD and MB and application of dark frame optimization usually will not produce a correct calibration result. Well-known examples of sensors that exhibit strong "amplifier glow" are: Sony IMX183, IMX294 and Panasonic MN 34230. Some other CMOS sensors are affected similarly.

Therefore flat frames of CMOS sensors should be calibrated using either only a MB or only a MFD, not applying dark frame optimization:

Enable section 'Master Bias', select the MB (or the MFD), disable option 'Calibrate'.
Disable section 'Master Dark'.
Disable section 'Master Flat'.

6.2.2 Integration of the calibrated flat frames
Subsequently, the calibrated flat frames are integrated to the MF with the following parameters (see [6]):

Section 'Image Integration': Combination 'Average', Normalization 'Multiplicative', Weights 'Don't care (all weights = 1)'
Section 'Pixel Rejection (1)': Normalization 'Equalize fluxes'
Which rejection algorithm in section 'Pixel Rejection (1)' is appropriate depends on the number of used frames, see the PixInsight Reference Documentation "ImageIntegration" [7] for recommendations. With the settings in section 'Pixel Rejection (2)', a fine tuning is achievable in order that only outliers are rejected.

When the process is completed, the integration result, the MF, must be saved to disk.


7 Potential Pitfalls in Calibration
7.1 Correct subtraction of the bias level
In [8], post #5, Juan Conejero states:

Most dark frame optimization problems (such as 'no correlation') happen because the bias level is not correctly subtracted from one or more calibration frames; usually from the master dark frame.
I agree, but would not confine this statement to the usage of dark frame optimization - it holds generally. In the same post, Juan specifies the three possibilities of subtracting bias correctly:

(1) You have simply integrated the individual dark frames to generate the master dark frame. In this case the master dark frame does have a bias pedestal. This is the most usual procedure, and also the one shown in Vicent's tutorial.

(2) After (1), you have calibrated the master dark frame with ImageCalibration to subtract the master bias frame. In this case the master dark frame does not have a bias pedestal.

(3) You have calibrated the individual dark frames by subtracting the master bias frame from each of them, before integrating them to generate the master dark frame. In this case the master dark frame does not have a bias pedestal, as in (2). This procedure is atypical.

Possibility (3), which Juan denotes as 'atypical', is the cumbersome version of (2): being mathematically equivalent to (2), it will produce identical results. There is just some (unnecessary) arithmethic involved.

With both (2) and (3), the subtraction of the MB is executed in a preliminary step. I will denote this procedures as 'Pre-calibration' of the MD. These approaches suffer from a serious drawback with modern cameras using CMOS sensors: in the calibrated MD or the calibrated dark frame, severe clipping (truncating of negative values) usually occurs. The cause of this issue is explained in the following sections.

7.2 "Dark Current Suppression" and its impact for calibration
Modern cameras using CMOS sensors have a dark current suppression mechanism (in the hardware) that subtracts the dark offset. Thus dark frames and bias frames have similar average intensities, almost independent of the exposure time of the dark frames! Subtraction of the MB from the MD (or from a dark frame) therefore results in negative values for a high fraction of the pixels. If this subtraction is carried out in a preliminary step, all resulting negative values will be truncated (clipped), and these data are lost. I experienced this situation with a calibrated MD of my Canon EOS 600D (= Rebel T3i), of my ZWO ASI294MC Pro and of my ASI071MC Pro. So I conclude that this is general behavior of cameras equipped with a CMOS sensor. Image 1 shows a screen dump of the histograms (here: of a Canon EOS 600D = Rebel T3i): on the left side the not pre-calibrated MD, on the right side the pre-calibrated MD in which about half of the peak is truncated (set to zero). If such a pre-calibrated MD is used for light frame calibration, only about half of the pixels in the light frame are calibrated correctly. For the rest of the pixels (in this example: for the other half), the correction for the fixed pattern generated by the term (dark current * exposure time) is not achieved. The consequence of using such a clipped, pre-calibrated MD is higher residual fixed pattern in the calibrated lights.

7.3 Truncation of negative values during a calibration process
Why are negative values simply truncated in the ImageCalibration process?
In [9], post #3, Juan Conejero explained in detail the necessity of truncation of negative values in the calibration process. In short: for the calibration process a coherent data set must be used. However, the truncation of negative values is NOT applied to intermediate results of a calibration step, but rather
Truncation to the [0,1] range is carried out as the very last step of the calibration task for each frame, i.e. after overscan, bias, dark, flat and pedestal correction.
In [9], post #3, Juan Conejero suggested the addition of a pedestal in order to avoid the issue of data loss by truncation of negative values during calibration. However, in most cases this is unnecessary. As shown in section 8.1, there is usually no need to use pedestals when you
1) don't pre-calibrate neither the MD nor dark frames and
2) calibrate the MD only during light frame calibration.

7.4 Is Pre-calibration of the MD reasonable?
Calibrating either the integrated dark frames or the individual dark frames may cause severe clipping in the pre-calibrated MD. In contrast, using PI's option of calibrating the MD only during light frame calibration, negative values in the intermediate result (the calibrated MD) will not be clipped.


The bottom line from sections 7.1 to 7.4 therefore is:
NEVER pre-calibrate your dark frames, neither the individual dark frames nor the integrated dark frames. Use instead PixInsight's option of calibrating the MD during calibration of the light frames.

--------------------------------------------------
There are tutorials that suggest to pre-calibrate the individual dark frames or the integrated dark frames AND don't even give a reference to truncation of negative values or how to avoid clipping in the pre-calibrated MD (namely by the application of a pedestal). Even the otherwise excellent tutorial [10] does. This is bad advice. For modern cameras using CMOS sensors which apply dark current suppression, such a calibration approach will result in a lower SNR of the calibrated light frames. Fortunately better instructions can be found, e.g. [11] and [12].
--------------------------------------------------


8 Light Frame Calibration with PixInsight's ImageCalibration
Select the ImageCalibration process and load the light frames by 'Add Files'.

8.1 Dark frame optimization
Once you decided to not pre-calibrate the MD, things become quite simple. In fact, for light frame calibration there are only two settings left, that will lead to a correct subtraction of the bias level. These are the two cases which have to be differentiated:

Case 1: Calibration WITH dark frame optimization
Dark frame scaling means that one MD with long exposure time is used for the calibration of light frames which were captured at less or equal exposure time. Assuming that a camera with temperature control is used at constant temperature, the dark current is constant, and the dark signal (= MD - MB) increases linearly with time. In this case it is possible to scale the dark signal according to the ratio: exposure time(LightFrame) / exposure time(MD). Note, however, that in dark frame optimization, PixInsight uses neither temperature nor exposure time to evaluate k0. PixInsight optimizes k0 for lowest noise in a certain area of the resulting calibrated light frame, see [13]. The general form of the calculation applied in the calibration of light frames is represented in equation {1}:

Code:
Cal1 = ((LightFrame - MB) - k0 * (MD - MB)) / MF * s0   (WITH dark frame optimization) {1}
where k0: dark scaling factor,
      s0: master flat scaling factor (= Mean of MF)
(By application of the master flat scaling factor s0, the MF is normalized.)

Conclusion: The only problematic term (concerning negative values) in equation {1} is: (MD - MB). When the calibration of the MD is executed only during light frame calibration, the term (MD - MB) is an intermediate result, accordingly negative values will not be truncated. The calibrated MD will not be saved to disk.

For the calibration of the light frames with dark frame optimization, use the settings in Image 2, right side:
Enable section 'Master Bias', select the MB, disable option 'Calibrate'.
Enable section 'Master Dark', select the MD, enable both options 'Calibrate' and 'Optimize'.
Enable section 'Master Flat', select the MF, disable option 'Calibrate'.


Case 2: Calibration WITHOUT dark frame optimization
Without dark frame optimization, k0 equals 1.0, therefore MB is canceled out from equation {1}. Thus equation {1} is simplified to equation {2}:

Code:
Cal2 = (LightFrame - MD) / MF * s0                      (NO dark frame optimization) {2}
Conclusion: The term that could be problematic concerning negative values, (MD - MB), does not appear in equation {2}. For a light frame calibration without dark frame optimization a MB is not needed at all.

For the calibration of light frames use the settings in Image 2, left side:
Disable section 'Master Bias'.
Enable section 'Master Dark', select the MD, disable both options 'Calibrate' and 'Optimize'.
Enable section 'Master Flat', select the MF, disable option 'Calibrate'.

Whether it is favorable to use dark frame optimization or not depends essentially on the camera model. Using dark frame optimization for a camera without temperature control may greatly improve the calibration result. This is because temperature deviations between dark and light frame acquisition are unavoidable in this case. For a camera with temperature control the benefit of dark frame optimization will be much lower, possibly even not detectable.

However, if the sensor shows "amplifier glow", this might not be calibrated out completely with dark frame optimization enabled. In cases of a not temperature controlled camera with "amplifier glow" it's worthwhile to accurately compare the results of both settings (compare the final integration results after image calibration, if applicable debayering, and registration of the light frames).

8.2 ImageCalibration's output to Process Console
After having adjusted the necessary settings for light frame calibration, the process can be executed. During or after the execution, observe the output to the process console. The output may look like the following example (extract):

--------------------------------------------------
Applying bias correction: master dark frame

Dark frame optimization thresholds:
Td0 = 0.00179381 (733282 px = 4.068%)
Computing master flat scaling factors ...
s0 = 0.093233 (1) <==
...
Writing output file: F:/Astro/171217/Calibrated/Light_360SecISO800_092718_c.xisf
Dark scaling factors:
k0 = 0.985 (2) <==
Gaussian noise estimates:
s0 = 1.687e-03, n0 = 0.911 (MRS)
...

--------------------------------------------------

(1):
The master flat scaling factor s0 is the mean of the MF.

(2) (Only when dark frame optimization is enabled):
k0 is the dark scaling factor, optimized for lowest noise in the resulting calibrated light frame.

8.3 Checking of the calibration result: Output pedestal
After following the above suggestions, you should spot-check some of the resulting calibrated light frames
- with HistogramTransformation:
check the histogram (adjust horizontal zoom that the histogram region around intensity 0 can be inspected carefully, this might be the case at a horizontal zoom of about 50 - 100) and
- with ImageStatistics (option 'Unclipped' disabled):
check count (%), it should be very near 100 %. The fraction of clipped pixels equals (100 % - count (%)).

Following the calibration approach recommended in this guide, the calibrated light frames normally should not be clipped. If you nevertheless detect clipping, the calibration result is not correct, so please check the following conditions carefully: acquisition of calibration frames, preparation of the master calibration files and the settings for the calibration of the light frames. However, it is possible as well that clipping is just caused by a too low signal in the light frames (e.g. due to very short exposure time or the usage of narrowband filters and short exposure time). If you cannot find another cause for clipping, it will be necessary to repeat the calibration of the light frames applying an output pedestal. If needed, this option has to be set in the 'Output Files' section of the ImageCalibration process. Usually a value in the range of 100 - 200 DN is sufficient. It doesn't hurt if the value is a little bit larger than necessary for avoiding clipping though. Again the new calibration results should be checked as described above in order to make sure that the value of the output pedestal was sufficient.

8.4 Checking of the calibration result: Result of the flat field correction
For a successfully achieved flat field correction, both a correct calibration of the flat frames and of the light frames is essential. Otherwise under- or overcorrection of vignetting and shadowing effects caused by dust particles will be the consequence.

Examples:
If the calibration of the flat frames is omitted, the result of the flat field correction will be undercorrected (= residual vignetting in the calibrated light frames).
If both the MasterBias and a not precalibrated MasterDark are subtracted from the light frames, the result of the flat field correction will be undercorrected.
If the dark-calibration of the light frames is omitted, the result of the flat field correction will be overcorrected (= "negative vignetting" in the calibrated light frames).

Calibrated light frames of an OSC camera must be debayered before they can be judged. A ScreenTransferFunction (STF) Auto stretch should be applied with option 'Link RGB channels' disabled.

For an OSC camera it is normal that the flat field correction will give rise to a color shift if one flat field scaling factor is calculated for the MasterFlat in the calibration of the light frames. The reason for this is that the color channels in the flat frames usually are exposed differently, but the master flat scaling factor is computed averaging all channels. Therefore the weakest channel(s) in the flat frames will be increased in the calibrated light frames, and the strongest color channel(s) will be attenuated. This color shift is meaningless in terms of signal-to-noise ratio. The correct color balance will be adjusted later in the workflow, e.g. with the PhotometricColorCorrection (PCC) process. From PixInsight release 1.8.8-6 on, the ImageCalibration process comes with the new option 'Separate CFA flat scaling factors' in the 'MasterFlat' section. If this option is enabled and the CFA mosaic pattern is either successfully detected or set explicitely by the user, 3 CFA scaling factors for the MasterFlat are computed. In this way the color shift that occurred with the flat field correction in previous versions is avoided.

8.5 Approved settings for light frame calibration (examples from personal experience)
Canon EOS 600D = Rebel T3i (no cooling, no "amplifier glow"):
Using this DSLR, the result of the light frame calibration was greatly improved by using dark frame optimization. The MB was used for flat frame calibration.

- Take dark frames, bias frames and flat frames,
- integrate the dark frames to a not pre-calibrated MD,
- integrate the bias frames to a MB,
- calibrate the flat frames with the MB and integrate to a MF,
- calibrate the light frames using the settings shown above in Image 2, right side.


ZWO ASI294MC Pro (cooling with temperature control, strong "amplifier glow"):
Using this CMOS camera, I did not notice an improvement of the light frame calibration result by using dark frame optimization. The "amplifier glow" tended not to calibrate out completely with dark frame optimization enabled. So I decided not to use it. A MFD was used for flat frame calibration.

- Take dark frames, flat-darks and flat frames,
- integrate the dark frames to a not pre-calibrated MD,
- integrate the flat-darks to a not pre-calibrated MFD,
- calibrate the flat frames with the MFD and integrate the calibrated flat frames to a MF,
- calibrate the light frames using the settings shown above in Image 2, left side.


9 After Light Frame Calibration
9.1 CosmeticCorrection
The calibration usually will leave some hot and warm pixels in the calibrated light frames. This is normal because hot pixels don't behave ideally. These remaining hot pixels should be corrected with the CosmeticCorrection process directly after the ImageCalibration process. For OSC cameras, the option 'CFA' has to be enabled in CosmeticCorrection.

9.2 Debayer
In case of an OSC camera, the calibrated light frames have to be debayered now, thus generating RGB images. If the files from the camera are written to disk in FITS format, the CFA mosaic pattern might have to be specified explicitely (see section 2.2.2).

9.3 Checking the quality of the frames
At this stage the frames should be checked visually with the Blink process in order to reject obviously bad frames (due to e.g. clouds, guiding error, etc.). Subsequently for the remaining frames the SubframeSelector process should be applied. The reference image for the StarAlignment process should be chosen (looking out for low FWHM and low eccentricity) and inspected visually as well.

9.4 Registration
The approved images are then registered by the StarAlignment process against the reference image.

9.5 Integration
Finally, the registered images are integrated by the ImageIntegration process using the following parameters:

Section 'Image Integration': Combination 'Average', Normalization 'Additive with scaling', Weights 'Noise evaluation'. If an output pedestal was used in the calibration of the light frames, the option 'Subtract pedestals' has to be disabled. Otherwise the very first step in the ImageIntegration process is subtraction of the pedestal, and the integration result will be the very same as if no output pedestal was applied at all!
Section 'Pixel Rejection (1)': Normalization 'Scale + zero offset'
Which rejection algorithm in section 'Pixel Rejection (1)' is appropriate depends on the number of used frames, see the PixInsight Reference Documentation "ImageIntegration" [7] for recommendations. With the settings in section 'Pixel Rejection (2)', a fine tuning is achievable in order that only unwanted signal is rejected (e.g. caused by cosmic ray artifacts, aeroplanes or satellites). The judgement whether the chosen parameters are appropriate or whether they should be tweaked further is greatly facilitated by comparison of the integration result with the rejection map "rejection_high", see [7], chapter 2.8 "Rejection Maps".

When the process is completed, the integration result must be saved to disk.

--------------------------------------------------
Note for OSC cameras:

The debayering is an interpolation algorithm which introduces certain kinds of artifacts that may show up even in the integration result. If you capture a sufficiently large number of light frames and dither sufficiently between light frames, it is worth trying to apply a method called "CFA drizzle". This is the approach recommended by Juan Conejero for OSC cameras [14]. It has two important advantages over debayer interpolation: interpolation artifacts are avoided and the resolution of the integration result will be higher.

When using CFA drizzle, the workflow after calibration of the light frames must be modified slightly:
Steps 9.1 Cosmetic Correction and 9.2 Debayer remain the same. In steps 9.4 Registration and 9.5 Integration, the option 'Generate drizzle data' has to be checked. Before ImageIntegration is executed, the Drizzle files (extension: xdrz) have to be added, additionally to the registered files. Subsequently an additional step, 9.6 Drizzle Integration, is necessary. The option 'Enable CFA drizzle' has to be checked for OSC cameras. When the process is completed, the drizzle integration result must be saved to disk.


Thereby, data reduction has been completed.


10 WeightedBatchPreprocessing (WBPP) Script
In the above description, the whole preprocessing was achieved by manually executing PixInsight's processes, step by step. For the beginner, this approach is recommended in order to learn what happens in preprocessing and to become acquainted with the peculiarities of his camera. It also might be advisable to use the step-by-step approach when issues related to image calibration appear. However, PixInsight also offers the possibility of executing the main part of the preprocessing automatically: by using the WeightedBatchPreprocessing (WBPP) script, see [15]. It is convenient to use the WBPP script for the preparation of the master calibration files, for the calibration of the light frames, for the application of cosmetic correction, (if applicable) for debayering and for registering. However, the parameters for image integration are best set manually, so better use the ImageIntegration process for this operation (see [15], post #2).

For a description of new features of the WBPP script see [16], and for a detailed reference documentation see [17].

Adam Block has created a series of free videos that explain the function and usage of the WBPP script extensively, see [18].

==================================================

By now I have searched in the PixInsight forum and found the following interesting threads ([19] and [20]) that I was not aware of before I posted the first version of this guide in Decmber 2017. So I discovered that my recommendations are anything but new:

The forum user Ignacio used and suggested the approach with dark frame optimization (Case 1, see [19] and [20]), and the forum user astropixel used and suggested the approach without dark frame optimization, referred to as 'bias in the dark' (BITD) method (Case 2, see [19]). This all was written already in 2014!

==================================================


--------------------------------------------------

References and endnotes:

[1] "ZWO ASI294MC Pro - Bad calibration frames", https://stargazerslounge.com/topic/347561-zwo-asi294mc-pro-bad-calibration-frames/ ,
"Strange bias frames with ASI294MC Pro (under Astroberry)", https://indilib.org/forum/ccds-dslr...with-asi294mc-pro-under-astroberry.html#51097 and
"ASI533MC Pro weird color balance", https://www.cloudynights.com/topic/708880-asi533mc-pro-weird-color-balance/

[2] Abbas El Gamal, "EE392B: Introduction to Image Sensors and Digital Cameras", http://isl.stanford.edu/~abbas/aeglect392b.php

[3] Bart Dierickx, Caeleste, "Imperfections of high-performance image sensors", chapter "Calibration" (pp. 38 - 50), https://www.researchgate.net/publication/273538329_Imperfections_of_high-performance_image_sensors

[4] Jon Rista in "QHY183M & Flats = Frustration", posts #57 and #63, https://www.cloudynights.com/topic/676574-qhy183m-flats-frustration/page-3

[5] Jon Rista in "Should we just stop talking about bias frames?", posts #54, #80, #160, #163 and #169, https://www.cloudynights.com/topic/709822-should-we-just-stop-talking-about-bias-frames/

[6] Vicent Peris: "Master Calibration Frames: Acquisition and Processing", https://www.pixinsight.com/tutorials/master-frames/

[7] PixInsight Reference Documentations are available for many but not all of the processes. They can be displayed either by clicking on the Browse Documentation icon (the sheet of paper icon at the bottom right of the process window) or by calling the Process Explorer and choosing the appropriate process from the select list.

In connection with Preprocessing, the following Reference Documentations exist, and should be consulted in case of ambiguity:
ImageIntegration, ImageCalibration, Debayer, SubframeSelector, StarAlignment, ScreenTransferFunction, HistogramTransformation.
There are processes for which a Reference Documentation does not yet exist. In this case please study the mouse-over texts that are available for most of the parameters in all processes.

[8] Juan Conejero in: "Warning: No correlation between the master dark and target frames", posts #3 and #5, https://pixinsight.com/forum/index....tween-the-master-dark-and-target-frames.3169/

[9] "Image Calibration and negative values (KAF 8300)", https://pixinsight.com/forum/index.php?threads/imagecalibration-and-negative-values-kaf-8300.6593/

[10] Kayron Mercieca, "2. Generating a Master Superbias and a Master Dark" and "4. Calibrating Lights and Correcting Hot and Cold Pixels", http://www.lightvortexastronomy.com...d-stacking-images-in-pixinsight.html#Section2

[11] David Ault, "PixInsight Manual Image Calibration, Registration and Stacking", http://trappedphotons.com/blog/?p=693

[12] Warren A. Keller, "Inside PixInsight", 2nd edition, Springer (2018)

[13] Juan Conejero: "Dark Frame Optimization Algorithm", https://pixinsight.com/forum/index.php?threads/dark-frame-optimization-algorithm.8529/

[14] "Bayer drizzle instead of de-Bayering with OSC", https://pixinsight.com/forum/index.php?threads/bayer-drizzle-instead-of-de-bayering-with-osc.12996/ and
"Additional Interpolation Methods in Debayer Module", https://pixinsight.com/forum/index....n-methods-in-debayer-module.13040/#post-80816

[15] "Weighted Batch Preprocessing update - v1.4.0", https://pixinsight.com/forum/index.php?threads/weighted-batch-preprocessing-update-v1-4-0.13677/

[16] Tommaso Rubechi, "The new Weighted BatchPreprocessing v1.4.0", https://www.tommasorubechi.it/2019/11/15/the-new-weighted-batchpreprocessing/

[17] Roberto Sartori, "Weighted Batch Preprocessing v1.4.0 - an improved version of PixInsight Batch Preprocessing", https://www.tommasorubechi.it/wp-content/uploads/2019/11/WBPP-Guide_v1.4.0.pdf

[18] "WBPP tutorial series", https://pixinsight.com/forum/index.php?threads/wbpp-tutorial-series.14632/

[19] "Preprocessing Canon DSLR frames - a different approach", https://pixinsight.com/forum/index....-canon-dslr-frames-a-different-approach.6773/

[20] "Bias frames - pixel rejection - temperature regulated DSLR", https://pixinsight.com/forum/index....el-rejection-temperature-regulated-dslr.6747/

--------------------------------------------------
 

Attachments

Last edited:

russp

Member
Mar 13, 2014
15
0
Great guide - very useful, can I ask what debayer pattern you're using for your ASI294 ? I've been using RGGB but I'm not entirely sure it's correct as it's giving me really noisy blotchy backgrounds even after DBE.
 

lage

Active member
Mar 24, 2017
25
0
Very informative and helpful, thanks.
I have one question though, that I haven't found the answer to;
I only use DSLR's for my astrophotography. Shooting lights and darks under the same circumstances is understandable, but how about flats and biases? Can you have one set of these for all your lights and darks, or do you need different sets for different ambient temperatures, ISO settings etc?

Best regards,
Lage
 

sn2006gy

Member
Jun 10, 2020
22
2
I have one question though, that I haven't found the answer to;
I only use DSLR's for my astrophotography. Shooting lights and darks under the same circumstances is understandable, but how about flats and biases? Can you have one set of these for all your lights and darks, or do you need different sets for different ambient temperatures, ISO settings etc?
Many people create a master bias and use that (or a super bias as it's called) and it can be done at any time and any temperature. Typically, I do flats using nina flat wizard at the end of observing (using sky flats during morning hours). Nina will calibrate the flats to optimum exposure. Goes so quick I haven't tried to do master flats and save them. Plus, I move my telescope around a lot so dust won't necessarily always stay in same place long enough for masterflats sets to remove motes reliably.
 

sn2006gy

Member
Jun 10, 2020
22
2
What are people doing with their ASI533? There is no real ampglow to speak of. I've seen a lot of discussions of people not using darks and some even dropping bias if they have good dark flats, flats and lights. I'd love a similar capable monochrome to the 533... would simplify processing.
 

umasscrew39

Well-known member
May 4, 2019
72
22
For the ASI533MC (and its big brother, ASI2600MC), I've have had great success with darks, dark flats, flats but no bias. I now use WBPP with minimal PP. I started out just testing it with no flats and darks and the images were truly amazing, due to low noise and no amp glow, especially considering nothing was done with the lights. Of course, if your optics are dirty, the image will look nasty. So, not a good practice.
 
  • Like
Reactions: JPetruzzi

JPetruzzi

Active member
Jun 18, 2020
34
9
I did some tests w/my 533, tried it with just darks, no dark optimization, no bias, then calibrated the same frames in the standard PI workflow, i.e. bias, and calibrated and optimized darks. Resulting frames were nearly indistinguishable. Pretty sure the sensor is so good it won't matter what route you take. :)
 

umasscrew39

Well-known member
May 4, 2019
72
22
Let's just say, compared to when I used SX CCD cameras, which are very good, the 533 and 2600 CMOS are very forgiving cameras. They make the hobby much more enjoyable, in my opinion. Especially so, when you do not (like me) have a lot of clear nights to experiment and keep tweaking things.
 
  • Like
Reactions: JPetruzzi

JPetruzzi

Active member
Jun 18, 2020
34
9
Let's just say, compared to when I used SX CCD cameras, which are very good, the 533 and 2600 CMOS are very forgiving cameras. They make the hobby much more enjoyable, in my opinion. Especially so, when you do not (like me) have a lot of clear nights to experiment and keep tweaking things.
Glad to hear it--my ASI533MC-Pro is my first cooled camera, looks like I started with a good one. :)
 
  • Like
Reactions: umasscrew39

sn2006gy

Member
Jun 10, 2020
22
2
Been happily just processing with Flats, Flat darks and Lights using WBPP. Super simple. Love this camera!