Author Topic: DarkBiasNoiseEstimator and FlatSNREstimator scripts  (Read 7345 times)

Offline mschuster

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1087
DarkBiasNoiseEstimator and FlatSNREstimator scripts
« on: 2014 December 05 12:23:22 »
Here is an easy to use script that estimates the standard deviation of temporal noise (e.g. read noise and dark noise) in a dark or bias subframe or integration. I have found it useful for dark and bias quality checks and also for sensor characterization.

Mike

https://dl.dropboxusercontent.com/u/109232477/PixInsight/Scriptbox/DarkBiasNoiseEstimator.0.11.zip

A dark or bias frame can be viewed as having three noise components: temporal noise (e.g. read noise and dark noise), spatial noise (e.g. offset and dark current nonuniformity), and spurious signal noise (e.g. cosmic ray hits). A good dark or bias master integration should bury temporal noise and reject spurious signal noise, and leave spatial noise intact. The spatial noise is what gets frame subtracted in the calibration of light frames. So, spatial noise is the "signal" that a good dark or bias master should contain, with minimal "noise" in the form of temporal noise and spurious signal noise. So one quality metric of dark or bias frame is how much temporal noise and spurious signal noise it contains, the less of both the better.

If you look at a single frame, spurious signal noise is often easy to see, but it is hard to distinguish temporal noise from spatial noise. They are easy to confuse, and even noise measurement tools can confuse them when analyzing a single frame. The script DarkBiasNoiseEstimator solves this problem by first subtracting two dark or bias frames to eliminate spatial noise, and then measuring the temporal noise in the difference using a robust technique that rejects spurious signal noise. The script normalizes its temporal noise measurement to account for the fact that there are two doses of independent temporal noise in the frame difference. The script provides the resulting standard deviation of temporal noise estimate. The script does not attempt to measure spurious signal noise.

So the script needs two frames to make a measurement. The script assumes that the spatial noise in both frames is equal, otherwise their difference won't properly eliminate spatial noise. This means that both frames should be captured within a reasonable time period, i.e. preferably hours, days, or maybe weeks, and not months or years apart. Both temporal noise and spatial noise on my sensor drifts measurably in a month or so. The script also assumes that for subframes both were exposed similarly, and for integrations that both are similar combinations of the same number of similarly exposed subframes. This assumption is required for normalization purposes.

The examples below show results for frames from my mono sensor operating at -20°C. The script also works on raw Bayer CFA frames and for channels extracted from de-Bayered CFA frames. The frames must be single channel and equal size.

Here are the results for flat_dark subframes, each binned 2x2 and exposed 24 seconds.



Since the flat_dark exposure is relatively short, and dark current is relatively low in this sensor, this result is a good estimate of sensor read noise (i.e. there is little dark noise in the subframes). Using the sensor gain of 1.059 e-/DN (provided by my vendor), an estimate of my sensor's binned 2x2 read noise is 12.65 DN RMS * 1.059 e-/DN = 13.4 e- RMS.

Here is the result for flat_dark integrations, each an average combination of 64 flat_dark subframes, each again binned 2x2 and exposed 24 seconds.



Temporal noise in the integration decreased by the factor 12.65 / 1.59  = 7.96, which is nearly equal to the expected square root of 64, the number of combined subframes. This means that the subframe integration is working about as well as is possible.

There is the result for darks subframes exposed 40 minutes (i.e. 2400 seconds). Additional dark noise results in a higher temporal noise estimate than that of the flat_dark subframes.



Given the results for the 24 second flat_dark and 2400 second dark, it is easy to estimate sensor dark current by exploiting the independence of the noise sources, photon statistics, the known time interval and sensor gain.

(15.18^2 - 12.65^2) / (2400 - 24) * 1.059 = 0.031 e-/second.
« Last Edit: 2014 December 19 17:09:40 by mschuster »

Offline mschuster

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1087
Re: DarkBiasNoiseEstimator and FlatSNREstimator scripts
« Reply #1 on: 2014 December 05 19:20:37 »
Here is a script that estimates the signal to temporal noise ratio and gain of flat subframes and integrations. The script works similarly to DarkBiasNoiseEstimator and provides a tool for flat quality evaluation and sensor characterization.

Mike

https://dl.dropboxusercontent.com/u/109232477/PixInsight/Scriptbox/FlatSNREstimator.0.15.zip

The three noise components for flats are: temporal noise (e.g. read noise, dark noise, and photon noise of the flat exposure), spatial noise (e.g. offset, dark current, QE, and flat exposure nonuniformity), and spurious signal noise (eg. cosmic ray hits and stars in sky flats). The script provides the resulting standard deviation of temporal noise estimate with spatial noise and spurious signal noise discounted, the signal level of the flat exposure, and derivative information. In addition to the flats, the script requires as input a dark or bias subframe or integration if the flats are uncalibrated. The script works on mono frames, raw Bayer CFA frames, and for channels extracted from de-Bayered CFA frames. The flats must be exposed within the linear operating region of the sensor. The script also assumes that for subframes both were exposed similarly, and for integrations that both are similar combinations of the same number of similarly exposed subframes. By default the script analyzes the brightest 20% of the flats. For full frame analysis, you can specify a 100% foreground quantile.

Here are results for 3nm Ha uncalibrated panel flat subframes, binned 2x2 and exposed 24 seconds. SNR is the estimated signal to temporal noise ratio. Gain is the estimated sensor gain in e-/DN units. The gain estimate is within 1% of vendor provided estimate of 1.059 e-/DN.



Here are results for calibrated panel flat integrations, each an average combination of 64 subframes. The integration increased SNR by a factor of 7.96, nearly optimal. Gain increased by a factor of 63.4, again nearly optimal. The gain increase is due to frame averaging, each DN unit corresponds to more electron units in an integration than in a subframe. SNR exceeds 1000, which is a good rule of thumb target for high quality flat masters.



Flat SNR varies across the full frame due to vignetting. You can estimate the vignetting impact on SNR by using information from the script FlatContourPlot. The resulting plot shows equal illumination contours, and the legend shows the illumination range from brightest to dimmest. From the legend,  the relative illumination loss in the corners is about 1 - (0.399 / 0.425) = 6%. The relative SNR loss in the corners is about 1 - (0.399 / 0.425)^0.5 = 3%.



The table below shows results from a Canon EOS Rebel T3i DLSR. Flat exposures were 1/22s at ISO 1600. I extracted the four channels from the raw CFA flat and bias images and ran the script once for each channel. The two G channels received the highest exposure, and as a result have the highest estimated SNR. The estimated gain values are nearly identical, as would be expected for a CFA detector. SNR's overall are relatively low compared to those shown above due the short exposure. The camera is 14-bit with a maximum DN of 16k. When read the raw CFA images were normalized into the range [0, 0.25].

Channel   Signal DN   Noise DN RMS   SNR   Gain e-/DN
CFA0 R   4675   183   25.5   0.139
CFA1 G   7103   227   31.3   0.138
CFA2 G   7146   228   31.3   0.137
CFA3 B   4281   176   24.3   0.138

« Last Edit: 2014 December 19 17:07:14 by mschuster »