SubframeSelector clipping too much data in low signal subs

[const]

I know, it is recommended to start with exposures high enough to decisively push the background areas off zero. Indeed, if number of image pixels that are less than master dark pixels is not so high, like 1%, you won't notice the problem.

For various reasons I often shoot short subs. One is my cheap mount that can handle 15s unguided and 60s guided. Another is exposing for stars of bright DSO cores, I intentionally shoot 2s at F/2.
Recently I shot the Eagle at F/7 through 6nm Ha filter with ASI2600MM-P. 60×60s subs at gain 100 (~0.25e/DN) offset 32 (zero=320 of 16-bit DNs). Each sub has about 20% pixels less than zero. And it is not bad data! After careful integration I got quite good picture with fainter dust at the edges.

OK, back to the problem. If I calibrate with output pedestal left at 0, I get those 20% pixels clipped to zero. Say, statistics shows median value 4.0 DNs if the Unclipped is checked. Otherwise it says the median is 7.0 DNs. So far so good. What is the true median sky background level? I would argue, it is 4. If it is not obvious, I have other data sets shots at two different exposures in the same night interleaved. Unclipped medians are very much proportional to exposures. SubframeSelector measures Median as clipped, or 7. I don't know what else follows from that. Likely, SNR is skewed similarly. It turns out, SubframeSelector not only uses 'non-unclipped statistics'. It clips all pixels with values less than or equal to 1 DN.

I usually calibrate with output pedestal high enough to avoid any clipping. 200 DN is usually good. In SubframeSelector I set pedestal to 200 accordingly and it mostly works. Except wrong medians and some other stats. I mostly use stars shape metrics that are not that much affected, so it worked for some time until I noticed.

Back to the experiment with subs having 20% zeros. I also tried to calibrate them without output pedestal. So the subs have 20% true zeros. Next I ran PixelMath on them with '$T + 1/65534'. Now SubframeSelector with Pedestal=0 shows Median=5, which is 1 more than true median, fine. If I change that Pedestal to 1, it says Median is 7 now. Notice, 1/65534 is a bit more than 1 DN, which would be 1/65535. Otherwise it would not be enough to convince SubframeSelector to not clip the pixels. As I trick the SubframeSelector to compute the median differently, SNRWeight also changes significantly. Other metrics also change, but I have not played with them extensively to conclude if that is significant or not.

It is unclear whether this clipping behavior exists in other processes and how much it affects them. For example, star detection in StarAlignment and location estimate in ImageIntegration.

Is this behavior intentional? If yes, how should I work with my data correctly?

 

[bulrichl]

I am not quite sure if I have understood you correctly. So I will describe the approach which is correct in my view:

When you detect that clipping occurs during subtraction of the MasterDark (MD) from the light frames (because the light frames contain very low signal due to e.g. very short exposure time or usage of a narrowband filter), you'll have to apply an output pedestal in the image calibration of the light frames. The output pedestal has to be adjusted that clipping is negligible (let's say <= 0.1 %).

In Subframe Selector you can choose whether intensity values are diplayed in DN or in units of e-. If you want to display intensity values in e-, you have to input the correct conversion gain that was used for the acquisition of the subframes. Add the light frames that were calibrated with the appropriate output pedestal (negligible clipping in the calibration result), let's say: the output pedestal is set to 200 DN. The Median values which are displayed in SFS are too large by that value. So if you chose 'DN' as unit for intensity values, you'll have to subtract the output pedestal (= 200 DN), if you chose 'e-', you'll have to subtract (200 DN * conversion gain).

The state of the 'Unclipped' option (unchecked or checked) in the Statistics process makes a difference for mean and median values only when clipping occurred. In the absence of clipping, the values are identical. Here is an example.

This histogram shows a light frame, MD-subtracted, no output pedestal applied:

Statistics:

_20210514_0110_cal_no_ped
Unclipped   unchecked   checked
count (%)   97.58642    100.00000
count (px)  25461906    26091648
mean        23.924      23.347
median      21.133      20.766
variance    28146.061   27480.215
stdDev      167.768     165.772
avgDev      13.875      14.174
MAD         11.837      12.132
minimum     0.000       0.000
maximum     65038.683   65038.683

And this histogram shows the same light frame, MD-subtracted, output pedestal of 100 DN applied:

Statistics:

_20210514_0110_cal_100_ped
Unclipped   unchecked   checked
count (%)   99.99867    100.00000
count (px)  26091302    26091648
mean        123.202     123.201
median      120.765     120.765
variance    27490.444   27490.280
stdDev      165.802     165.802
avgDev      14.353      14.355
MAD         12.132      12.132
minimum     0.065       0.000
maximum     65138.682   65138.682

The correct median in this case is 120.765 DN - 100 DN = 20.765 DN. If an appropriate output pedestal is applied (negligible clipping in the calibration result) the value is irrespective of the state of the 'Unclipped' option.

If this is not the answer you wanted, please explain.

Bernd

 

[const]

I am not quite sure if I have understood you correctly. So I will describe the approach which is correct in my view:

In Subframe Selector you can choose whether intensity values are diplayed in DN or in units of e-. If you want to display intensity values in e-, you have to input the correct conversion gain that was used for the acquisition of the subframes. Add the light frames that were calibrated with the appropriate output pedestal (negligible clipping in the calibration result), let's say: the output pedestal is set to 200 DN. The Median values which are displayed in SFS are too large by that value. So if you chose 'DN' as unit for intensity values, you'll have to subtract the output pedestal (= 200 DN), if you chose 'e-', you'll have to subtract (200 DN * conversion gain).

What you described is fine and works as expected. It seems you didn't make the last step to see the issue Let me emphasize that more clearly.

I calibrate with pedestal 200 DN, which is enough to have only 9 pixels clipped, great.

I claim the true median in this frame is 4.931 DN (204.931 from above stats minus pedestal 200).

SubframeSelector without adjusted pedestal:

Good, computed Median is 200 more than wanted. But it is not useful like this, because other measurements seem to depend on correct background level estimation. Only the very brightest stars are detected, SNR is likely off, etc.

Let's subtract the pedestal:

See!? It computed the median 7.543, not 4.931. It clipped internally all the camera noise preserved in calibration thanks to pedestal. In fact, I could have calibrated with pedestal 0 and get the same wrong median estimate in SubframeSelector. Median is almost twice higher than it should be. Other metrics are also off, but within reasonable bounds, so it is mostly useful. But still...

 

[bulrichl]

Now I understand the issue, and I agree completely. SFS should calculate the Median from the image that contains the pedestal and then subtract the pedestal. It should NOT subtract the pedestal from the image and then calculate the Median - this is a bug. Hopefully it will be fixed soon, it should be easy for a programmer.

I guess this is important for Mike Schuster's MureDenoise script?

Bernd

 

[fredvanner]

Hopefully it will be fixed soon, it should be easy for a programmer.

Having glanced at the code, I'm not so sure. The pedestal is subtracted from the image immediately after loading, before all processing. Calculating the median is just one of the many things done to the resulting image. Both the subtraction and the median calculation are not even in the SubframeSelector code (it uses the "apply, opSub" and "median" methods of the "image" object, which are embedded in the "image" object code).

 

[fredvanner]

The best compromise is to estimate the "real" pedestal, P (the one that would just not clip any real image data). Something like P=2xMAD would be a reasonable guess. Then specify the SS pedestal as (200-P). If you get P right this is not a fudge, it should give the correct results for all calculated parameters.

 

[bulrichl]

Perhaps we should leave the decision what to change in the code to the author of the SFS process (cameronleger)?

Bernd

 

[Juan Conejero]

Now I understand the issue, and I agree completely. SFS should calculate the Median from the image that contains the pedestal and then subtract the pedestal. It should NOT subtract the pedestal from the image and then calculate the Median - this is a bug.

I am not very sure about this. The pedestal has been added to the actual data acquired by the sensor, so the image with the pedestal added is not truly raw data. Maybe, to stay on the safe (and bright ) side of life, this could be implemented as an option, but I am not really sure.

 

[fredvanner]

Perhaps we should leave the decision what to change in the code to the author

Don't worry! I wouldn't dream of trying to change anything myself, I was just checking on what is involved; the way the code is arranged makes it more difficult than you might otherwise expect (my "best compromise" suggestion was as a user work-round, not a proposed code change).

The pedestal has been added to the actual data acquired by the sensor, so the image with the pedestal added is not truly raw data.

True, but...
... the image with the pedestal added (in ImageCalibration) and then subtracted again (in SubframeSelector) should be very close to the (calibrated) raw data.

 

[Juan Conejero]

... the image with the pedestal added (in ImageCalibration) and then subtracted again (in SubframeSelector) should be very close to the (calibrated) raw data.

Indeed, but then, shouldn't we calculate statistics after subtracting the pedestal in SFS, and not before? Because the intent is (or should be) to compute statistics representative of the calibrated raw data...

 

[fredvanner]

shouldn't we calculate statistics after subtracting the pedestal in SFS,

That is exactly what it does. Unfortunately, the subtraction clips the data, so we are back to square one with regard to clipping. That is why Bernd suggests calculating the median before subtracting the pedestal, then subtracting the pedestal from the result. However SS just isn't organised to make it easy to do this.

 

[Juan Conejero]

Hmm, it shouldn't clip anything, since our statistics calculation routines (median in particular) fully support floating point pixel data with negative values. I'll take a look to investigate why this truncation is happening.

 

[const]

Indeed, but then, shouldn't we calculate statistics after subtracting the pedestal in SFS, and not before? Because the intent is (or should be) to compute statistics representative of the calibrated raw data...

The important part, in my opinion, is to avoid clipping. Pedestals are there precisely for that. As I understand, the main purpose of pedestal is to augment the data when stored in a format that does not easily support negative values. Further, PI supports (with nuances) negative values in the internal representations. So from high level perspective, it is OK to subtract pedestal immediately on file loading and proceed with data potentially spanning below zero.
Edit: OK, you reached the same point a few seconds before I posted

 

[bulrichl]

The best compromise is to estimate the "real" pedestal, P (the one that would just not clip any real image data). Something like P=2xMAD would be a reasonable guess. Then specify the SS pedestal as (200-P). If you get P right this is not a fudge, it should give the correct results for all calculated parameters.

This suggestion does not work at all, why should it!

I have applied different pedestals to my example above (MAD = 12). The median values and the difference (median-pedestal) are as follows:

When I load the calibrated light frame that contains a pedestal of 24 (= 2*MAD) into SFS ("pedestal" in SFS/Star Detector Parameters = 0), it shows a median of 44.782. When I adjust the "pedestal" in SFS/Star Detector Parameters to 100 - 24 = 76, the histogram as expected will be heavily clipped and SFS shows a mean of 9.282.

Repeating the same with a calibrated light frame that contans a pedestal of 48 (= 4*MAD), the mean values are 68.765 (SFS "pedestal" = 0) and 17.916 (SFS "pedestal" = 52) respectively.

The correct value in my view is 20.765, calculated as:
mean(calibrated light frame with pedestal of x) - x. The order of steps matters here.

Bernd

 

[Juan Conejero]

PI supports (with nuances) negative values in the internal representations

No nuances AFAIK. The only limitation is that floating point pixel data must be referred to the normalized [0,1] range (0=black, 1=white) only when they are going to be represented on the screen. The rest of the time, real-valued (and complex-valued) floating point images can have arbitrary pixel sample values.

As I've said, there must be an error in SFS that is forcing it to clip all negative pixel values before calculating the median and other descriptive statistics. I am going to investigate this and will release an update to fix this problem as appropriate.

 

[fredvanner]

I'll take a look to investigate why this truncation is happening.

The pedestal subtraction is done by an "image.apply (pedestal, opSub)" right after loading the image. The "image.median" call is much later in the "measurement" code. I can't find any explicit normalisation in between (but I haven't looked that hard).

 

[fredvanner]

This suggestion does not work at all, why should it!

I think you have misunderstood both my suggestion and my intention.
Consider a simple image with just 5 pixels with ADU values 1, 2, 3, 4, 5. The mean (and median) value is clearly 3.
Now suppose a master dark that oversubtracts 3 ADU, so the calibrated values are -2, -1, 0, 1, 2.
These are clipped to 0,0,0,1,2 with mean 0.6 and median 0.
Iif we add a pedestal x=5, the calibrated values are now 3, 4, 5, 6, 7. The mean and median is 5.
Then (mean-x) = 5-5 = 0. This is indeed the mean of the oversubtracted calibration data.
But that is not what we want - we want the mean of the correctly calibrated data.
If we pick x = 3, we restore the "correct" values, and get the correct mean. That's what I meant by:

If you get P right this is not a fudge, it should give the correct results

But how do you pick x? In this simple example, if m is the minimum non-pedestal calibrated value (-2), then x=1-m would work.
The idea of my suggestion (median-2*MAD) is that this is towards the lower end of the true signal values, and thus a reasonable estimate of a zero signal reference value (which is what we really want as the pedestal value).
Why bother with this "fixup"?
The OP believes that:

... other measurements seem to depend on correct background level estimation. Only the very brightest stars are detected, SNR is likely off, etc.

So the question I was trying to answer is "can I find a compromise value of x to specify in SS, likely to reduce these impacts on other performance".
I never intended to suggest that this was correct, but (mean-x) isn't correct either if we are looking for the true signal mean.

 

[bulrichl]

I cannot follow your argumentation. If the camera works correctly, and the settings and conditions for the acquisition of light and dark frames match (gain, offset, exposure time, sensor temperature, no light leakage, etc.), what is the reason to conclude that some negative pixel values are caused by "oversubtraction"? This word implies that these negative values are generated by a fault, and that the calibration result is wrong. Your precondition is that single pixels must not assume a negative value after image calibration, but this is arbitrary in my view. Negative values in single pixels can occur after image calibration. As long as there is no evidence for a systematic error, there is no reason to arbitrarily apply corrections to the calibration result.

If clipping occurs in image calibration, this is normally due to low signal in the light frames. An appropriate output pedestal has to be applied in order that clipping is avoided. Then the location information (e.g. the median value of the calibrated image) of the calibrated light frame has to be corrected by the applied pedestal - that's it.

Bernd

 

[fredvanner]

My camera only records positive ADU numbers. These are made up of a camera-imposed bias, dark current electrons, and detected photoelectrons, all of which increase the ADU value from a base of zero. The purpose of calibration, as I understand it is to estimate and subtract the first two components, leaving only the detected light signal. Since I can only detect positive numbers of photons, this detected light signal must be greater than or equal to zero. If calibration results in a negative signal, I would claim that the calibration is wrong (or it is not doing the task I have described above, and is therefore not "calibration" in this sense). The fact that PI has no way of displaying negative signal values in images suggests that a similar assumption is deeply embedded in the PI model (I know and fully understand that internally images are floating point arrays that can contain any valid FP values - including extended values like NaN and Inf - but when they are interpreted as images for display, they must be non-negative - and I assume that includes calibrated images). In any case, I believe that the signal values used by SubframeSelector (and other image statistics) are based on the assumption that image signal values are non-negative.
Don't get me wrong. I fully understand that a statistically averaged master dark subtracted from an image with low signal values may result in negative values - but I would assert that, for this particular image, this is by definition an oversubtraction, since there is no such thing as negative photoelectron detections. A pedestal is used to compensate for occasional oversubtraction. Examination of an unclipped calibrated data histogram would support estimation of the minimum signal value (excluding artefacts), and this would indicate the minimum pedestal value required to ensure no clipping. Application of this minimum pedestal should really be regarded as part of the calibration.

 

[const]

My camera only records positive ADU numbers. These are made up of a camera-imposed bias, dark current electrons, and detected photoelectrons, all of which increase the ADU value from a base of zero. The purpose of calibration, as I understand it is to estimate and subtract the first two components, leaving only the detected light signal. Since I can only detect positive numbers of photons, this detected light signal must be greater than or equal to zero. If calibration results in a negative signal, I would claim that the calibration is wrong (or it is not doing the task I have described above, and is therefore not "calibration" in this sense).

Did you forget the read noise? It exists in both light frames at full scale and in master calibration frames at reduced (thanks to integration) scale. Camera captures only positive number of photons, but together with the noise the output DNs can legitimately fall below logical zero. Which is mitigated by camera's own offset/bias/brightness. The mitigation is later negated by calibration. (Even more, the noise from light and calibration frames is mashed up together, so I would expect calibrated frame to have somewhat larger noise scale than a single light frame.) Negative individual calibrated values are not a direct sign of overcorrection. Negative median would definitely be.
As a most extreme example, take a single bias frame and calibrate with master bias. You should expect zero median and half pixels below zero.