DSLR Pure Raw decoding gives inusual low values of pixels

[Astrocava]

Hi all!

I'm starting again to process my images after a long break and I'm having problems. If I set DSRL Raw settings in Format explorer to "Pure Raw" I get extremely low readings with k around e-2 in the normalized 0-1 range.
The histogram in other programs shows the peak well separated from the left part in light frame, around 60% in flat frame and at 0% for bias frame so I can't explain what's happening. This seems to lead to bad flat correction, even to messages abut its near zero values when bias substracting.

I've shared in Google Drive 3  .CR2 files one flat, one bias and one light so you can try and give me some hints.

Thanks in advance,

Sergio

[msmythers]

Sergio

I'm not the one to answer your readings questions. What I did do though was make 2 copies of each file and then run the BPP script using percentile clipping defaults. I just wanted to visually 'see' the image after calibration with these files. The flat correction 'looks' good to me for such a simple test. I am also using the 'Pure Raw' defaults. The stretch was simply STF auto unlinked.


Mike

[Astrocava]

Sergio

I'm not the one to answer your readings questions. What I did do though was make 2 copies of each file and then run the BPP script using percentile clipping defaults. I just wanted to visually 'see' the image after calibration with these files. The flat correction 'looks' good to me for such a simple test. I am also using the 'Pure Raw' defaults. The stretch was simply STF auto unlinked.


Mike

Thanks Mike for answering and playing with the files.

When integrating with BPP with all frames following your method, It seems that makes a good correction in the central part, but not in the borders. I suspect some truncating errors there, still don't know why the difference of intensity between the BIAS and the FLAT is so small according with statistics.

These are the statistics
BIAS

Code: [Select]

B_M12_5387
            K
count (%)   100.00000
count (px)  18024930
mean        3.1250e-002
median      3.1250e-002
avgDev      1.2207e-004
MAD         1.1312e-004
minimum     2.8809e-002
maximum     3.3494e-002
FLAT

Code: [Select]

IMG_0001
            K
count (%)   100.00000
count (px)  18024930
mean        4.7511e-002
median      4.9836e-002
avgDev      4.0327e-003
MAD         1.4479e-003
minimum     3.6484e-002
maximum     5.5055e-002

[msmythers]

Here is that single sub with and without using ABE.


Mike

[Astrocava]

Thanks again

The same here, with the integrated light. Think I need to work more with ABE or DBE, but is strange, I've never had this kind of problems before.

[bulrichl]

I'm starting again to process my images after a long break and I'm having problems. If I set DSRL Raw settings in Format explorer to "Pure Raw" I get extremely low readings with k around e-2 in the normalized 0-1 range.
The histogram in other programs shows the peak well separated from the left part in light frame, around 60% in flat frame and at 0% for bias frame so I can't explain what's happening. This seems to lead to bad flat correction, even to messages abut its near zero values when bias substracting.

Yes, 'Pure Raw' is the right setting in DSLR_RAW. Don't care about the low readings, it really is no problem. The histogram displayed on the camera monitor respectively displayed by most other programs show gamma stretched data, not the linear data like Pixinsight does. You cannot campare that directly (also see below).

If I understand you correctly you fear, that clipping of data (zero values) may occur during calibration:
1) There are no clipped (zero) pixels in the three frames (see ImageInspection/Statistics and the histogram in IntensityTranformations/HistogramTransformation).
2) When subtracting Bias from the light frame (L-B) and from the flat frame (F-B) with PixelMath, no clipping (zero values) results either.
Conclusion: there is no clipping with your data during calibration provided that the calibration parameters are set correctly.


Your frames are from a 550D which according to Wikipedia has a 14-bit A/D converter. Theoretically this means a range of 16383 ADus. However, the maximum possible intensity of your camera can be gathered from the light frame: it is limited to around 15300 ADUs which stems from some saturated hot pixels and the brightest star in the upper left. That's normal; my 600D is similar and also has a maximum intensity of 15300 ADUs. That means: the available range of your camera goes from 2048 (minimum = Bias) to 15300 (maximum = saturation) ADUs, the difference beeing about 13250.

Now you should take a look at the statistics and the histogram of your flat frame:

IMG_0002
count (%)   100.00000
count (px)  18024930
mean        3090.3
median      3239.0
stdDev      304.9
avgDev      206.0
MAD         63.0
minimum     2376.0
maximum     3568.0

There are two peaks in the histogram. In your case the peak at lower intensity corresponds to the red channel, the peak at higher intensity to the 2 green and the blue channel. The peaks are narrow which is very well. The peak at higher intensity has maximum counts around 3270 ADUs. The maximum value in the flat frame can be gathered from Statistics: 3568 ADUs, about 300 ADUs more.

What you really want to avoid is any saturated (clipped) pixels in the flat frame. So let's consider only the peak at higher intensity.  The center of this peak shall be situated in the middle of the available range: at about (2048 + (15300 - 2048) / 2) = 8700 ADUs. In your flat frame, its center is situated at 3270 ADUs.

You gained only (3270 - 2048) = 1222 ADUs by the exposure of your flat frames, but should have gained about (8700 - 2048) = 6652. So my conclusion is: If the conditions of your flat frame acquisition are reproducible, you should extend the flat frame exposure time by a factor of about 5. OK, the frames are 3.5 years old, and you will not have the chance to make new flat frames. So take it as a suggestion for the future. In any case, you have to judge the exposure of the flat frames at linear state. Use the great tools of Pixinsight for that purpose (Statistics and the histogram in HistogramTransformation), don't rely upon the camera's histogram or programs displaying histograms of gamma stretched data. However, I don't believe it would have avoided the special problem with the integration.

Mike already has shown before that the result of the flat frame correction seems to be OK and that the remainig unevenness of the background can be removed well by DBE. I think the strange look of the integration and the not optimal result when applying DBE to the integration is attributable to changing conditions with regard to straylight (light pollution, air glow and/or changing thin clouds) during the exposure session. Did you try to apply DBE to each of the calibrated light frames before the integration?


I guess you didn't use darkframes, because you acquired the light frames at low temperature (camera temperature was 3 °C for the light frame shared) and there is low noise. Did you eventually check whether dark frames further improve the SNR?


Bernd

[Astrocava]

I'm starting again to process my images after a long break and I'm having problems. If I set DSRL Raw settings in Format explorer to "Pure Raw" I get extremely low readings with k around e-2 in the normalized 0-1 range.
The histogram in other programs shows the peak well separated from the left part in light frame, around 60% in flat frame and at 0% for bias frame so I can't explain what's happening. This seems to lead to bad flat correction, even to messages abut its near zero values when bias substracting.

Yes, 'Pure Raw' is the right setting in DSLR_RAW. Don't care about the low readings, it really is no problem. The histogram displayed on the camera monitor respectively displayed by most other programs show gamma stretched data, not the linear data like Pixinsight does. You cannot campare that directly (also see below).

If I understand you correctly you fear, that clipping of data (zero values) may occur during calibration:
1) There are no clipped (zero) pixels in the three frames (see ImageInspection/Statistics and the histogram in IntensityTranformations/HistogramTransformation).
2) When subtracting Bias from the light frame (L-B) and from the flat frame (F-B) with PixelMath, no clipping (zero values) results either.
Conclusion: there is no clipping with your data during calibration provided that the calibration parameters are set correctly.

Yes, that's my fear. Well, after your explanation it "was" my fear. So, I'm in the good direction pure raw and while there are no zero values, PixInsight can handle them.

Your frames are from a 550D which according to Wikipedia has a 14-bit A/D converter. Theoretically this means a range of 16383 ADus. However, the maximum possible intensity of your camera can be gathered from the light frame: it is limited to around 15300 ADUs which stems from some saturated hot pixels and the brightest star in the upper left. That's normal; my 600D is similar and also has a maximum intensity of 15300 ADUs. That means: the available range of your camera goes from 2048 (minimum = Bias) to 15300 (maximum = saturation) ADUs, the difference beeing about 13250.

Now you should take a look at the statistics and the histogram of your flat frame:

IMG_0002
count (%)   100.00000
count (px)  18024930
mean        3090.3
median      3239.0
stdDev      304.9
avgDev      206.0
MAD         63.0
minimum     2376.0
maximum     3568.0

There are two peaks in the histogram. In your case the peak at lower intensity corresponds to the red channel, the peak at higher intensity to the 2 green and the blue channel. The peaks are narrow which is very well. The peak at higher intensity has maximum counts around 3270 ADUs. The maximum value in the flat frame can be gathered from Statistics: 3568 ADUs, about 300 ADUs more.

What you really want to avoid is any saturated (clipped) pixels in the flat frame. So let's consider only the peak at higher intensity.  The center of this peak shall be situated in the middle of the available range: at about (2048 + (15300 - 2048) / 2) = 8700 ADUs. In your flat frame, its center is situated at 3270 ADUs.

You gained only (3270 - 2048) = 1222 ADUs by the exposure of your flat frames, but should have gained about (8700 - 2048) = 6652. So my conclusion is: If the conditions of your flat frame acquisition are reproducible, you should extend the flat frame exposure time by a factor of about 5. OK, the frames are 3.5 years old, and you will not have the chance to make new flat frames. So take it as a suggestion for the future. In any case, you have to judge the exposure of the flat frames at linear state. Use the great tools of Pixinsight for that purpose (Statistics and the histogram in HistogramTransformation), don't rely upon the camera's histogram or programs displaying histograms of gamma stretched data. However, I don't believe it would have avoided the special problem with the integration.

Yes, you nailed it down. Fooled by the in-camera and capture program histograms I ended up with a low intensity flat . I cannot make them again because I have sold both camera and telescope, but is a great advice. It seems is useful to have PixInsight installed in the capture laptop too and check the flats on the fly to adjust the values. I don't know if I can test your recommendation in a near future because I'm going to change to lucky imaging to give it a try.

Mike already has shown before that the result of the flat frame correction seems to be OK and that the remainig unevenness of the background can be removed well by DBE. I think the strange look of the integration and the not optimal result when applying DBE to the integration is attributable to changing conditions with regard to straylight (light pollution, air glow and/or changing thin clouds) during the exposure session. Did you try to apply DBE to each of the calibrated light frames before the integration?

You are right again... There is a change along the frames, don't know if it is related with high clouds or increasing moisture in air, but definitively, it's there. I've attached the preview of the complete series. I'll try to pass a DBE to each calibrated frame before integration but I don't know when and how to correct them when using drizzleintegration.

I guess you didn't use darkframes, because you acquired the light frames at low temperature (camera temperature was 3 °C for the light frame shared) and there is low noise. Did you eventually check whether dark frames further improve the SNR?

I've started to avoid darks with the 350D relying in a good dithering, and didn't change my procedure with the cooled 550D, so I can't answer your question.

Thanks Bernd for your detailed answer, it has helped me a lot now that I want to try to free some HDD space integrating all the objects that remain forgotten in its guts.