K-Value versus statistics

[wkrispler]

Hi,
I have a correct exposed flat from a 14bit DSLR (Max ADU 16384).
Mean value is 11.500 - 70% saturation--> good!
Please have a look at the attachment.

Why does the k-value (normalized) on the bottom left show 0.135 then? Should that not be
0.7? I do not understand..

Many thanks for your help.
All the Best,
Wolfgang

[bulrichl]

I don't understand your question. In PixInsight you can configure Readout options either to display intensities in Normalized real range or in Integer range. The displayed value also critically depends on other settings of Readout options: e.g. Readout mode (Mean, Median, Minimum, Maximum) or Probe size. Of course there is a large difference whether you set Probe size to 1 pixel or say 15 pixels, in which case the intensities of this number of pixels is averaged. So the displayed value depends on your settings.

Besides, since you obviously are using a DSLR camera (an OSC camera), the image is still mosaiced, and the different CFA channels may have different mean values. In order to see this, take a look at the histogram. You can also split the raw "image" into channels with SplitCFA and take a look at the statistics of each channel separately.

Hope this helps,

Bernd

[wkrispler]

Hi Bernd,

thank you for your feedback!

I know that there are different settings...My setting is normalized real range 0.001.
 The probe size in a flat does not really matter as it varies just between 0.14 (single pixel) and 0.17 (probe size 5).

My question is , if the flat is 70% saturated why the k-value does not show approx. 0.7? The statistics process shows an ADU value of 11500 out of 16384 (14 bit camera).

I am also aware that the different CFA channels may have different mean values. But I am talking about the raw image before debayering.

Why does the k-value show 0.14 or 0.17 and not 0.7. This is a huge difference.
Thank you,
Wolfgang

[bulrichl]

Two points:

When using the proprietary raw format of the camera (Canon's CR2 format), there is NO scaling of the intensities (note that this is different when using FITS files with the same camera!). So PixInsight gets intensity values (16-bit unsigned integers, range [0,65535]) that do not exceed a maximum value of 16383. These values are held in floating point format internally in PixInsight and the unsigned integer range of [0,65535] is rescaled to the range of [0,1]. So a value of just under 0.25 in this case means saturation.

It is not a given at all (particularly for flat frames) that the CFA channels have the same mean or median. When judging whether the exposure time of a flat frame is appropriate, all channels must be considered. This is easier when you split the channels. So please do what I suggested above: apply SplitCFA and take a look at the histograms and statistics of all individual channels. This is much more meaningful than one averaged value for the whole data.

Bernd

[ngc1535]

Hi Bernd,

I think you may have gotten sidetracked by Wolfgang's mention of the mean brightness- although your point about the mean values being different in each bayer filter is interesting.
The fundamental question was (which I believe you answered, see below), why does he read 0.135 instead of 0.7 when he reads in his 14-bit camera RAW image?
His flats have values that correspond to 70% of the full 14-bit range.

Your answer, if I understand correctly, is that the image is read in as a 16-bit unsigned integer (range [0,65535]) instead of data in the range of 14-bits (range [0,16383]).
Thus his value os 11,500/65535 = .18  ... or as you mentioned a maximum value of 16383/65535 = 0.25   ... which if you take  0.18/0.25 you get 72%... (QED) . lol

The follow-up question- and likely answered elsewhere- is there an option to force a camera RAW image to read in as a user supplied bit-value?
There is no loss of information in leaving the maximum of 0.25... it is really a question of consistency in what ".25" means with respect to the range of the original input data.

Your note about the FITS behavior is very much appreciated!

-adam

[pfile]

your understanding is correct - but it does go farther than this. some canon cameras have 12-bit ADCs so in that case 4095/65535 would be full saturation (0.0625). however there haven't been any 12-bit cameras from canon in a long time so this is probably not something we'll see a lot more of.

i don't know if it is universally true that FITS files will be expanded, but it is true that Sequence Generator Pro expands non-16 bit data to fill the 16-bit space before writing a FITS file from a connected DSLR. i don't know what other capture programs do but there's nothing that says they have to behave this way if writing out FITS.

with PI now using libRAW i'm not sure if there is an option to expand the data. i think DCRAW could do it but i'm not sure if there was a way to configure PI so that it would call DCRAW with the argument it wants to do the expansion (-d). i think perhaps in libRAW unticking the "no highlights clipping" checkbox will expand the data, since it mentions the "pink shades" issue you sometimes see with 14-bit cameras when the data has not been expanded.

rob

[wkrispler]

Many thanks to all of you! I finally have understood the k-value for my 14bit camera.
It though is confusing that you always have to think about the dynamic range of the camera before being able to interpret the k-value
correctly. It should also be possible in the Format Explorer to define the input range for raw files based on the camera one is using.

Best Wishes,
Wolfgang

[bulrichl]

@Adam:
Thanks for explaining intelligibly what I meant to say.

I think you may have gotten sidetracked by Wolfgang's mention of the mean brightness- although your point about the mean values being different in each bayer filter is interesting.

The effect of differing mean values in each bayer filter is especially surprising in the way how PixInsight handles the flat correction of bayered data. I will open a new thread with that topic.

i don't know if it is universally true that FITS files will be expanded, but it is true that Sequence Generator Pro expands non-16 bit data to fill the 16-bit space before writing a FITS file from a connected DSLR. i don't know what other capture programs do but there's nothing that says they have to behave this way if writing out FITS.

with PI now using libRAW i'm not sure if there is an option to expand the data. i think DCRAW could do it but i'm not sure if there was a way to configure PI so that it would call DCRAW with the argument it wants to do the expansion (-d). i think perhaps in libRAW unticking the "no highlights clipping" checkbox will expand the data, since it mentions the "pink shades" issue you sometimes see with 14-bit cameras when the data has not been expanded.

My guess is that in case of a CMOS astro camera, it is the camera driver that does the scaling, and SGP does the same for DSLR cameras when saving the data as FITS file.

Disabling the 'No highlights clipping' option in RAW format preferences produces a different result: in this case, the highest actually found intensity value of the input data is set to 1 in the range of [0,1].

It should also be possible in the Format Explorer to define the input range for raw files based on the camera one is using.

I know of only one setting of an input range in PixInsight: in the FITS format preferences ('Default floating point input range'). Of course it should be possible to implement a similar input range for RAW format preferences. The question is whether this is desirable. A change would probably cause side effects in some scripts, e.g. BasicCCDParameters.

Bernd

[pfile]

yes you are right, these libRAW/DCRAW options are normalizing the data to [0,1] rather than just scaling the data to 16-bit.

i don't know why SGP chose to do the expansion; it has been several years since i looked at this but i seem to recall they are just using DCRAW in a similar fashion to normalize the data. i guess if a driver expands the x-bit data to 16-bits then in my mind that's sort of part of the hardware, but the canon SDK does not do this and so i would have hoped that SGP left the data alone - to me it's a sin for a capture application to silently modify the data coming from the camera. i did ask the developers about adding an option not to do this in SGP but they did not want to change things - once they started doing it this way it was too late. but it does make CR2 and FITS from SGP incompatible with one another which occasionally leads to people having trouble with calibration.

rob

[bulrichl]

I agree completely with your statement:

"it's a sin for a capture application to silently modify the data coming from the camera. i did ask the developers about adding an option not to do this in SGP but they did not want to change things - once they started doing it this way it was too late. but it does make CR2 and FITS from SGP incompatible with one another which occasionally leads to people having trouble with calibration."

At least with the 600D, with a 14-bit ADC, the scaling can be reversed though, see
https://forum.mainsequencesoftware.com/t/option-to-not-scale-dslr-fits-files/5395/3

Bernd

[pfile]

that's good to know, i didn't see that thread. i don't really agree with jared that CR2/unscaled FITS data would necessarily still be incompatible.

anyway for me this is largely a thing of the past for me, though ironically rescaling of the 14-bit data into 16-bit space was necessary when processing my 2017 eclipse images.

rob