Re-scaling Bug for DSLR Pure Raw?

[Howdy]

I'm not sure if this is a bug, or I'm just not understanding some aspect of the Pixinsight workflow when dealing with DSLR raw images.

It seems to me that a 14bit DSLR image that is opened in RAW mode is incorrectly treated as if it is a 16bit image. (ie, as if the camera had recorded pixels in the range of 0,65535 rather than a range of 0,16383).

I have a .NEF image taken during the eclipse with a Nikon D810, thus a 14bit image.  The particular image has intentionally saturated pixels (ie, overexposed near sun to get outer corona).  As such, the saturated pixel value should be 16383.

If I open the file as Pure Raw ( Create raw Bayer CFA, No White Balance)
Set statistics process to report as 14-bit [0,16363]:
- minimum pixel value is 160
- maximum pixel value is 4095.6  (ie, one-quarter of what it should be).

Set statistics process to report as 16-bit [0,65353]
- minimum pixel value is 640
- maximum pixel value is 16363 (saturated pixel for a 14bit image).

Set statistis process to report as Normalized Real [0,1]
- minimum pixel value is .0097658
- maximum is .25 (actually, it reports as .249999)

If I open HistogramTransformation, the saturated pixels show a definite peak at .25

In short, it is treating the 14bit image AS IF it was a 16bit image with all pixel values in the range 0,16383.  Regardless of the reporting range, the saturated pixels are reported with a value that is 1/4 (.25) of the maximum of that range.

(The same occurs if I open as RAW Bayer RGB, except the Minimum value for each channel reports as zero.)

To actually pre-process these files and have the Saturated Pixels treated as Saturated, I have to apply the HT process, bringing white point down to .25 (And otherwise leaving the image linear, ie, no adjustment to Shadows or Midtones).

Conversely, IF I open the same file as a deBayered RGB (VNG, but NO White Balance), the image is rescaled in the process so that, regardless of the pixel range, the saturated pixels always report as the maximum value for the reporting range.

To my mind, if the recording range is 14bit, then it should be opened as a 14bit image, rather than a 16bit image.

[pfile]

this is how PI has always done it - it is calling DCRAW with arguments such that you get the raw 14-bit data. generally speaking since there's no such thing as the i14 format, the data is stored in 16-bit integers with the top two bits set to 2'b00. this is actually how DCRAW returns the data when called the way PI calls it. DCRAW also has an argument (-H i think) that lets you control whether or not the data is rescaled. however, this only does anything when DCRAW is commanded to debayer the image, so it's not applicable to "pure raw" which is what you absolutely need in order to properly calibrate your images.

setting the stats process to report 14 bits, while it seems to make sense, actually does not, because the data is now 16 bit data that ranges from 0-16383 as you have found.

overall this probably does not matter too much, but it is responsible for the pink hue you find in calibrated DSLR images where the image is overexposed. i believe this is because since the saturated data is not represented as 1.0, after flattening you get different, but non-1.0, values in each channel where the data is saturated.

note that if you are going to do HDRComposition with your data, the fact that the saturation point for the R, G and B data differs due to the color of the corona means that HDRC probably will not work that well on the RGB files. you'll get better results if you split the debayered files into their component channels and do the HDRComposition on the mono files. i believe this is because HDRC only computes a single mask for each step and that mask will be wrong for one or more of the channels due to the differing pixel values at saturation.

rob

[Juan Conejero]

To my mind, if the recording range is 14bit, then it should be opened as a 14bit image, rather than a 16bit image.

When you load the data as pure raw, they are loaded uncompressed exactly as they are stored in the device. As Rob has said, an image cannot be stored in memory as a sequence of contiguous 14-bit chunks (well, strictly speaking, this is doable, but the resulting data structure would be extremely inefficient, to the point that it would be unmanageable). Pixel data have to be stored as sequences of contiguous bytes, and two bytes are required to store 14 bits. Excess most significant bits, 2 of each 16-bit word in this case, are set to zero. The resulting range is [0,65535] instead of [0,16383]. This is how digital images work, and also how the data are acquired and stored in your camera.

To actually pre-process these files and have the Saturated Pixels treated as Saturated, I have to apply the HT process, bringing white point down to .25 (And otherwise leaving the image linear, ie, no adjustment to Shadows or Midtones).

Not at all. Just let the image calibration, registration and integration tasks do their work. They have been designed to work with the data as they are. Image integration handles the data adaptively based on robust statistics, so the actual range of pixel values, provided it is consistent for the entire data set, is irrelevant.

[Howdy]

Thanks for the replies.  I understood it was a 14 bit image in a two-byte format.  My intuition suggested that, once opened in  to Pixinsight, it would be scaled to [0,1], and that seemed the way Pixinsight was treating the files when opened as deBayered.  Your insights were helpful!