Possible problem in BasicCCDParameters and CalculateSkyLimitedExposure

[dmcclain]

Both of these utilities depend on the underlying PI normalization to derive image related quantities. And both then scale the results by the CCD A/D depth.  But that is only correct for FITS files where the CCD data are left aligned in the FITS words.

So for example, in a Canon EOS 6D with 14-bit A/D, and with camera controllers that produce right-aligned data in 16-bit FITs format, and in the DSLR_RAW reader of PI, the maximum value possible in a frame will be 0.25 in normalized PI values. But when means and standard deviations are derived from images in PI and then scaled back to integer ADU by the CCD word size, the values will be off by a factor of 4 due to right-alignment in 16-bit words.

Hence, the gain in e-/ADU, dark current, full well capacity, and dynamic range will also be off by a factor of 4, as will estimated exposure times for sky limited exposures.

I don't know what to suggest here, because if another implementation provides left-aligned data it will produce correct values. There ought to be a way, akin to endian-specifiers, for the system to ascertain bit alignment in a FITs file.

[Juan Conejero]

By "left aligned" and "right aligned", do you refer to endianness? In the case of FITS, there is no choice in this regard, since the FITS standard clearly states that all the data stored in a FITS file have to be represented as big-endian; see section 5. Data Representation of the current FITS standard 3.0:

FITS data shall be interpreted to be a byte stream. Bytes are in big-endian order of decreasing significance. The byte that includes the sign bit shall be first, and the byte that has the ones bit shall be last.

Both of these utilities depend on the underlying PI normalization to derive image related quantities.

PixInsight does not "normalize" or modify unsigned integer pixel data in any way when it loads them from a disk file. For example, 16-bit raw data read from DSLR raw files or FITS files are loaded exactly as they are stored as unsigned 16-bit integers.

PixInsight supports 5 numerical formats transparently: 8, 16 and 32-bit unsigned integers, and 32 and 64-bit IEEE 754 floating point real and complex pixel values. Irrespective of the data format used, you can obtain readouts in a variety of numerical ranges and representations, including the [0,1] normalized real range that is used by default (see the ReadoutOptions tool). But the fact that you can read pixel values in the [0,1] range does not change the fact that a 16-bit integer image is stored and manipulated as 16-bit integers in memory.

The PixInsight JavaScript runtime (PJSR) virtualizes all pixel data as real numbers in the [0,1] format. A JavaScript script "sees" pixel values in the [0,1] format irrespective of the actual data format of the image. This is just a virtual representation managed internally by PJSR routines because the JavaScript language does not support integer data types formally. For example, for a 16-bit image, the translation between [0,65535] and [0,1] is performed as a simple multiplication/division. The transformation is always a simple linear scaling operation. 

But when means and standard deviations are derived from images in PI and then scaled back to integer ADU by the CCD word size, the values will be off by a factor of 4 due to right-alignment in 16-bit words.

Not at all. The endianness rules imposed or allowed by different file formats are managed in a completely transparent and automatic way by file format support modules in PixInsight (e.g., DSLR_RAW, TIFF and FITS). This means, for example, that numerical data read from big-endian FITS data streams are always converted to little-endian before storing them in memory, since as you know, all Intel/AMD architectures supported by PixInsight use the little-endian convention. The same happens when you load a TIFF file stored in big-endian mode. In other words: there are no endianness conflicts at all---and if there were one, it would be a maximum priority show-stopper bug for sure.

There ought to be a way, akin to endian-specifiers, for the system to ascertain bit alignment in a FITs file.

As noted above, there cannot be a way to indicate the endianness of a FITS file: All FITS data streams are stored as big-endian byte sequences. And if there are .fit files out there that don't follow this rule, they are not FITS files.