(in)Compatibility of fits floating point data format in PixInsight

[gvanhau]

Hello

Maxim does the following:
"One shortcoming of the FITS standard is that the 16-bit number format is signed, producing a range of -32768 to +32767. The range of data from many cameras is 0 to 65535. To overcome this problem, MaxIm DL subtracts 32768 from each pixel prior to saving. The FITS standard requires that the BZERO and BSCALE keywords be used when the value in the array is not the physical value; MaxIm DL sets BZERO to 32768 and BSCALE to 1. Other applications that interpret these keywords correctly should automatically add 32768 to the image pixels, resulting in a correct image display. This default behavior can be overridden using the File menu Settings command."

And this is just what the fits standard suggest including an example for it:
"Proposals to add 16- and 32-bit unsigned integer types have been discussed from time to time, but a consensus in favor of these proposals has not developed. Data with the numerical range corresponding to unsigned integers can be included in a FITS file by using signed integers and then scaling, for example, setting BZERO equal to 32768 (2^15) for 16-bit integers."

Geert

[gvanhau]

Hello

This is for Juan:

What units are used by PixInsight for saving images in floating point?

This is what The fits standard says about it:
3.1.1.4 Units

  The units named in the IAU (1988) Style Guide are recommended for array and keyword values, with the exception of measurements of angles. This set of units includes the SI units and additional units that are standard in astronomy, for example, magnitudes. Using BUNIT and TUNITn keywords will remove any confusion; similarly, the comment field following CTYPEn keywords can be used to specify the units. Fractional units should not be used. If, for example, for space reasons, temperatures must be listed in the file in units of tens of degrees, the units specified by BUNIT should be degrees, and scaling (section 3.1.2.1) should be used to transform from the file values to physical values. Always explicitly describe non-standard units. A copy of the IAU Style Guide recommendations is available on the IAU World Wide Web site, currently at http://www.iau.org/units.html.

Geert

[Juan Conejero]

Hi Geert,

What units are used by PixInsight for saving images in floating point?

In units of The Force power, of course.

Seriously, pixel sample values represent arbitrary units of intensity in PixInsight. How these values are to be interpreted is application-specific. For example, the FluxCalibration tool converts DN values (or instrumental pixel values) to spectral energy flux intensities measured in erg × cm^-2s^-1nm^-1. Once a set of images have been calibrated this way, the B3Estimator tool can use them to synthesize new images at arbitrary wavelengths.

This is what The fits standard says about it:
3.1.1.4 Units

The fragment you have excerpted is not part of the FITS standard. I have found it in A User's Guide for the Flexible Image Transport System (FITS), published in 1998. The current FITS standard is version 3.0, released 2008 July 10.

[gvanhau]

Hello
Even if I know beforehand that this discussion is hopeless,  see the attached image taken from the fits standard PDF.

Since BUNIT is not writen by PixInsight,  and BSCALE is not writen by pi in floating point and that  BSCALE=1 by default  and also 1 in PixInsight 32 bit integer output,  I want to know why PixInsight fits floating point output is at such different scale comparing to PixInsight  32 bit integer fits format.
For me they slould be the same value, or at least the same magnitude, even if at integer values you may have a rounding or truncation due to overflow or underflow.
We are talking about the same image, the same BUNIT(default), the same BSCALE so the formula BZERO+BSCALE x array_value should yeald the same physical_value.

Geert

[Andres.Pozo]

5.3. IEEE-754 floating point

Transmission  of  32-  and  64-bit  floating-point data  within  the FITS format shall use  the  ANSI / IEEE-754  standard  (IEEE1985). BITPIX = -32 and BITPIX = -64 signify 32- and 64-bit IEEE floating-point numbers, respectively; the absolute value of BITPIX is used for computing the sizes of data structures. The full IEEE set of number forms is allowed for FITS interchange, including all special values.
The BLANK keyword should not be used when BITPIX = -32 or -64; rather, the IEEE NaN should be used  to  represent an  undefined value.  Use  of  the BSCALE and BZERO keywords is not recommended.

The bold emphasis is mine.

[gvanhau]

5.3. IEEE-754 floating point

Transmission  of  32-  and  64-bit  floating-point data  within  the FITS format shall use  the  ANSI / IEEE-754  standard  (IEEE1985). BITPIX = -32 and BITPIX = -64 signify 32- and 64-bit IEEE floating-point numbers, respectively; the absolute value of BITPIX is used for computing the sizes of data structures. The full IEEE set of number forms is allowed for FITS interchange, including all special values.
The BLANK keyword should not be used when BITPIX = -32 or -64; rather, the IEEE NaN should be used  to  represent an  undefined value.  Use  of  the BSCALE and BZERO keywords is not recommended.

The bold emphasis is mine.

It is not recommended, because it was originally intended to allow fractional data when there was no floating point. It is not forbiden!!
The problem is if you don scale the floating point, using the apropiate exponent, then you should use at least use bscale to acomodate the values.
The point here is that PixInsight changes the data by not scaling the values and/or by not outputing the correct exponent.

Geert

[gvanhau]

This is what The fits standard says about it:
3.1.1.4 Units

The fragment you have excerpted is not part of the FITS standard. I have found it in A User's Guide for the Flexible Image Transport System (FITS), published in 1998. The current FITS standard is version 3.0, released 2008 July 10.

May be no the official text, but if it is referenced at the official site, I assume it is valid  you can follow the link at the official page (see red link in image )

Geert

[jkmorse]

How do I verify the Bzero/Bscale of the fits I import?

By inspecting the FITS header keywords after loading the image. You can use the FITSHeader tool, or even faster, select the image and enter the lskwd in the Process Console window. Make sure that BZERO is 32768.

For example, your "16bit_signed.JPG" screenshot shows an image correctly imported in 16-bit unsigned integer format. Judging by this screenshot, it seems MaxIm DL is using a sort of "inverse logic" in this case, since it writes unsigned integers with the "Save Data in Unsigned Format" option disabled. Anyway, this is how you have to write your raw images.

Juan,

That response may be exactly what I have been looking for for years on why I haven't been able to get MaximDL to play nice with PixInsight.  This has created enormous frustration but I will test this "inverse logic" myself and see if it doesn't fix my issue (hopefully for good).  Have been using the SkyX Camera Add on but for some programs such as ACP, Maxim is the only option.

Huge shout out to Aircrftr for bringing this to my attention. 

Best,

Jim

[jkmorse]

yes, 16-bit unsigned from maxim driven by ACP... camera is FLI16803.

Rob, just starting out on my remote adventure.  Exploring using either ACP or AutoPilot5.  Would love to hear how you like ACP.

Thanks,

Jim