BZero and BScale not included in 32bit Float Files

[mcintyre_sj]

I have started to use some calibration and stacking sw that uses FIT files with 32bit float array values. But trying to use these file interchangably with PI is not working.

1) When PI writes a 32bit float, the bzero and bscale tags are not included in the FITs header.

2) PI is scaling or normilizing the original data to the [0..1] range.

I cannot therefore use the PI files in the calibration and stacking sw since there is no way to reproduce the physical values - ie something in the original ADU range (albeit now in 32bit float). The calibration and stacking sw needs values in the ADU range for pedistal, avg read out noise value, dark current etc.

3) PI is scaling the original data so that in PI min gets reset to 0 and the max to 65535 (from image statistics). PI seems to be scaling the data using the min of the original data as the zero point and the max of the original data as the range high end. "Bzero" and "bscale" are 0 and 1 in the original fits file before opening in PI with "bitpix" set to -32 (i presume this means 32bit float).
 
For instance, when i open a calibrated light sub[exposure] image into PI, the min from statistics is 0 and the max is 65535. Same if i open the bias frame. So there is no way to use these files together since the original zero and scale are lost and therefore there is no consitancy betwen the data ranges between the files.

I am assuming that PI will also scale the final stacked image. Which means there is no headroom for stretching. ie how can i stretch an image if the max is already 65535 and then save it later as a 16bit uInt to use in image editing sw.

Looking up FITs keywords at:
   http://heasarc.gsfc.nasa.gov/docs/fcg/standard_dict.html
i find this reference to using bzero and bscale:
   "physical_value = BZERO + BSCALE * array_value"

Which means that down stream sw could then recreate the values in ADUs.

[Juan Conejero]

I have started to use some calibration and stacking sw that uses FIT files with 32bit float array values.

Raw CCD data are 16-bit unsigned integers and hence should be stored in that format. An image calibration routine should use raw data, so it is at least surprising that a calibration software requires floating point data.

1) When PI writes a 32bit float, the bzero and bscale tags are not included in the FITs header.

Section 5.3 of the FITS standard document version 3.0 (page 14):

5.3. IEEE-754 Floating point
Transmission of 32- and 64-bit floating-point data within the
FITS format shall use the ANSI/IEEE-754 standard (IEEE
1985). BITPIX = -32 and BITPIX = -64 signify 32- and 64-
bit IEEE floating-point numbers, respectively; the absolute value
of BITPIX is used for computing the sizes of data structures. The
full IEEE set of number forms is allowed for FITS interchange,
including all special values.
The BLANK keyword should not be used when
BITPIX = -32 or -64; rather, the IEEE NaN should be
used to represent an undefined value. Use of the BSCALE and
BZERO keywords is not recommended.
Appendix E has additional details on the IEEE format.

Any competent FITS reader software should be able to correctly read and interpret a FITS file with no BSCALE and BZERO keywords, using the default values of these keywords (1.0 and 0.0, respectively). These keywords are not mandatory; its use is only necessary to store unsigned integer data (the well-known BSCALE/BZERO trick) since the FITS standard does not support unsigned integers natively, but it is explicitly discouraged for floating point data by the standard.

2) PI is scaling or normalizing the original data to the [0..1] range.

Not actually. Rescaling only takes place when a floating point FITS or TIFF file is being loaded and PixInsight does not know the actual range of pixel values to which the data are referred. In other words, PixInsight has to know which values correspond to the no signal and full signal states (usually corresponding to the black and white pixel value interpretations, respectively, in all supported color spaces). This is necessary to correctly interpret any image in PixInsight. In absence of a known data range, the only way to interpret an image is by assuming that min=black (or perhaps 0=black if min >= 0) and max=white, where min and max are the minimum and maximum values present in the data array.

Unfortunately, the TIFF and FITS standards do not provide a standard way to define the numeric range to which pixel values are referred in a floating point image (see the text marked as green in section 5.3 above). With integer data there are (almost) no problems, since integers are naturally bounded by their bit sizes. However, real numbers allow for arbitrary ranges, and here is where the craziness begins. Basically each application uses its own, usually undocumented and sometimes inconsistent range to store floating point images, and this is the cause of most of the existing problems to transport floating point data between applications.

PixInsight is very different from most applications in this regard, for the following reasons:

- PixInsight always uses the same numeric range to store floating point pixel data: the [0,1] range, where 0=black and 1=white. This feature is publicly documented and used consistently.

- PixInsight is extremely flexible to import floating point data in virtually any possible range. A custom default floating point input range can be defined with the FITS and TIFF format preferences dialogs, available from the Format Explorer window. The default input range of floating point data is [0,1], but it can be changed to meet the ranges used by alien data; for example [0.0,65535.0] is used by some well-known software packages.

- The ImageCalibration, StarAlignment, ImageIntegration and HDRComposition tools support a set of format hints (small words that change the behavior of a format support module) that allow you to load floating point FITS and TIFF files with arbitrary ranges, orientations, data formats, format features, etc. This has been a further step to provide the necessary flexibility to load data produced by virtually any application.

3) PI is scaling the original data so that in PI min gets reset to 0 and the max to 65535 (from image statistics). PI seems to be scaling the data using the min of the original data as the zero point and the max of the original data as the range high end. "Bzero" and "bscale" are 0 and 1 in the original fits file before opening in PI with "bitpix" set to -32 (i presume this means 32bit float).

I have already answered most of these questions in the preceding paragraphs. The BZERO and BSCALE keywords don't define the range of floating point data, so PixInsight does not rely on them for this purpose. You can load the original data without problems in PixInsight if you know the range to which they are referred (the black and white points). Once you know it, you can enter the corresponding default input range values in the FITS Format Preferences dialog, or use format hints in specific tools, as appropriate.

[mcintyre_sj]

Thanks for the detailed reply. I now have a better appreciation for the complexities.

Using the "Default Floating Point Input Range" in the FITS format preferences now allows me to load various 32bit float files into PI and see consitancy between them - a pixel with a value of 02602 in one file, means the same physical intensity as a pixel with that same value in another file. So i can now use files generated by the calibration and stacking app in PI.

However, i still cannot use files saved from PI in the stacking tools. The work flow is to calibrate the images using the other app. Then register (align) the sub exposured in PI. Then stack in the other calibration and stacking tool.

Using " lower-range 0.0 upper-range 65535.0 " as in input hint in the register process is fine also. But lower-range and upper-range are not accepted (just ingored) in the output-hints. This is consistent with the documentation on format hints. But still leaves me without being able to use PI in the middle of this process.

Nor does it allow me to use a mix of other tools in the post-stacking phase, since, as you state, "the TIFF and FITS standards do not provide a standard way to define the numeric range to which pixel values are referred in a floating point image".

You've obviously giving this a lot of thought and have a thorough understanding of the FITs standard, so i don't expect to sway your opinion. But it would seem to me that since the FITs standard does not have a mechanism when using floats to convert between physical and array data, nor even understand the ranges, that an application would be better off not mucking with it. Its fine to say PI is documented and consistent, but the output files are in a form not usuable by other apps. Unless they too provide some renormalizing function.

I agree that raw camera data is better as integer data. But calibration sw is taking dozens of inputs and making fine adjustments with a larger data set to the pixel data. So i think its fair game to expect the calibration sw to store data with more resolution. Floats allow this. Its the FITs header/std that is making it difficult to use floats.

btw, bzero and bscale don't need to be in the fits header when 1) its integer data and 2) the app is adhering to the defaults. I understand this does not apply to floats as per the standard.

[Juan Conejero]

So PixInsight is flexible enough to load valid FITS images generated by almost any application. On the other hand, FITS files generated by PixInsight are fully standards compliant and use numeric ranges that are publicly documented and consistent. Now what you are proposing is that we should be also flexible enough to generate images adapted to other applications. Flexibility is an important feature which we pursue in PixInsight, so a priori I'm not reluctant to do that. However, what seems funny to me is that most (if not almost all) of the other applications are unable to load floating point images generated by PixInsight. Have you asked the authors of the other application(s) if they could support floating point FITS images generated by PixInsight? It is very easy, and we are always willing to help. Why are always we who have to adapt to the others? Why are we treated as if we didn't exist?

That said, a future version of the FITS and TIFF format support modules will implement user-defined, arbitrary floating point output ranges. This has been planned some time ago, but we currently have many high-priority tasks. These versions might be available on Q1 2012, perhaps sooner.

BTW, I am sure you have good reasons to use a different software to calibrate and integrate your images. I honestly think that our ImageCalibration and ImageIntegration tools are powerful, flexible and accurate, but we are always open to ideas for improvement. So if you find that our tools are lacking some essential feature or functionality, I'd appreciate if you could describe them.

[mcintyre_sj]

I look forward to seeing the output range features in future.

It is true that the other apps are actually quite inflexible while PI is doing more to adapt. The other apps i think get by because they don't have any good reason to renormalize the data. If by change the data ranges between apps are the same, then they are "campatible". Chance is helped a bit because they start by assuming 16bit interger type ranges and then do things like clip hi and lo values to this range while still working with large floats. One app (which will remain nameless) simply truncates the decimnal portion of a float and deals with just the integer value - surprise!

I generally save as u16 if i do need to use another app for some unique feature or just that i like the app better for what ever reason. But then the data percision is lost.

I will give PI a try for calibration and stacking. As i mentioned, i was starting to use the registration feature. The new "other" calibration sw is home grown by a collegue. I had been using another commercial product that was not adequately removing dark frame artifacts. My collegues sw did a much better job.

Thanks for all the info.