Author Topic: How to read master made by other programs (FITS format issue)  (Read 6901 times)

Offline bitli

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 513
Hello,
When I process images from a remote telescope, I receive master darks/bias created by another program (probably maximdl). These images are in 16 bits integer format(BITPIX 16, BZERO 32767, BSCALE 1, SAT_LEVE 50000)- Although the other programs read the files correctly (the values are scattered around 1000 with a few hot pixels around), PI always load an uniformely gray images with all values at 0.5.  I tried a couple of options on the Fits perference, but they have no effect (and probably shouldn't in this case).

Any idea how to read them (other than ordering MaximDL to convert the images,) ?

-- bitli

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
by any chance are you talking about lightbuckets?

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1458
    • http://www.harrysastroshed.com
Hi

Checking the " signed integer stores data " box normaly sort this

Or a quick fix is to click the rescale icon

Harry
Harry Page

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Hi Bitli,

Quote
BITPIX 16, BZERO 32767, BSCALE 1

There should be no problems with this FITS file; it is a standard file storing 16-bit unsigned integers. If you open FITS Preferences from the Format Explorer window, then click Reset and OK, these files should load fine.

If this doesn't work, then I need one of these files to see what happens.

Quote
Or a quick fix is to click the rescale icon

Note that this will invalidate the image as a master calibration frame.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline bitli

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 513
Thanks, I will try the 'reset' trick when back at home, and send the file if this is not solved.
No it is not LB, but similar.  And I had the problem with LB too some time ago.
-- bitli

Offline bitli

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 513
Tried, did not work.
I have sent the image address via PM.
Clear sky!
-- bitli

Offline bitli

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 513
I found some interresting stuff.

If change the FITS keyword BZERO from 32767 to 32768 (with an hexadecimal editor, directly peeking in the file), then PI can read the file properly. Not exactly convenient, but works...
I am not sure if this does not introduce a '1' offset to the file (anyhow the values are around 1000, so this is probably not meaningful).

Also I found that FITSLibrerator can read the file properly and write it back as TIFF.
 
A more convenient solution would still be appreciated, but I can survive for the couple of files I have to transfer.

Good night
-- bitli

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Hi Bitli,

Thank you for the sample file you uploaded for me. The problem is with the BZERO keyword value. This file has the following keywords:

BITPIX=16
BZERO=32767
BSCALE=1

As you know, the FITS standard does not support unsigned integer data natively. To overcome this, the well-known 'BZERO trick' is always used as a de-facto standard. The trick consists of setting these keywords:

BITPIX=16
BZERO=32768
BSCALE=1

to transform the 16-bit signed integer values stored in the file into unsigned integer numbers with the same bit length. As specified by the FITS standard, the following operation has to be applied to each pixel to convert the stored values into physical data:

physical_pixel_value = BZERO + BSCALE*fits_pixel_value

So if we have BZERO=32768 and BSCALE=1, the above transformation converts the stored data from its original signed range [-32768,+32767] into the unsigned range [0,65535]. This is how all unsigned integer images are managed as FITS files.

In the case of the file you have provided, however, we have BZERO=32767. So the transformation is as follows:

[-32768,+32767] -> [-1,+65534]

So the output pixel data are still signed integer values. Even worse, the output range cannot be represented with 16-bit integers. It requires 32-bit signed integers. Indeed, when you load this file in PixInsight you get a 32-bit image, not a 16-bit image as expected. Note that this conversion is automatically performed by CFITSIO (which out FITS support module uses internally) as a result of the exotic BZERO value.

In my opinion, this BZERO value is an error. It is fully standards-compliant, that's true (because the FITS standard does not impose any particular value for this keyword), but I cannot see any valid reason to use BZERO=32767. In my opinion also, the way PixInsight is reading this file is the correct way.

Naturally, I can modify our FITS module to support these files, for the sake of compatibility. It is not very easy, but of course can be done. However, please realize that my workload is currently so huge that fixing this problem has to be a very low-priority task.

The best workaround to this problem is patching the file to replace 32767 with 32768. This can be done very easily with a binary file editor, or you can also write a simple PixInsight script to perform this task automatically. If the file has been generated in a coherent way (and we have no reason to doubt about this), by doing this patch you'll load the image with a constant pedestal of +1 added to all pixel values. This shouldn't be a problem if you calibrate with dark frame optimization enabled. To be meticulous, you can subtract this pedestal very easily with PixelMath:

$T - 1/65536

with the rescale result option disabled.

Hope this helps.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Just another quick note to say something I forgot in my previous message.

If these files are intended to be master calibration frames, then the people operating this remote observatory are making a serious mistake. The master frames should *always* be 32-bit floating point images, or you'll be losing data.

Furthermore, the numeric range to which the master frames are referred must be specified without ambiguity, since the FITS standard does not provide any way to specify the numeric range of floating point images (this topic has already been discussed on this forum many times).

For example, a floating point image generated with PixInsight is *always* in the [0,1] range, where 0=black and 1=white. A similar assessment must be provided by the software they use to generate the master frames. Otherwise you are being forced to use the same software. Alternatively (and ideally), they should provide the individual raw calibration frames as 16-bit unsigned images, so you can generate your own masters.

Finally, please note that this file has not been generated by Maxim DL. PixInsight is 100% compatible with integer FITS images created by Maxim DL.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline bitli

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 513
Thanks for the detailed explanation. I did use the hexadecimal edit trick and it seems to work.

I already had the problem a couple of years ago with another remote observatory. My experience is that they are not willing to change the format of their file - at that time I did not have the trick of the hex editor, so I was not able to properly use the calibration files received. 

I understand that it is difficult and time consuming to change the library and I prefer that you spend that time to write the doc  ;) ... However if it would be possible to log a message even after reading the file, may be based on the values of the FITS keywords, like 'Inconsistent BZERO/BASCAL/BITPIX value, file may not be properly read', this could save user and support time in the future.

rgds -- bitli

Offline dayers

  • PixInsight Addict
  • ***
  • Posts: 201
    • The Orlop
Quote
If these files are intended to be master calibration frames, then the people operating this remote observatory are making a serious mistake. The master frames should *always* be 32-bit floating point images, or you'll be losing data.

I have found that it's hard to interest those running remote observatories in changing what they provide. Sometimes it's easier to just use their bias and dark master frames in another program (like CCDStack) for file calibration and load the calibrated files into PI for further processing. 

Juan, what is the minimum I should ask for in the way of flat files from the remote observatory so that I can better control the calibration process in PI?

Dave
Dave Ayers
  Stellarvue 80 mm refractor on CG-5 mount, Orion 50mm guide scope. Imaging camera SBIG STF-8300M, guide camera ASI120mm. PHD Guiding. Sequence Generator Pro, PixInsight.

Offline bitli

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 513
Hello,
Indeed it seems that the original program used to make the problematic master was CCDstack.  The are in a parallel universe, no worm hole was yet found to make them communicate with the PI universe  ;)
-- bitli

Offline dayers

  • PixInsight Addict
  • ***
  • Posts: 201
    • The Orlop
Bitli,

Master calibration files created in CCDStack can be used to produce calibrated lights that can continue on in PI without any problems that I am aware of. I think I am also hearing that if the flat files used to produce the remote observatorie's masters are still available, the masters themselves can be created in PI. Either route seems sufficiently seamless to me to produce a valid workflow starting with a remote observatory's files. CCDStack does not seem to me a barrier between the two universes. In fact, I find CCDStack more versatile for inspecting a stack of files in some cases than for loading them into PI.

We are fortunate to have such a great set of tools to use.

Dave
Dave Ayers
  Stellarvue 80 mm refractor on CG-5 mount, Orion 50mm guide scope. Imaging camera SBIG STF-8300M, guide camera ASI120mm. PHD Guiding. Sequence Generator Pro, PixInsight.