Integrating Bias frames from Prism

[KevinG]

I am capturing my images in the package Hyperion Prism and processing in PI.
(I have been using PI for quite some time before this  with SGP...)

I recently encountered a problem where my bias frames appeared mid-grey with no STF applied (readouts around 0.5). I have tracked this down (after much reading about FITS and scaling etc...) to the following..

Prism captures in 32-bit floating point but the numbers stored are simply floating point renditions of integers; so the range for a 16-bit camera is
[0 65535].

My understanding is that reading in FITS to PI scales to [0 1] using v -> (v - v_min)/(v_max - v_min).

However for images like bias, where the range is really constrained to be the low hundreds, this wont work. Thus, if the maximum is at 500, say, then a bias pixel of 400 will be rendered at 0.8.

For many images this rescaling 'kind of' works because bright stars or hot pixels anchor the top of the range. But bias doesn't look like this...

I therefore used the format editor to read in using the range [0 65535]. Great - the images now look 'normal' (black without STF, and with sensible readouts...).

So I proceeded to integrate to make a master bias but PI insists on rescaling back to mid grey... Then I discovered format hints but they dont make any difference (in fact, I wonder if they are being used at at all as I can type garbage in there and its doesnt complain...)

Any ideas please?

[RickS]

You could try the SampleFormatConversion process to convert the floating point image back to 16-bit uint and see if you get correct values.  The ideal solution would be to convince Prism to preserve the correct format (16-bit unsigned integer) of the raw data from the camera.

Cheers,
Rick.

[Juan Conejero]

My understanding is that reading in FITS to PI scales to [0 1] using v -> (v - v_min)/(v_max - v_min).

PixInsight rescales floating point pixel data this way only when it has no way to know the black and white points of the image. Since these frames store floating point data, and the FITS format cannot provide any information about the actual range to which pixel values are referred, the above rescaling operation will be applied. In other words, floating point pixel data stored in FITS files may use any arbitrary range and there is no way to know it. That's just how FITS is, and one of the many reasons why FITS has been deprecated in PixInsight.

I therefore used the format editor to read in using the range [0 65535]. Great - the images now look 'normal' (black without STF, and with sensible readouts...).

So I proceeded to integrate to make a master bias but PI insists on rescaling back to mid grey...

This should not happen (modulo bugs, but then this problem would have been reported long time ago). If you define a floating point input range in FITS Format Preferences, then that range will be applied throughout the entire PixInsight platform to load FITS files by default.

However, changing the default input range is not necessary, and is in fact a bad idea in general. As you have discovered, the correct way to do this is through format hints.

Then I discovered format hints but they dont make any difference (in fact, I wonder if they are being used at at all as I can type garbage in there and its doesnt complain...)

This is the expected behavior. Unrecognized format hints are always ignored, never taken as 'invalid parameters'. In this case you should use the following input hints in ImageIntegration:

lower-range 0 upper-range 65535

This should allow you to integrate these frames correctly, provided that all of them have been represented in the [0,65535] range.

As Rick has said, the actual problem here is that the raw frames have been stored as floating point images. Any image acquisition application should write raw frames as 8-bit or 16-bit unsigned integer images, depending on the device's bit depth of digitization. There is nothing between two discrete values read from a digital device—or otherwise, the data is not raw. Besides wasting storage space and bandwidth, storing raw data in floating point format only serves to generate interoperability issues.

[Niall Saunders]

Hi all,

I am astonished that here we are, decades after the FITS format was introduced, and yet software vebdors have no real idea about how to encode data in that format. After all, it is not really likely that they are acquiring floating-point data from the attached imager, so why enforce the conversion?

Even the very best cameras might only be producing 32-bit integer data (although, in reality, most only produce up to 16 bits). If this was software that I had purchased, then I would definitely be looking for an integer-based resolution and, if I didn;t get offered that (free of charge) then I would be pressing the -Not fit for purpose, I wan't my money back- button. After all, this software is not cheap - far from it.

[KevinG]

Thanks to all for chipping in...

In defence of Prism, I have to say that floating point is not obligatory, but the integer option is 16-bit signed with no BZERO keyword set, to allow true 16 bit readout (so it looks to me like 15-bit unsigned). I am rasing this in the Prism forum, and maybe I have misunderstod... In any case, there are a load of reaons why I migrated from SGP, although here is clearly not the place to discuss 

Kevin

[Niall Saunders]

Hi Kevin,

I can't remember the details now, but I do seem to remember something similar with Meade's "Envisage" software when it came to saving their DSI image files in FITS format.

If you can save images in 16-bit format, post the FITS header from any saved image and let us see what you are up against. (You could also save the header data from one of your 32-bit images, for comparison).

If you can, try and get a third-party program to open your FITS file - even NotePad or NotePad++.

[KevinG]

OK so for the record, I tried all four combinations of:

Norm to [0 1], or [0 65535] on reading the files
Use or don't use format hints.

(an yes, I disabled use of file cache to force rereading of files ...

They all result in a mid-grey (K ~ 0.5) bias integrated image.
I'll try the sample format process thing...

[KevinG]

Hi Niall

see below for FITS headers... I think they make sense.. ?

Integer header (relevant parts)

01 : SIMPLE  =                    T /FITS header                                     
02 : BITPIX  =                   16 /No.Bits per pixel                               
03 : NAXIS   =                    2 /No.dimensions                                   
04 : NAXIS1  =                 2749 /Length X axis                                   
05 : NAXIS2  =                 2199 /Length Y axis                                   

07 : SWCREATE= 'PRISM,  Version  10.3.49.416,  27/08/2018' /Software Name and version
                           
31 : COMPRESS=                    0 /Number of compression                           
                                                 
37 : DATAMAX =                  411 /Maximum data value                             
38 : DATAMIN =                  159 /Minimum data value                             
       
42 : BITCAMPX=                   15 /Native camera dynamic in bits per pixel         
         
53 : HISTORY none                                                                   
54 : END                                                                             


--------------

Floating point
01 : SIMPLE  =                    T /FITS header                                     
02 : BITPIX  =                  -32 /No.Bits per pixel                               
03 : NAXIS   =                    2 /No.dimensions                                   
04 : NAXIS1  =                 2749 /Length X axis                                   
05 : NAXIS2  =                 2199 /Length Y axis                                   

37 : DATAMAX =               463.00 /Maximum data value                             
38 : DATAMIN =               269.00 /Minimum data value

42 : BITCAMPX=                   16 /Native camera dynamic in bits per pixel         
       
53 : HISTORY none                                                                   
54 : END                                                                             


many thanks

Kevin

[KevinG]

I think I have a method that works (touch wood..)

(i) Set the floating point upper range to 65535 in the FITS format dialogue (in format viewer)
(ii) Import the files into the batchformatconverstion script
(iii) Save all as .xisf in floating point

Thanks to Niall (I think it was Niall!) for pointing out this script..

Kevin

[Niall Saunders]

Hi Kevin,

Well, I believe (but would be happy to stand corrected) that your INTEGER-based FITS-format image, if it is a Bias image, may well be quite close to the truth.

The header reports that the camera source was a 16-bit source, and that the data in the frame had a maximum value of 411ADU, and a minimum value of 159 ADU (giving an estimated mean value of around 285ADU, assuming an even and balanced distribution  - which is probably not the case).

Now, you can assume that, for all intents and purposes here, a Bias frame really represents the 'Black Point' for your data. In other words, you can't really get a 'darker' or 'blacker' image. What you now need is a super-saturated image - one where you really can't get it much 'brighter' or 'whiter' - and this would let you get the 'White Point' for your camera. (** see later)

Given that you already have a Float value for your Dark Point (the lower of the two values in the Float32 image, you just need to get the White Point in the same way as you did for the Int16 image.

Remember, in both cases the full range is 65536 separate values - from 00000 to 65525 for your Int16 data, and some as yet undefined range of the same 65536 'buckets' for your Float32 data. And, therein lies the problem.

You could now estimate the 'size of the buckets' for the Float32 data, but (as far as I can see) this would be a 'guess-timate' at best, and totally erroneous at worst. For example - let us say that you assume (we are scientists, we shouldn't be making 'assumptions'  ) that both Int16 and Float32 share the same ZERO point (there is absolutely no reason why they would have to, and no evidence whatsoever that they do).

This would suggest that the bucket-size for the Int16 data is  1 (159 ADU counts / number of buckets) - which is nearly always the case for Int data. However, approaching the Float32 data the same way gives us ~169.2 Float32 buckets per Int16 bucket (269 / 159). This would mean that you should get a value of ~110,875.4 for your Float32 White Point.

However - another 'assumption' has been made (did you spot it?), and that is that the saturated White Point (the highest ADU count that a camera can give - the full-well ADU) is, in fact, going to be 65535 for an Int16 camera. Unfortunately, it isn't! It might, for example be 45678ADU (picking a totally random number), the ADU count that you might always get once you shine a bright enough light source at the imager for long enough. So, you can get a reading of 45678ADU, but you would never be able to get up to 65535ADU.

Does any of this make sense? Have I made any glaring omissions or mistakes?. I hope it's helping someone - when I had to get my head round the wierd method used by Meede, I was 'on my own' (save for the Mike Weasner Mighty ETX site and help from others setting off down the path of astroimaging - so I feel your pain!!!)

[Niall Saunders]

Thanks to Niall (I think it was Niall!) for pointing out this script

Sorry Kevin - it wasn't me who pointed the script out, nor can I make any claim on its contents or functionality

[KevinG]

First - you a right, thanks should go to Nick for noting the format conversion script..

Anyway, I think things are relatively straightforward. I can view the images in Prism and can confirm that a fully saturated image does indeed produce 65535.
I think it is simply rendering a 16-bit (unsigned) count as a float so integer N is just N.0
Anyway, I am going to reprocess soeme data based on these assumptions and hopefully some weirdenss I had encountered there will now go away

Cheers

Kevin

[Niall Saunders]

I can view the images in Prism and can confirm that a fully saturated image does indeed produce 65535

Hi Kevin,

As I tried to describe in my earlier post, that is exactly what should not be happening. A 'fully saturated' ADU value, in a RAW, unprocessed, image is indicative of some level of processing having taken place either in the camera, or in the image acquisition software. And that means that you are NOT dealing with a RAW image any more - which makes image calibration and subsequent post-processing all the more difficult.

An astro-image acquisition package should be written with that precept in mind, and should not force the user to perform any subsequent steps within the package itself. Users should always have the choice - and the biggest choice is to decide whether to even use the software in the first place.

So, again, an ADU value of 65535ADU for a 16-bit RAW image, means that the image is not RAW, and so subsequent calibration steps (as astroimagers know and understand them) cannot now be applied

[KevinG]

Hi Niall

I should have made it clear I determined the saturation value in an experiment where I deleiberately exposed to obtain this result. (I think I looked at a specroscopy calibration frame where the strong lines had been overexposed...)

Kevin