Very short exposures from DSI-C camera don't load properly

[Nocturnal]

Hi,

this is an issue I've worked on with Luc (DSS) but I think you may be interested as well. Envisage FITS3P images in general load fine in PCL but when I create a bias frame (very short exposure) the image shows up all white instead of nearly black. The histogram is also all the way to the right. DSS had the same issue but Luc was able to work around some quirks in the format (32 bit float FITS with R, G and B panes in the same file).

I've uploaded a zip with 2 fits files:

http://www.tungstentech.com/files / pcl-fits.zip  (remove the spaces please, I don't want robots to download the file)

The names indicate the differences.

Thanks,

  Sander

[Juan Conejero]

Hi Sander,

I have seen the files and they are fully standard-compliant (if they weren't, PixInsight would reject them). There's nothing wrong in the FITS header. In fact, these files have been written with the CFITSIO library, which PixInsight's standard FITS format support module also uses.

So the problem is with the actual pixel values that have been stored in the files. There is nothing I can do to fix this. PixInsight is reading and showing just what is stored in each file, no more and no less.

Having said that, I'll tell you that the 0.001 s file seems to be inverted. If you invert it (Ctrl+I), it shows a more than reasonable histogram for such a short exposure. Just an idea

[Nocturnal]

Hi Juan,

when you parse a float FITS file, do you look at all values to determine max and min? In the case of these files the range is 0.0 to 64K.0. So integer values stored as floats. I'm not sure how that leads to these stats though:

Code: [Select]

           ____R____ ____G____ ____B____
Mean....... 0.9694947 0.9696215 0.9694630
Median..... 0.9695229 0.9696455 0.9694430
StdDev..... 0.0035113 0.0026615 0.0055351
AvgDev..... 0.0027975 0.0021157 0.0044030
Variance... 0.0000123 0.0000071 0.0000306
Minimum.... 0.9485166 0.9538309 0.9284922
MinPos..... (0,229) (89,0) (4,0)
Maximum.... 0.9883521 0.9843149 1.0000000
MaxPos..... (590,0) (0,123) (0,388)
Count...... 316224

The maximum value should be some float represented int smaller than 64K.

Luc suggested adding a 64K pixel in a corner to force the proper maximum of the file. Would that help? I set the datamax FITS field to 64K.0 but this did not change the behavior. Are you purposely ignoring that field?

Thanks,

  Sander

[Juan Conejero]

when you parse a float FITS file, do you look at all values to determine max and min?

We use the BZERO and BSCALE keywords. For floating point FITS files these keys are in fact mandatory, since they are the only way to know the actual data range in which stored pixel values are represented.

Please refer to this document:
http://archive.stsci.edu/fits/users_guide/node21.html

The standard says that when no BZERO or BSCALE keys are found, their values must be assumed to be 0 and 1, respectively.

Only if the BZERO and BSCALE values are bad, or if existing pixel values exceed the declared range, we take the existing extreme values in all channels of the image as a "last-chance range", which we map to the [0,1] range.

In your case, both files do provide standard BZERO=0 and BSCALE=1, so this is the range used by PixInsight to decode pixel values. If the actual values stored are outside the declared [0,1] range, then blame the guys who wrote the software that creates these files...

I set the datamax FITS field to 64K.0 but this did not change the behavior. Are you purposely ignoring that field?

We don't use the DATAMIN and DATAMAX keywords because they don't define the actual range of values. In plain words, DATAMIN does not represent black and DATAMAX does not represent white. These keywords are just informative values that can be used to know in advance which are the minimum and maximum values present in the file, but they don't add more information than what we can already know by reading all pixels.

BZERO and BSCALE are the only keywords that define how pixel values have to be mapped to the [0,1] range in a floating point FITS file.

You could overcome this problem by writing a PixInsight module to handle FITS file reading with low-level PCL routines, replacing the standard FITS support module. Although this is quite easy to do, in fact, have you considered writing the authors of the original software to see if they can fix this issue to write correct FITS files?

If you decide to write a specific module, I'll help you of course.

Luc suggested adding a 64K pixel in a corner to force the proper maximum of the file. Would that help?

Edited: I didn't understand this well. I see. You mean writing a 65535.0 pixel with the same software that creates the wrong image. Please try it; it indeed could solve the problem.

However I think you should contact the guys that wrote the original software.

[Juan Conejero]

Another idea. You could write BZERO=0.0 and BSCALE=65535.0 in the file's header. If the original software doesn't allow you to modify these keywords (it shouldn't, because they are reserved keywords), you can try to change them by editing the FITS file with an hexadecimal editor.

[Nocturnal]

Hi Juan,

thanks for the detailed response. I could try to reach Meade and do the right thing but it's not likely my request will be honored. I'm actually working on a page that lists all shortcomings I've found in their DSI related software I've found so far:

http://www.tungstentech.com/Articles/Astronomy/MeadeEnvisageandASIPproblems/tabid/76/Default.aspx

I'm still gathering details and this is helping a lot.

So, assuming Meade won't fix this problem it might be possible to write a utility that fixes up the BZERO and BSCALE values. I tried setting SCALE to 64K in PCL but it won't let me. It's a locked value apparently. I'll try a different utility.

Thanks,

  Sander

[Nocturnal]

Another idea. You could write BZERO=0.0 and BSCALE=65535.0 in the file's header. If the original software doesn't allow you to modify these keywords (it shouldn't, because they are reserved keywords), you can try to change them by editing the FITS file with an hexadecimal editor.

Yes, this is what I intended to do :-)

[Juan Conejero]

Wait a moment, that I have fixed your problem  :lol:

[Juan Conejero]

OK, here we go:



Now the image loads nicely. I have used KHexEdit on my linux box. Do you have access to a similar tool?

I'll upload the fixed FITS file in a moment.

[Juan Conejero]

Here is the fixed FITS:

http://pixinsight.com/tmp/1msDark1-verylight-fixed.fts.zip

Please verify that it's correct now.

[Nocturnal]

Hi Juan,

ok, I'm a bit confused :-)

The file works great, thanks. I was not able to do the same here though. I opened the file in vim and edited it to set the bscale value. I guess the format is critical as it did not 'take'. PCL showed the right BSCALE value but the image was still white.

I then saved the fits file to see what was different about it and found that bscale was set to 1 again! Luckily the file still loaded fine so it appears PCL normalized all the values to a range 0-1 instead of keeping them as they were. Does that sound right?

Now what I need to do is write a little fitsio based app that sets the bscale value correctly. Now I need to go back to talking with Luc to figure out why DSS is not honoring the bscale value. Knowing Luc he probably has a good reason :-)

Thanks,

   Sander

[Juan Conejero]

Sander,

Just overwrite the existing white spaces with "65535.". You must be careful not to expand or shrink the keyword entry, so don't delete or insert characters.

I have simply left it as BSCALE = 65535.

The spaces at both sides of the equal sign are mandatory. The dot is also mandatory since this is a real-valued keyword. Replace the old 1. with two white spaces. That's all; it should work.

Actually, you don't need to write a cfitsio-based program. It's much simpler: just open the FITS file in read/write binary mode (fopen( "foo.fts", "r+b" )), load it, look for the BSCALE word, replace the keyword value, then fwrite the file, and job done.

This can also be done as a JavaScript file in PixInsight; it's easy

[Nocturnal]

Hi Juan,

since the 1. was right justified all the way to the \ I put 65535 there too. You are right, doing a simple binary replacement in the file is much easier than dealing with fitsio.

Thanks,

  Sander

[Nocturnal]

Hi Juan,

have another look at the images please. Take a fresh copy from the zip. The pixel values are in the thousands, not between 1 and 0. Setting BSCALE to 64K will not help. I tried it and the file does not get loaded correctly in PCL. I loaded the images in fv, hoping to see the raw numbers in the file. Files from Envisage list pixel values in the 3000 range. The fixed FITS you made for me lists values in the millions. I don't understand that at all :-)

It seems PCL 'normalizes' FITS files to its own BSCALE and pixel value set. Is this true? If so, could that affect it?

Thanks,

   Sander