Something weird about STF8300M images?

[brew]

I have started looking at some test images taken with my SBig STF8300M camera. The images are acquired with Maxim DL6, where they look correct. The brighter stars, both within the cluster and individual stars, have intensities up to 65535. The overexposed stars clip at the 65535 level, as I expect. I believe the image is saved as 16 bit integer; when I manually save an image that is the default setting on the dialog box. The fits header shows BITPIX of 16.

When these images are opened in PI, I get a strange clipping of the stars as seen in the attached screen grab of a 30 minute exposure. The center of the cluster is clipped at 32767. The brighter stars are also clipped at this level. Note that the regions around the stars are at levels above 32767, perhaps 40000 or 50000. This results in "holes" in the stars (and the cluster).

This clipping is seen at shorter exposures as well, down to 1 minute in my tests. At that level the clipped regions are smaller.

Also, it seems like the background levels are very high as well, around 35000. I have reset the Probe to read 16 bit integers so I can compare the values to the ADU levels shown in Maxim. This results in the background being very gray.

Am I missing some configuration setting for opening these files? I have used an SBig ST2000XM camera for years without problems; it has a maximum value of 32767, so I have not encountered this situation. I seem to be missing something about the 8300 files.

Another thread http://pixinsight.com/forum/index.php?topic=1669.30 reply#41 shows changing a configuration parameter in the PixInsight.ini file. However, when I look at the file I do not see the settings described. Perhaps that topic is obsolete?

I am running reasonably current 1.08.03.1123 PixInsight.

Thanks,
Robert Brewington

[oldwexi]

Hi Robert!
Did you ensure to save the image as UNSIGNED 16-Bit integer Fitfiles?
Usually Maxim saves as SIGNED 16-Bit Integer fit files what gets the lowest value to 32767 or
in PixInsight normalized to Pixel values larger than 0.5 ! This clippes raw data down to 15 Bit.
So, save the RAW-images with UNSIGNED 16-Bit Integer.
Save processed images as Float 32/64 Bit FITS file.

Gerald

[brew]

Well, I don't know EXACTLY how Maxim saves the files.

If I manually save a file, it seems to default to 16 bit integer. There is no option regarding signed/unsigned.

I don't find a setting in Maxim that indicates what format files will be saved in.

Under Settings/Files there is a checkbox that indicates Non-standard Fits Image Modes should Save Data in unsigned format. It also has a checkbox that says "When opening 16 bit images with negative numbers, treat data as unsigned". Both checkboxes are checked; however, this sounds to me like it is discussing opening files from other applications rather than what Maxim is using to save files.

Oldwexi indicates "Usually Maxim saves as SIGNED 16-Bit Integer fit files". How can I verify this, or change the default setting?
 
The FITS header (BITPIX) in Maxim says "16/8 unsigned int, 16 & 32 int, -32 & -64 real". I don't know what that means, it seems to be listing all the possible values? Maxim appears to be using unsigned integers, since it shows values up to 65535.

In PI, the File Explorer shows 16 bits, floating point = false. The median is 0.56, which seems pretty high. The FITS Keyword BITPIX says 16, no indication of signed or unsigned.

The PI View explorer says the sample format is 16-bit integer, no indication of signed/unsigned.

I was thinking I was running into a signed integer problem as well, but then how would I get values above 32767? There are a lot of bright areas with values of 40000-50000. It seems like everything would be clipped at 32767, nothing above.

Looking at the values some more:
At locations where Maxim reports 65535, PI reports 32767 (0.50 scaled).
At position 411,712 Maxim reports 36456 , PI reports 3688 (0.0563) and it is a dark pixel.
At position 412,712 Maxim reports 41976 , PI reports 64703 (0.9873) and it is a white pixel.
At position 428,717: Maxim reports 26592, PI reports a value of 59368 (0.9370). 
I am not seeing the scaling pattern.

[jkmorse]

Robert,

Welcome to the world of FITS incompatibilities.  I know exactly what you are experiencing since I have run into that problem many times myself.  After some fairly exhaustive research, here is a post setting out the fixes that should work under any circumstances:

http://pixinsight.com/forum/index.php?topic=8021.msg52728#msg52728

Love to hear if that fixes things and which step did the trick (for me its the input hints that usually do it, either singly or, in some tough cases, together).

Best,

Jim 

[jkmorse]

Robert,

As a follow up, most of my problems occur with MaximDL as well, but typically with my Apogee F16M.  For whatever reason my STF8300M comes out fine most of the time.  I know others have seen this with other software packages, but I am going to start using the SkyX Camera Add-on to see if that fixes things for me. 

Best,

Jim

[brew]

Well, I tried saving the file as floating point in Maxim; the file does then open correctly in PI.

I tried checking the two boxes in Format Explorer; I still get clipped values.

However,

a) can I set Maxim to save files in floating format? I see options for Maxim to open files, but nothing for how Maxim saves its files. I would like to avoid the extra work of converting all the Maxim files.

b) It looks to me like Maxim saves the files in 16 bit format. I don't see any indication as to whether Maxim uses signed or unsigned. The display of 65535 as an intensity leads me to think it is using unsigned integers. It also seems like PI is expecting unsigned 16 bit. I certainly see an issue if Maxim is writing signed integers, but it doesn't seem that way. How can I verify whether the physical values in the file are signed or unsigned?

c) It is established that saving in floating point is not the best technique, right?

d) The files are fine in Maxim; it is not like Maxim is messing up files. I just need something like the input hints to apply when opening the files in PI. Perhaps some kind of configuration parameter, or a settings file?

I should note that at this point I am not yet Integrating; I am simply testing/looking at some files. I suppose if the Integration works OK with format hints I will be OK overall. However, it seems like PI should have the capability of opening files created in Maxim for an 8300. This is not a particularly rare situation:)

I should note that I don't really have the option to use other image collection software since I use ACP.

[jkmorse]

Run the calibration using the input hints and that should take care of it.  If you read the posts from Juan I attached, it is more than PI "fixing" something on their side but a general weirdness about FITS generally and it only happens to an unlucky few.  One other solution mentioned is to save the images in MaximDL as TIFFs but then you lose the FITS header information.

Best,

Jim

[brew]

OK, this is still driving me nuts.

First, I should mention that I do not do image calibration in PI. My environment uses ACP/Maxim to collect images; Maxim does the calibration automatically. It seems to do a good job, and I don't have to save the darks/flats/bias frames for each image run. It is often several months before I get around to processing, during which time I might update the various masters several times.

I generally start processing with SubframeSelector, followed by StarAlignment. I tried running StarAlignment and specifying the input hints, but it doesn't work. PI sees flat gray images with all pixels = 0.5. So, no stars found, alignment immediately fails. So, the input hints does not work at the StarAlignment stage.

I cannot use SubframeSelector or Blink, since the files do not read correctly. I cannot run any tools on the files, such as PixelMath, Rescale, etc.

I looked at some files collected last night, and see another oddness. Maxim saves the "RAW" image before it does the calibration. It then calibrates and saves a second file. I used another program FITSview downloaded from the web to also try and look at the files. I wanted to attach 2 files, RAW-Iris (before any calibration is done) and Calib-Iris (calibrated file), but they are too big:(

An example image RAW_iris: this is the uncalibrated image saved by Maxim. It appears to be 16 bit integer.
  - The FITS BITPIX says 16 in Maxim and PI (FitsHeader window).
  - PI says sample format is 16 bit integer in View Explorer.
  - FITSView indicates the pixels are in the range -32721 to 32743, so it sounds like the values are 16 bit signed integers although Maxims's Settings/Files has "Save data in unsigned format" checked.
  - BSCALE = 1, BZERO = 0.
  - CBLACK=2826, CWHITE = 6028   Don't know what this is, maybe what Maxim uses to screen stretch?
  - Pedestal = 0    ? I thought this was generally 100


The corresponding calibrated file Calib-Iris is different. It appears to be saved as unsigned 32 bit.
  - Maxim says BITPIX is 32. PI shows BITPIX = 32 in FITS header as well.
  - PI says sample format is 32 bit integer in View Explorer.
  - FITSView indicates the file has values from 0 to 74052. So, this one is unsigned?
  -  BSCALE = 1, BZERO = 0
  - CBLACK = 1185, CWHITE = 3928
  - Pedestal is -100

It appears Maxim decides how to save the file on the fly, and may save in different formats based on the image. I do not see anywhere that ACP tells Maxim what format to save in, nor does Maxim seem to have such a setting. I cannot configure Maxim to always save in the (strictly incorrect) floating format.

I believe Maxim is certainly a common tool in the field; it is not a little corner case program. Additionally, the 8300 is a common camera. PI should be able to open files created in Maxim! I understand that everyone does not seem to agree on the FITS definition, etc, but there should be some mechanism for doing this.

It looks like my only workaround if I want to use PI is to manually convert hundreds of existing images collected in Maxim to floating point format, and to continue doing this conversion after every observing run.

[pfile]

not to be unhelpful, but it's a really bad idea to let maxim touch your files at all. you want it to just save the images at 16-bit integer... and do not calibrate anything with maxim. the less maxim does, the better... just configure ACP to do autoflats and let them pile up. as for bias and darks, you can probably get away with making a set every 3-6 months. personally i've gone a year between collecting darks and bias with my STT-8300M and never had a problem.

i am using maxim with ACP and an FLI camera and somehow we have got it saving everything as 16 bit integers with no further processing. i don't think ACP tells maxim this, but i can check. i think it's just static config in maxim. the rig is running right now so i can't poke around in maxim to find out for you.

if you can figure out a good way to convert your maxim files, you can always use ImageContainer to do a whole bunch at once. but to keep from pulling out your hair, i'd stop letting maxim calibrate images and do it all in PI. the results are superior anyway...

rob

[jkmorse]

Robert & Rob,

I wholeheartedly agree with everything in Rob's note.  Only caveat I have is that on rare occasions, I and several others have had MaximDL images that are fine in all respects except that in PI all saturated pixels read zero and so are black.  You can do a PixelMath work-around, but I have found that simply doing a batch convert in MaximDL to 32bit floating point solves the problem entirely.

But as to calibration or any other fixes other than the one above, I avoid MaximDL like the plague once it has captured the image.  I have just had too many things go wrong.  Robert, you really need to try out the PI calibration routines.  You can automate them with BPP, but I do it all manually and love the control it offers over the canned solutions like the MaximDL routine.

Best,

Jim

[brew]

So, at this point it appears that there are two conclusions.

1) PI cannot read the STF8300M 16-bit image files created by Maxim.

It does read files produced by other cameras in Maxim, ostensibly with the same format. I have been using an ST2000XM for years without problem; pfile has no problem with his FLI or STT-8300M, I have used various other cameras without issue. I will have to manually convert the STF-8300M files to float format for handling in PI.

2) Maxim changes the output format based on the image.

For example, PI can read the dark, bias, and flat frames from the STF-8300 without conversion, although they appear to be in the same 16-bit format as the image sub. When Maxim creates Master files they are in float format, not in the 16-bit format. The 16-bit images produced by the ST2000 are different from the STF-8300 somehow, perfectly readable in PI. Apparently the STT-8300 is also different.

3) There is clear sentiment that PI does a better job calibrating than Maxim.

I have not seen any studies on which program does a better job calibrating. I looked around on the net and didn't find anything, although hardly an exhaustive search. I have previously studied Star alignment results and found that PI does a better alignment than Maxim or RegiStar.

Clearly you guys feel that PI does a better job. This may well be true, although I can imagine that the software originating the image might have subtle tricks about calibrating it.

So, I did a bit of a test myself over the last 3 days, which I summarize here https://brewobservatory.wordpress.com/2015/06/20/pixinsight-versus-maxim-subframe-calibration-comparison/. Basically,

- The Maxim calibrated image clearly appeared to have less noise visually.

- I have to be very careful with the calibration subs to get it to work in PI; many of the calibrations came out very poorly.

    = The PI flat masters in particular seemed to be sensitive to failure. They often came out looking like bias frames.

    = Interestingly, using float conversions of the calibration subs worked poorly. I had to use the original Maxim 16-bit subs for PI to get the calibration right. Makes me wonder if I will encounter problems with the float conversions of the light subs.

    = Using Maxim's masters did not work. PI has to build its own masters.

    = Creating a Superbias made things worse.
 
- I don't know how to measure noise very well; there are a number of tools in PI that report different things. However, these tools seemed to be contradictory in their results. Sometimes PI came out a bit better, sometimes Maxim came out better. I don't know if the measured values are significantly different.

So, for now it appears that my optimal solution is to continue to calibrate in Maxim and then convert the light subs to float for processing in PI.

[pfile]

just some more thoughts - i drive the STT-8300M with equinox image (macosX), so it could still be that there is some issue with SBIG and Maxim playing nice together. a couple of years ago i watched from the sidelines as (apparently) a new developer took over the windows drivers (and perhaps the FPGA programs) for the SBIG cameras. there seemed to be a whole lot of thrashing going on and since what i had mostly worked (occasionally i'll get a frame with a white bar in the top 75 pixels or so) i did not upgrade any firmware. the drivers for the mac did not seem to change, so there was nothing to do there either.

since cyanogen bought SBIG i have not seen any more traffic on the sbig yahoo group from that developer, but on the other hand doug george seems to want to keep all his support "secret" and so this guy may have disappeared behind cyanogen's registration wall.

this is a very long-winded way of saying that perhaps there are new drivers or FPGA programs for the STF that may fix this problem. it really should be possible to get maxim to save the data from that CCD in unsigned, 16 bit integer format. that's about the only "standard" FITS format that actually exists

as for masters, yes, i don't think i've ever come across a data reduction package that was able to understand the masters from another piece of software. this is generally because, as you note, the masters are integrations and are always stored in some kind of floating point format. the FITS standard does not specify any particular legal range for floating point numbers and so the authors of each program have done what they deem sane. in pixinsight, floats are always stored in the range [0.0 - 1.0]. maxim probably does something different, which is where all the problems arise. the masters and the calibrated frames from maxim are using some other range and they must be normalized properly when coming into PI.

as for noise, i know juan has written a thread here explaining how to use the script called NoiseEvaluation. i also think PI's dark scaling algorithm is somewhat unique; darks are scaled until the noise in the final product is minimized. a lot of programs simply look at the duration of the dark and the duration of the light and 'blindly' scale based on the ratio. PI does not actually look at the duration or the temperature of the dark or the light when calibrating.

can you post one of these 16-bit images that maxim has downloaded from the STF8300M? it might be interesting to compare it to what i see from the FLI.

i suspect that all the problems in the latter part of your message are due to PI 'misinterpreting' the floating point data it's seeing. in the end everything is going to go way, way more smoothly if you can coerce Maxim to produce proper unsigned 16-bit FITS files.

rob

[brew]

I have attached a crop of the test file I used. The entire file is too big to be an attachment, which is why I hadn't sent one before. The crop shows the same problem being opened in PI, but I don't know if it is exactly the same format as the original. It was saved as "16-bit" in Maxim; there are no indications of signed versus unsigned.

The noise thing is kind of odd. I recently attended a PI workshop by Vicent Peris. Strangely (to me) none of the participants used any noise tools on their images; they just eyeballed two images and could tell which one had more noise. I was unable to accurately see the noise (as well as the results of some of the PI operations!) so I kept wanting a tool to measure the noise. Vicent showed me how to LinearFit two files before comparing their NoiseEvaluation results. Watching them, I can now do the "eyeball test" - I think . I will look for Juan's discussion.

I will check again for the drivers; I had updated them about a month ago, but who knows. I agree that Doug George's support model is a head scratcher; the pricing clearly encourages people to not maintain support.

in the end everything is going to go way, way more smoothly if you can coerce Maxim to produce proper unsigned 16-bit FITS files.

Exactly - this is what I am searching for.

[pfile]

you can sign up for a free google drive account and upload the files there, then share them globally and post the link here. i am curious to see the entire fits header for the files…

or i suppose you can also ask juan for an account on endor (the PI fileserver) and put them in the forum shared area, if you don't want to make a google account…

rob

[pfile]

oh i guess the fits header is still on the cropped image you uploaded.

rob