Saved Image Format

[sreilly]

I usually do all my calibration in MaxIm as it does an intelligent job of it and I already own and use it for data acquisition. What I'm unclear of is when I save the calibrated images in MaxIm the ADU pixel values are high, being usually well over 100,000 ADU on a combined Summed or Median combined image. Even the individual images are at least saturated above 45,000 and all are saved using IEEE Float fits format. PI has no issues reading the images and working with them so all is good up till here. No if for some reason I align and combine the images in PI and open those images in MaxIm my pixel values are like 1. The data appears to be there but really strange things happen if I try to say compile an RGB image from the PI images. The PI images are also saved in 32-Bit IEEE 754 Floating Poir (point I think). I'm missing a setting I think but can't figure this out to save me. I've always seen this but just getting around to asking what needs to be done to correct my problem. Any image that I save as 16 bit unsigned is fine but I'm not wanting to use this format till I'm ready to make the web version of the image for posting. Any ideas?

Thanks,

[Philip de Louraille]

Isn't PI saving data between 0 and 1 (with the precision depending on saved bits)? For instance if I want to use PI images in Nebulosity, I open them and then multiply the image by 32768 so that Nebulosity can display it correctly. Not sure if this is what you are talking about.

[pfile]

i went thru this a while back when trying to bring maxim-calibrated/stacked masters into PI. the short answer is that maxim is using f32 format but storing the values as i16, that is [0.0-65535.0].

so it sounds like you need to multiply the values in the PI f32 file by 65535 to make them display right in maxim...

[sreilly]

This leads me to believe that there are no standards for file formats. Having to do anything to a saved image that is saved in 32 bit seems really off to me and those images should display fine in any program that is written for 32 bit files. I'm very unclear what is being done to read these properly in MaxIm. Saved 16 bit unsigned works fine but at this point I'm not willing to reduce my image depth as I still have processes to do to the final image. For some it may seem strange to want to display the data in another program but for me it serves as a check that allows me to do things as check the color balance compared to a color image using ratios such as 1:1.605:2.757. To do this in PI I need to divide everything by my largest ratio number and get something like .36:.58:1 When I compare the two images, MaxIm's color and PI's color using the given example, they are different. The ratio given is derived for using eXcalibrator http://bf-astro.com/eXcalibrator/excalibrator.htm for color ratios. I've found it to be very helpful especially in galaxy images.

[Emanuele]

This is interesting. So what do I have to do in order to use Master Calibrations files created in Maxim (Flat, Bias and Darks) to calibrate images in PI?

[pfile]

apparently the FITS standard is not defined well enough to specify what the min and max values for a pixel should be when the pixel format is f32. that leaves just the defacto standards that we have. pixinsight uses [0.0-1.0] in f32.

i think when you open an f32 file from maxim, pixinsight by default is going to ask you how to scale the file, because by default pixinsight gives 0.0 for the lower range value of a fits file and 1.0 for the upper range value. if you double-click the FITS tab in the format explorer, you get a dialog box which lets you set up what the low pixel value is and what the high pixel value is. if you set them to 0 and 65535 when importing a maxim f32 FITS you should get the right result in pixinsight.

i don't mean to be a smartass but you should be able to do your calibration, integration and color balancing all within pixinsight, and the results should be superior to maxim.

[Philip de Louraille]

In a 16 bit signed file, multiply by 32767
In a 16-bit unsigned file, multiply by 65535
In a 32-bit signed, multiply by 2147483647
In a 32-bit unsigned, 4294967295.
I think...

[sreilly]

I haven't spent much time with it but from what I've seen, calibration is pretty clumsy in PI. In MaxIm I can load all my calibration files in groups of darks, flats and bias frames, create the masters and then MaxIm is smart enough to read the fits header and use the appropriate calibration frames to calibrate the image. I have -15 through -30 darks loaded as well as flats and bias frames loaded, both 1x1 and 2x2 binned. It makes calibration way too easy. I may be wrong but I don't understand how one master dark made from the same frames would be different that in PI. Point is MaxIm is fast, effective and seamless. I already own the expensive software and use it for getting the data so it just makes sense to me to use it for what it's good at.

[pfile]

i think improvements to the calibration flow are coming. i agree that it's a bit cumbersome. however, it works fine for me.

pixinsight evaluates the noise in your lights and scales the master dark frame accordingly. this is why the temperature/duration of the dark is not nearly as important as it is in other programs. so there's no need to have an extensive dark library or match up darks of different duration/temperature with your lights. that kind of eliminates the need for a whole binning infrastructure.

all i know about maxim is that it did very strange things to the master flat that was created. the illumination profile was wildly exaggerated. but i had no control over the process - the calibration masters were provided to me by lightbuckets.

[Emanuele]

ok, thank you for this discussion.
I just tried calibrating a FIT and defining the lower limit and upper limit as 0 and 65535, but the results is even worse than before. The picture is all black. So that's not the right method.

Any other ideas. This shouldn't be happening!

[sreilly]

ok, thank you for this discussion.
I just tried calibrating a FIT and defining the lower limit and upper limit as 0 and 65535, but the results is even worse than before. The picture is all black. So that's not the right method.

Any other ideas. This shouldn't be happening!

Are you saying that the calibration frames work well in MaxIm and not in PI?

Steve

[sreilly]

pixinsight evaluates the noise in your lights and scales the master dark frame accordingly. this is why the temperature/duration of the dark is not nearly as important as it is in other programs. so there's no need to have an extensive dark library or match up darks of different duration/temperature with your lights. that kind of eliminates the need for a whole binning infrastructure.

There are several camps on calibration frames. One likes to take them each imaging session, another likes to use scaled darks along with their flats, and then there is the camp I'm in where I have plenty of hard drive space and cloudy/moon lit nights that allow me to keep a dark library with a range of temperatures and lengths (although I almost exclusively expose for 20 minutes with exception of 30 minutes for Ha) as well as bin 1 and 2. While this may not be the most economic of time approach, it's for the most part quite easy to setup as a sequence in MaxIm. I do automatic sky flats using ACP so that's painless. What is most important is that the frames used create a well calibrated image. A current matched master dark, made with sufficient sub-frames (usually 20-30 for me either Median or Sigma combined) , does an excellent job. Not sure how any program is going to change that. Flats will either work or not. Making proper flats within the linear range of the camera, usually 25-50% of saturation, is key as well as enough to increase the signal to noise ratio of the master frame. I have a tendency of taking darks every season, 3-4 months apart, and take flats at that same time. Masters are created and named including the date for use should I decide to reprocess an image sometime down the road. I am aware of scaled darks but have yet to have as good a reduced image as I get with non-scaled darks so I quit trying that.

I'm not saying it doesn't work, just not for me. The bottom line is getting the best results for you using whatever method works best for you. Again, I have owned MaxIm for many years and it does this well. I need a program, in this case MaxIm for using ACP, so I might as well use it's strengths. After calibration, it's almost always 100% PI until I make my JPG files for the website. Then I use Photoshop. I already own it and it gives me previews of the images and size before creating them.

Steve

[Philip de Louraille]

Emanuele, I am not sure I understand what you are trying to do.
"I just tried calibrating a FIT and defining the lower limit and upper limit as 0 and 65535, but the results is even worse than before. The picture is all black. So that's not the right method."

Are you trying to use a PI FIT file in Maxim or a Maxim FIT file in PI?

[Emanuele]

Pde: I am trying to calibrate, in PI, an image that has been acquired in Maxim.
The Master Dark, Master Bias, and Master Flats have also been created in Maxim.

[Philip de Louraille]

Well I can't be of any help here as I do not use Maxim. However, you should not have to modify (e.g.: scale) your Maxim FIT file in PI, nor should you (re)scale them in Maxim because you know your next step is in PI. PI will automatically rescale them (if it can't figure out automatically what do to, I believe it asks questions.)