PI and MaxImDL ADU Values

[sreilly]

I spent the afternoon working on using calibration in PI. I made master bias, darks, and flat frames without issue. Where I hit a wall is de-blooming which I normally did in MaxImDl using Ron Wodaski's De-Bloomer plug-in after calibrating the images. Any image I work on and save using PI that is opened in MaxImDl has ADU values that MaxImDl doesn't seem to handle. See the attached screen shot. What I'm use to seeing is values for 16 bit being 65,535 and 32 bit being greater. I'm not sure where I'm going wrong but de-blooming in MaxImDl would not work on the PI generated images saved as 32 bit IEEE format. The problem stems from the fact that the de-bloomer plug-in is looking for a value of somewhere from 45,000-52,000 ADU.

Any suggestions?

[Nocturnal]

You could save the PI image as a 16 bit unsigned int FIT file. Maxim shouldn't have any issues with it. Of course you loose 32b resolution.

[sreilly]

Actually if my understanding is correct, the camera is saving the data as 16 bit initially and is only being saved as 32 bit after having any calibration if selected as such or you need to stretch the data to fit into the 16 bit format. If this is true, then I need to calibrate and save my images in PI as 32 bit IEEE Floating Point data. Then I can combine and save the masters as 32 bit IEEE Floating Point data and continue my processing. so I'd calibrate the raw data in PI, save as 32 bit IEEE Floating Point data, bring into MaxImDl and de-bloom (that isn't working), and save as 32 bit IEEE Floating Point data again. Back in PI I'd do my image registration, combine and so forth saving each step as 32 bit IEEE Floating Point data.  The attached screen shot shows the de-bloomer plug-in.

Am I missing anything? The camera actually has a 16 bit A/D converter so if I'm looking at this properly the raw data would have an ADU range of 0-65,535. I see those values increasing after calibrating using MaxIm and it appears that I would need to save as 32 bit IEEE or higher at that point so this may not work properly. So why will the de-bloomer plug-in work on these images at 32 bit depth in MaxIm and not after calibration in PI. The difference in ADU values is the difference and I have no idea how to have PI save so that MaxIm can do the de-blooming. Shortening the exposures are not an option as I'm trying to maximize SNR and more shorter exposures will increase the noise. For example 12 - 20 minute exposures would have a higher SNR than 24 - 10 minute exposures as the read noise is doubled. Depending on your sky conditions, light pollution, you may not be close to sky limited. According to Clear Dark Sky Pollution map I'm "0.11 to 0.33 on artificial/natural sky brightness, 21.89 to 21.69 Sky Brightness mags / sq arc-sec V Band, 3 Bortle Scale approx, described as Low light domes (10 to 15 degrees) on horizon. M33 easy with averted vision. M15 is naked eye. Milky way shows bulge into Ophiuchus. Limiting magnitude 6.6 to 7.0." So my understanding is that longer exposures are better. I guess I need to find a way to have PI save the ADU values as a numeric value higher than 65,535 so de-bloomer plug-in can work.

This is a lot of rambling on and I hope I'm making some sense.

[Juan Conejero]

Hi Steve,

so I'd calibrate the raw data in PI, save as 32 bit IEEE Floating Point data, bring into MaxImDl and de-bloom (that isn't working),

So why will the de-bloomer plug-in work on these images at 32 bit depth in MaxIm and not after calibration in PI.

I have no idea how to have PI save so that MaxIm can do the de-blooming.

The problem here is of data incompatibility between applications. Unfortunately, the other application (Maxim) cannot use an arbitrary input range of floating point data. PixInsight writes all floating point images in the range [0,1], where 0 corresponds to the 'no signal' state (usually black) and 1 corresponds to 'full signal' (usually white). So Maxim expects floating point pixel data in its native range (which I believe is [0.0,65535.0], although this is just a guess based on previous issues similar to the present one) and doesn't allow you to define a custom input range —or that's what I think; correct me if I'm wrong.

So I'm afraid that the only way you can export FITS files to Maxim is writing them in 16-bit unsigned integer format, as Sander has already pointed out.

... and save as 32 bit IEEE Floating Point data again. Back in PI I'd do my image registration, ...

Contrarily to the situation I have depicted above, PixInsight does allow you to define a custom input range for floating point FITS (also TIFF) files. Since Maxim uses its own floating point range, which I think —although I may be wrong— is [0.0,65535.0], Maxim floating point files cannot be imported in PixInsight unless you define a custom input range:

- Open the Format Explorer window.

- Double click the FITS item on the left side list

- On the FITS Format Preferences dialog, enter 65535 as the Upper Range of Default Floating Point Input Range.

Now you can read Maxim floating point FITS images without problems with the StarAlignment tool. Note that the output range for floating point images is always [0,1] in PixInsight, irrespective of a custom input range. So once you're done with StarAlignment, don't forget to reset the input range in FITS Format Preferences to its default [0,1] setting, or a subsequent tool (e.g. ImageIntegration) won't load your images properly.

Shortening the exposures are not an option as I'm trying to maximize SNR and more shorter exposures will increase the noise.

Yes, short exposures are the only option IMHO. Bloomings must be fixed through observational techniques. Any software deblooming tool works by guessing unavailable data, basically in the same way you would guess them using a clone stamp tool, but in an automatic way instead of manual. Although I agree that software solutions to bloomings can be useful, they are the worst way to deal with this problem. For this reason, we stopped some R+D lines that we had started in 2010 to create a deblooming tool in PixInsight; we decided to focus our (always scarce) resources on more important and priority topics, at least temporarily.

Think of bloomings as regions of your image where you don't have data. Then the optimal solution is to acquire the missing data and put them on place. Simple and clean. Take a look at these processing notes to see an example (jump to the Blooming Suppression section from the table of contents at the left):

http://pixinsight.com/examples/NGC6914-CAHA/en.html

In this case we were using a square CCD sensor. This has the advantage that we can just rotate the camera 90 degrees in the middle of the observing run. Since one half of the raw frames have bloomings oriented in the opposite direction, it it very easy to fix them with a simple mask and standard pixel rejection.

If your CCD is rectangular, then the option to square out your image is something that won't make you very happy, I guess (although it is an option). With a rectangular CCD you have to acquire a set of short exposures. Short enough to get rid of the bloomings, but as long as possible to increase SNR before the bloomings appear. Then the procedure is essentially the same as in the example above: build a mask to isolate the blooming areas and replace the bloomings selectively on your long exposures with data from the shorter exposures. The tools involved are ATrousWaveletTransform (mask generation), HistogramTransformation (clipping mask levels) and PixelMath (pixel replacement). Of course, this task must be carried out when the images are linear.

With these techniques you recover the bloomings with real data. Software deblooming tools have to invent the missing pixels by guessing them from neighbor areas, similar to a clone stamp. Yes, the replaced data have lower SNR, but this can be overcome well with noise reduction techniques and the same mask used for blooming replacement. A few small bloomings that may appear in the sort exposures for the brightest stars can normally be fixed manually with the CloneStamp tool because they are little and hence the manual guessing work isn't invasive. In any case, my personal opinion is that if a blooming cannot be removed without inventing nonexistent data to the point one feels uncomfortable, better leave the blooming as is.

Hope this helps.

[sreilly]

The problem here is of data incompatibility between applications. Unfortunately, the other application (Maxim) cannot use an arbitrary input range of floating point data. PixInsight writes all floating point images in the range [0,1], where 0 corresponds to the 'no signal' state (usually black) and 1 corresponds to 'full signal' (usually white). So Maxim expects floating point pixel data in its native range (which I believe is [0.0,65535.0], although this is just a guess based on previous issues similar to the present one) and doesn't allow you to define a custom input range —or that's what I think; correct me if I'm wrong.

So I'm afraid that the only way you can export FITS files to Maxim is writing them in 16-bit unsigned integer format, as Sander has already pointed out.

Hope this helps.

Juan,

If I understand this correctly, while both programs are saving the data as 32 IEEE Floating point, there are different values for each program? The following is from MaxIm's help file:

"Preserving Bit Depth
An important feature of MaxIm DL is that it stores images internally in 32-bit floating-point format. Although this increases memory consumption compared to standard desktop image editing programs, it gives much greater flexibility and helps preserve data precision.

If you were to sum or average together 256 individual 16-bit images, the actual bit depth of the result could be as high as 24 bits. In reality, you probably have only 12 to 14 bits of usable data in each image; nevertheless the final result would still have 20 to 22 bits. Obviously if you store this result in a 16-bit or 8-bit file format, you will need to rescale the image and clip and/or truncate part of the useful data. If you scale the image so that you maintain faint image detail and good background statistics, you will end up clipping off the brighter objects. Therefore when working with images like this, it is best to save them in FITS format with IEEE Float pixel format.

Most desktop image editing programs only work at 8 bits. Even if you plan to export data to a desktop editing program for final tweaks, it makes sense to do as much processing as possible at high bit depth first. When that is completed, you should save the result to a high bit depth format, and then export the data to a format compatible with your editing program. This way if you find you need to make additional adjustments you can easily go back without having to start over."

And:

"FITS File Header Definitions
The native file format for MaxIm DL images is FITS (Flexible Image Transport System), a standard widely used in the astronomical community. This is an excellent choice for all types of astronomical images since the format supports 16-bit and floating-point data, and it includes a flexible and highly extensible header. Various standard and widely used non-standard keywords can transfer information about the image between applications. In addition, public domain “C” source code for reading and writing FITS files is available on the web.

One shortcoming of the FITS standard is that the 16-bit number format is signed, producing a range of -32768 to +32767. The range of data from many cameras is 0 to 65535. To overcome this problem, MaxIm DL subtracts 32768 from each pixel prior to saving. The FITS standard requires that the BZERO and BSCALE keywords be used when the value in the array is not the physical value; MaxIm DL sets BZERO to 32768 and BSCALE to 1. Other applications that interpret these keywords correctly should automatically add 32768 to the image pixels, resulting in a correct image display. This default behavior can be overridden using the File menu Settings command."

Is it possible that a conversion could be done to facilitate the images in both programs say via a script? I am assuming that something along this line is happening when the MaxImDl raw or calibrated data is brought into PI, am I correct?

Thanks for your time.

[Juan Conejero]

If I understand this correctly, while both programs are saving the data as 32 IEEE Floating point, there are different values for each program?

That's correct. The core of the problem is the FITS format. FITS has many —too many— problems; one of them is its lack of standardization regarding floating point data. Some call this 'flexible'; a more realistic word is 'undefined' in my opinion.

The FITS format does not specify a standard way to define which pixel values correspond to the black and white points in a floating point image. Consequently, each application is free to use any range that the authors consider appropriate. PixInsight uses the [0,1] range as it is the best option for many reasons which are out of the scope of this post —we can discuss them if someone is interested. Other applications seem to be using a pseudo-16-bit range [0.0,65535.0] as it seems is the case of Maxim, and others clearly use arbitrary ranges that vary inconsistently (as seen from 'outside') from image to image. This is a big mess and there seems to be no intention to fix it.

Is it possible that a conversion could be done to facilitate the images in both programs say via a script?

Our approach with PixInsight is to allow virtually any floating point range for reading FITS and TIFF files (the TIFF format also suffers from the above lack of standardization). This can be done with the FITS and TIFF Format Preferences dialogs. However, PixInsight will always write floating point pixels in the [0,1] range; that won't change. Others should write adaptable interfaces to be able to read our images, as we already have done to be able to read their images. I think this is quite reasonable.

A last note about file formats. We are relatively close to release a first version of a new XML-based image file format, which will be the basis to the new project support in PixInsight. The new format will not have any of the above mentioned problems because everything will be strictly defined and standardized. It will be an open standard and we'll also release open-source development tools to easily integrate the format with existing applications. So I think we'll make a significant contribution here. Of course, a different thing is whether other applications will want to work with us or not. We'll try. Anyway, when we release this new format, FITS and TIFF will be obsolete in PixInsight and we'll maintain them exclusively as legacy modules for compatibility with other applications.

[Carlos Milovic]

However, PixInsight will always write floating point pixels in the [0,1] range; that won't change.

Sorry for throwing the flames here, but I'm strongly against that. As you know, several technical tasks needs "out of normalized range" values. Just to mention one example, a wavelet layer has positive and negative values. So, if we want to save (or display) such kind of images, we need to rescale them. That's a very limiting approach, IMO, and needs to be addressed. It is not a problem of compatibility with other applications, but it is a severe limitation that we have right now, for scientific and technical approaches.
If the solution goes from using a completely new file format, great. The issue is to store the data as it is.


BTW: We should think on a new STF, more flexible, which allows out of range values visualization...

[Juan Conejero]

but it is a severe limitation that we have right now, for scientific and technical approaches.

A limitation exists —although not as severe as you describe it—, but it is not due to the fact that we are using the [0,1] range for *images* —that won't change. It is in the fact that PixInsight right now can only work with images. This will change. As you probably remember, I have already planned a solution to overcome this. I recall it anyway.

A future version of PixInsight will be able to handle two main types of data objects: Image and Array. Image objects will remain exactly as they are now. When you save a floating point image to disk using the graphical interface —you can bypass this feature right now from a script or a PCL-based module— it will be written in the [0,1] range, as happens now.

An Array on the other hand can contain an unlimited number of planes of elements in arbitrary numeric ranges. Array elements are real or complex numbers in 32-bit, 64-bit or 128-bit floating point format. They can be either pixels or voxels, although 3D support won't be available in the initial versions. An Array can be displayed (visualized), although in a less optimized way (the fixed [0,1] range allows for fast LUT table access, partial precalculations and other optimizations). In addition, the Core application will have to detect the minimum and maximum values each time one of these objects is modified in automatic visualization mode, or a fixed visualization range will also be definable.

For example, raw images, wavelet layers and Fourier transforms will be Array objects. You'll be able to work with them with their actual numeric contents. An Array transports arbitrary data. However, an Image object is not intended to transport arbitrary data, but pixel data that has a perceptual meaning. Image objects provide integrated color management, optimized bitmap rendering and other features that don't make any sense for Array objects.

As it's easy to understand, this project requires a lot of work. It involves a redesign of many core subsystems to turn what right now is an image processing application into a general-purpose data processing application. There are critical changes involved at all levels, from the innermost core routines to the whole PCL and PJSR frameworks.

So don't expect this to happen soon; probably not before PI 1.9, perhaps PI 2.0. But it definitely is on the way.

Right now, however, you can write PJSR scripts and PCL C++ modules that work with arbitrary data ranges internally without problems. As you know well, a PCL image can store an arbitrary range of values. In fact, virtually all standard processes work internally outside the [0,1] range. The only 'limitation' is that once you've finished working with your data, if you want to present them as an ImageWindow then you must rescale the values to [0,1], or their screen rendition will be 'funny'. Not a so severe limitation IMO.

[sreilly]

For giggles I decided to save the calibration frames created in PI at 16 bit unsigned to use in MaxIm. What I found is that the needed fits header information is missing. The exposure times, CCD and set temperature values, binning state and others are missing making them useless in any other program. As much as I enjoy the power of PI, it seems that until it can at least do everything from acquiring the data to all processes there should be some conformity to what the other programs support. I thought that was the whole point behind FITS format. Granted, if I didn't use a CCD camera that bloomed this wouldn't be as much an issue but I'd also loose a good deal of sensitivity. If my camera had a square array I'd have another choice of dealing with the blooms but it doesn't.  Now another new format is being discussed that will show up in the near future, relatively speaking, and unless it has a great deal more to offer that already exists, why would any other program want to support it if the difference only applies to giving PI a better processing ability.

Before getting PI I found myself using a number of different programs because no one program is best at all things. I used MaxIm for image acquisition because it does a good job and is required by ACP for automation. I used CCDStack to create my master calibration frames because of its better choices of combining the frames to create the master. MaxIm again because it does a great job of bloom removal using Ron Wodaski"s De-Bloomer plug-in. CCDStack has a great DDP process. MaxImDl does a great job calibrating images in mass by allowing you to load all the calibration frames needed for all the images (Bin 1x1, 2x2, darks with different temperatures, exposure times and bin rates) and matches that to the images being calibrated. I can have 100 images calibrated at one time as long as all the needed cal frames are loaded. Talk about saving time. I'll open all my red, green, blue, and luminance images (rgb at bin 2x2), have my master darks, flats and bias frames loaded in the calibration tool and it reads the FIST header and applies the proper calibration frames to each image automatically.

So while PI has some extremely strong processing tools, it doesn't do all things best. I wouldn't give up PI but it hasn't relieved me of my other programs yet. So yes, it would be great if PI could play nice with the other programs out there. I could be wrong but I haven't had any FITS compatibility issues with any other program. I know some have stated Juan often seems to re-invent the wheel but what advantage is the PI FITS format to what seems to be the accepted FITS format everyone else uses? And as hard as it is setting a tone in e-mail, I'm not hot about this issue but indeed very interested in it.

As for now, I'll just go back to creating my master calibration frames in CCDStack and calibrating the images in MaxIm where I can also de-bloom. From there on I'll continue to work in PI. The attached images show the FITS header for PI created master dark and one from CCDStack. The last is a flat made in PI FITS header.

[Juan Conejero]

The attached images show the FITS header for PI created master dark

You are comparing very different FITS files. One of them (the FITS file with the largest header) is a single dark frame. Of course a dark frame has a number of FITS header keywords to describe acquisition time and conditions, CCD parameters, etc. You are comparing this with the integration of several dark frames, that is, a master dark frame. For example, if you integrate 20 dark frames into your master, then 20 FITS headers should be concatenated in order to preserve the whole original information of the 20 integrated files. Yes, this could be done, and I can modify the ImageIntegration process to do that (as an option). We don't do that because we think it's a waste of space, considering the fact that in PixInsight you can (should) create and save an ImageIntegration process icon where the 20 files are referenced, along with all of the integration parameters used, in a much more efficient way. The same happens with the master flat that you have generated in PixInsight: it is the integration of several flat frames and hence it has no relevant FITS header information for the same reason.

That said, it is true that the ImageIntegration process should save important information to FITS headers, such as clipping statistics, noise estimates, etc. I confess I haven't paid too much attention (or the required attention) to this task, knowing that we'll have a much more capable file format in relatively short time. I'll fix this problem in the next version though.

it seems that until it can at least do everything from acquiring the data to all processes there should be some conformity to what the other programs support.

Soon you'll be able to acquire data with PixInsight (See the work of David Raphael). In the meanwhile, you can acquire the data with other applications and import it into PixInsight without any problem. Hundreds of users do that. Just make sure that all of your raw data are stored in 16-bit unsigned integer format, which is the native format of your CCD camera.

On the other hand, you can calibrate, register and integrate (stack) your images in PixInsight. Our calibration processes involve some manual work, which in the case of CCD images is really easy to carry out:

http://pixinsight.com/tutorials/master-frames/en.html

In addition, we now have a very nice calibration pipeline script that works really well, written by John Brown:

http://pixinsight.com/forum/index.php?topic=2861.0

This script is currently under development. When I have a bit of time I'll help John and hopefully we'll have a first release version soon. We are really excited with this pipeline script.

Along with excellent calibration routines, I honestly think our image registration tools (StarAlignment and DynamicAlignment) are also excellent, as well as our ImageIntegration tool.

If you still want to calibrate, register and integrate your images with other applications, then you should ask the author(s) of the other applications how they write 32-bit floating point images. In particular, you need to know the numeric range to which pixel values are referred (the black and white points). Once you know that values, you can enter them in the FITS Format Preferences dialog and PI will read those images without flaws.

For example, 32-bit floating point images preprocessed with DeepSkyStacker (excellent application) can be loaded in PixInsight without any problems. This happens because DeepSkyStacker writes floating point TIFF images in the [0,1] range.

In a previous post I have explained an optimal solution for blooming removal based on observational techniques (shorter exposures to replace bloomings with real data selectively with a mask). Anyway, if you want to use a software-based deblooming tool with Maxim, it isn't so difficult:

- Save your image as a 16-bit unsigned integer FITS file in PixInsight.

- Open it with Maxim. There should be no problems.

- Use the dblooming tool.

- Save it as a 16-bit unsigned integer FITS file in Maxim.

- Open it in PixInsight.

If the 32-bit -> 16-bit downgrading is a concern, then you can build a mask to isolate your bloomings in the 32-bit original image and replace them with pixels from the debloomed 16-bit image. This will leave your 32-bit data intact (the blooming replacement data are synthetic, so you aren't getting true signal on blooming areas). If you want you can upload a 32-bit PixInsight image with bloomings and a 16-bit debloomed image, and I'll explain you how to build the mask.

A last note about FITS. FITS files generated by PixInsight are fully compliant with the FITS standard. So any FITS-capable application should be able to open and read them without problems. When you open a FITS image with PixInsight and then save it in FITS format again, the whole original metadata (the entire set of FITS header keywords) is always fully preserved. The same is true after calibration. However, when you integrate a set of images with PixInsight, we don't include the FITS headers of all the source frames because we consider that we provide more efficient features to keep the relevant data. Anyway, as noted above I am conscious that we can improve this aspect of the ImageIntegration process, and we'll do. However, don't expect that PixInsight will ever try to reproduce private —and nonstandard— FITS keywords used by other applications. In the same way the other applications will never try to reproduce our private keywords. That is logical. The problem is not in the applications, but in the characteristics and limitations of the FITS format.

Now another new format is being discussed that will show up in the near future, relatively speaking, and unless it has a great deal more to offer that already exists, why would any other program want to support it if the difference only applies to giving PI a better processing ability.

A strong reason is providing compatibility with PixInsight. Along with that, other reasons are:

- Versatile XML format that can be easily extended and adapted.

- Strictly defines a number of critical image elements, such as data types, image geometry, color spaces, numeric ranges, thumbnail images, metadata, ICC color profiles, RGB working spaces, arbitrary image extensions, etc. Metadata are fully compatible with current FITS header keywords, but are not limited by the archaic FITS restrictions: keyword identifiers and values can have unlimited length and fully support Unicode (UTF-8 format).

- Supports real and complex pixel data in all signed and unsigned integer formats from 8 bits to 64 bits, and floating point formats from 32 bits to 128 bits. Numeric ranges are strictly defined for all supported data types. Images can have an unlimited number of dimensions and channels. Optional interpretation of non-nominal channels as transparency data (multiple alpha channels) is also strictly defined.

- Supports multiple images and multiple arrays of arbitrary data (the equivalent to FITS tables).

- Completely isolates metadata from pixel data. This has the advantage that metadata can be easily streamlined. For example, the entire description of a multiple image file can be acquired as a contiguous block of bytes before reading any pixel data.

- Provides nondestructive compression natively. The compression schemes are similar to the ones implemented by the TIFF format.

- Provides incremental reading and writing (reading/writing sequentially by strips of pixel rows) in all formats, even for compressed data.

- It is an open standard, free and freely available.

- It provides cross-platform, open-source development tools released under a liberal BSD-like license. This makes the new format easy to be integrated with existing applications.

Summarizing, our intention is to provide a contemporary and efficient format for storage and transmission of image data. Among a large number of advantages and benefits, such a format would solve all inter-application compatibility problems.

[sreilly]

The attached images show the FITS header for PI created master dark

You are comparing very different FITS files. One of them (the FITS file with the largest header) is a single dark frame. Of course a dark frame has a number of FITS header keywords to describe acquisition time and conditions, CCD parameters, etc. You are comparing this with the integration of several dark frames, that is, a master dark frame. For example, if you integrate 20 dark frames into your master, then 20 FITS headers should be concatenated in order to preserve the whole original information of the 20 integrated files. Yes, this could be done, and I can modify the ImageIntegration process to do that (as an option). We don't do that because we think it's a waste of space, considering the fact that in PixInsight you can (should) create and save an ImageIntegration process icon where the 20 files are referenced, along with all of the integration parameters used, in a much more efficient way. The same happens with the master flat that you have generated in PixInsight: it is the integration of several flat frames and hence it has no relevant FITS header information for the same reason.

That said, it is true that the ImageIntegration process should save important information to FITS headers, such as clipping statistics, noise estimates, etc. I confess I haven't paid too much attention (or the required attention) to this task, knowing that we'll have a much more capable file format in relatively short time. I'll fix this problem in the next version though.

Actually that is a master dark from most likely 20-30 dark frames combined in CCDStack as a master.

it seems that until it can at least do everything from acquiring the data to all processes there should be some conformity to what the other programs support.

Soon you'll be able to acquire data with PixInsight (See the work of David Raphael). In the meanwhile, you can acquire the data with other applications and import it into PixInsight without any problem. Hundreds of users do that. Just make sure that all of your raw data are stored in 16-bit unsigned integer format, which is the native format of your CCD camera.

I saw that and am excited at the prospect but if PI can not be scripted by ACP, it won't do me much good. I spent a good amount of time and money automating the observatory with which I get a good amount of data without having to baby sit. ACP scripts the acquisition of images, plate solves for centering accuracy, focuses at given intervals, and shuts down the observatory at the end of a run or if the weather gets bad.

On the other hand, you can calibrate, register and integrate (stack) your images in PixInsight. Our calibration processes involve some manual work, which in the case of CCD images is really easy to carry out:

http://pixinsight.com/tutorials/master-frames/en.html

In addition, we now have a very nice calibration pipeline script that works really well, written by John Brown:

http://pixinsight.com/forum/index.php?topic=2861.0

This script is currently under development. When I have a bit of time I'll help John and hopefully we'll have a first release version soon. We are really excited with this pipeline script.

Along with excellent calibration routines, I honestly think our image registration tools (StarAlignment and DynamicAlignment) are also excellent, as well as our ImageIntegration tool.

I have no issue with any of this. I have promoted the use of PI in any image processing circle I happen to be in. I find alignment as good if not better than Registar which I bought a number of years ago and has been a registration standard for many users. 

If you still want to calibrate, register and integrate your images with other applications, then you should ask the author(s) of the other applications how they write 32-bit floating point images. In particular, you need to know the numeric range to which pixel values are referred (the black and white points). Once you know that values, you can enter them in the FITS Format Preferences dialog and PI will read those images without flaws.

For example, 32-bit floating point images preprocessed with DeepSkyStacker (excellent application) can be loaded in PixInsight without any problems. This happens because DeepSkyStacker writes floating point TIFF images in the [0,1] range.

PI has no problem reading MaxIm's image files. It's the other way around and only an issue for de-blooming using software which works well for me as well as many others. CCDStack also has a de-bloomer tool which can't be used with PI images saved as 32-bit floating point for the same reason.

In a previous post I have explained an optimal solution for blooming removal based on observational techniques (shorter exposures to replace bloomings with real data selectively with a mask). Anyway, if you want to use a software-based deblooming tool with Maxim, it isn't so difficult:

- Save your image as a 16-bit unsigned integer FITS file in PixInsight.

- Open it with Maxim. There should be no problems.

- Use the dblooming tool.

- Save it as a 16-bit unsigned integer FITS file in Maxim.

- Open it in PixInsight.

If the 32-bit -> 16-bit downgrading is a concern, then you can build a mask to isolate your bloomings in the 32-bit original image and replace them with pixels from the debloomed 16-bit image. This will leave your 32-bit data intact (the blooming replacement data are synthetic, so you aren't getting true signal on blooming areas). If you want you can upload a 32-bit PixInsight image with bloomings and a 16-bit debloomed image, and I'll explain you how to build the mask.

That would be great!

A last note about FITS. FITS files generated by PixInsight are fully compliant with the FITS standard. So any FITS-capable application should be able to open and read them without problems. When you open a FITS image with PixInsight and then save it in FITS format again, the whole original metadata (the entire set of FITS header keywords) is always fully preserved. The same is true after calibration. However, when you integrate a set of images with PixInsight, we don't include the FITS headers of all the source frames because we consider that we provide more efficient features to keep the relevant data. Anyway, as noted above I am conscious that we can improve this aspect of the ImageIntegration process, and we'll do. However, don't expect that PixInsight will ever try to reproduce private —and nonstandard— FITS keywords used by other applications. In the same way the other applications will never try to reproduce our private keywords. That is logical. The problem is not in the applications, but in the characteristics and limitations of the FITS format.

Just to be clear I've attached a combined PI FITS Header screen shot and a MaxIm FITS Header screen shot. These are master frames combined in their respective software of real images.

Now another new format is being discussed that will show up in the near future, relatively speaking, and unless it has a great deal more to offer that already exists, why would any other program want to support it if the difference only applies to giving PI a better processing ability.

A strong reason is providing compatibility with PixInsight. Along with that, other reasons are:

- Versatile XML format that can be easily extended and adapted.

- Strictly defines a number of critical image elements, such as data types, image geometry, color spaces, numeric ranges, thumbnail images, metadata, ICC color profiles, RGB working spaces, arbitrary image extensions, etc. Metadata are fully compatible with current FITS header keywords, but are not limited by the archaic FITS restrictions: keyword identifiers and values can have unlimited length and fully support Unicode (UTF-8 format).

- Supports real and complex pixel data in all signed and unsigned integer formats from 8 bits to 64 bits, and floating point formats from 32 bits to 128 bits. Numeric ranges are strictly defined for all supported data types. Images can have an unlimited number of dimensions and channels. Optional interpretation of non-nominal channels as transparency data (multiple alpha channels) is also strictly defined.

- Supports multiple images and multiple arrays of arbitrary data (the equivalent to FITS tables).

- Completely isolates metadata from pixel data. This has the advantage that metadata can be easily streamlined. For example, the entire description of a multiple image file can be acquired as a contiguous block of bytes before reading any pixel data.

- Provides nondestructive compression natively. The compression schemes are similar to the ones implemented by the TIFF format.

- Provides incremental reading and writing (reading/writing sequentially by strips of pixel rows) in all formats, even for compressed data.

- It is an open standard, free and freely available.

- It provides cross-platform, open-source development tools released under a liberal BSD-like license. This makes the new format easy to be integrated with existing applications.

Summarizing, our intention is to provide a contemporary and efficient format for storage and transmission of image data. Among a large number of advantages and benefits, such a format would solve all inter-application compatibility problems.
[/quote]

I hope this is picked up and time will tell.

[sreilly]

If the 32-bit -> 16-bit downgrading is a concern, then you can build a mask to isolate your bloomings in the 32-bit original image and replace them with pixels from the debloomed 16-bit image. This will leave your 32-bit data intact (the blooming replacement data are synthetic, so you aren't getting true signal on blooming areas). If you want you can upload a 32-bit PixInsight image with bloomings and a 16-bit debloomed image, and I'll explain you how to build the mask.

After sleeping on this thought it occurred to me that this process would need to be done on every single image that had a bloom, one at a time. Meaning if I have 50 images with blooms, not unusual, I would need to create a mask for each frame as the intensity of the bloom varies with the night's imaging conditions. As an example, the data for the Thor's Helmet image consists of 41 images and this isn't as deep as I usually image. If my thinking is correct I'd need to create a mask for each that contained at least one bloom, most of them, and remove each bloom.

[pfile]

If the 32-bit -> 16-bit downgrading is a concern, then you can build a mask to isolate your bloomings in the 32-bit original image and replace them with pixels from the debloomed 16-bit image. This will leave your 32-bit data intact (the blooming replacement data are synthetic, so you aren't getting true signal on blooming areas). If you want you can upload a 32-bit PixInsight image with bloomings and a 16-bit debloomed image, and I'll explain you how to build the mask.

After sleeping on this thought it occurred to me that this process would need to be done on every single image that had a bloom, one at a time. Meaning if I have 50 images with blooms, not unusual, I would need to create a mask for each frame as the intensity of the bloom varies with the night's imaging conditions. As an example, the data for the Thor's Helmet image consists of 41 images and this isn't as deep as I usually image. If my thinking is correct I'd need to create a mask for each that contained at least one bloom, most of them, and remove each bloom.

can't the deblooming be done on the stacked result? might not be optimal, but the result should be similar.

i would be interested to see the procedure for creating the deblooming mask. in my case i think i'd want to fill in the bloomed areas with data from shorter exposures; i assume LinearFit would come into play on the shorter exposure before you copied those pixels in.

[sreilly]

I wonder of the effect the bloomed area being substituted with the shorter exposure would be like? The signal would certainly be lower than the rest of the bloomed image. How many "shorter, non bloomed" images would it require to fill the bloomed area satisfactorily? The idea of rotating a square CCD array 90 degrees is great if you have a square array and a guide star at both rotated positions. Having used 24" RCs in the past I could image unguided for several minutes and this would have been no issue but with a 12.5" f/9 system, not nearly as many photons being collected in that short period of time.

I should note that de-blooming a stacked image has way too many artifacts left compared to each being de-bloomed using the de-blooming software. Using the mask method Juan is referring to may have a different outcome. I'll upload the needed frames and see what Juan can suggest shortly.

[sreilly]

I have posted the two images with http://www.astral-imaging.com/NGC%202359-S004-R011-C001-Luminance_c.fit being the PI calibrated bloomed image (single frame) and
http://www.astral-imaging.com/NGC%202359-S004-R011-C001-Luminance16.fit being the 16 bit de-bloomed identical image. It should be interesting what can be done with image masking to replace the data lost to the blooms.