New File Format: XISF - Extensible Image Serialization Format

[rga218]

Philip: Storing images as complex numbers is useful in a number of domains, most notably for polarimetry and for radio astronomy (where you need to capture phase information).

[Carlos Milovic]

Phillip,
Complex samples are those that have real and imaginary components. This sample format is used a lot when working with fourier transforms, since the fourier transform of a real image is a complex image. Also complex samples are used in the context of biomedical imaging. It's more important use is in Magnetic Resonance Imaging.

So, if anyone is working with theyr images in the fourier domain, or planning to do biomedical imaging, this is good news.

[Juan Conejero]

Hi Bob,

Lack of support for Multi-Extension FITS files (the standard adopted by most large telescopes) has been the #2 issue in my attempt to get more of my professional astronomer colleagues to use PixInsight

Our implementation of the FITS format has supported multi-extension FITS files for a long time. At least since 2006 if I can remember well; probably even 2005. Just to put a well-known example for everyone, if you use the BatchPreprocessing script and select ".fit" as the output file suffix, all generated master files will be multi-extension FITS files that you can use in PixInsight without problems. Each of these masters store three or four images in a single FITS file: the integrated master frame (science, bias, dark, or flat) and the low and high rejection map images, plus the slope map image if you select linear fit clipping. It has always worked that way.

Another example: we use FITS extensions to store ICC color profiles, which are absolutely essential in PixInsight—well, they are essential for any image processing application that works with real-world images.

It is true that we could develop a better graphical interface to manage multiple images—it is also true that other tools commonly used by professionals lack any graphical interface at all—, but multi-extension FITS are definitely supported in PixInsight.

(#1 has been missing documentation).

Touchée. Believe it or not, we really do what we can with the resources that we have.

In the professional community FITS (and its gradual evolution to embrace extensions and tables) is generally viewed as a universal standard that all software really must adhere to in order to gain acceptance. Multi-Extension FITS is backwards compatible with vanilla FITS (at least when using CFITSIO, though I guess it might be quite a lot of work to implement in PixInsight as I don't believe you're using CFITSIO).

We use CFITSIO in our FITS support module. This is true since the first versions of PixInsight LE that we released back in 2003.

Most aspects of the new XISF format are already implementable using multi-extension FITS, and in my opinion the exceptions (e.g. unicode, signatures, and the distributed access model) probably would be viewed by most of my colleagues as insufficiently important to require a new standard.

I really don't want/need to criticize FITS here—if you have read me on this forum, you probably know some of the problems that we have been forced to face as a result of FITS limitations. We prefer to look forward for efficient solutions now, and that's the main reason for starting this project. Now that you mention it, the lack of Unicode support, distributed storage, unsigned integer pixel data (the BZERO/BSCALE trick is, well, a trick), a formal description of floating point pixel sample ranges, color spaces, color profiles, image thumbnails, strongly typed properties, data compression, along with the obsolete file header organization, are important for us.

...I do appreciate the desire to innovate!

Thank you so much for that!

P.S. PixInsight was used to reduce much of the data for this recent paper: http://arxiv.org/abs/1410.8141

And I am very proud of that

[jerryyyyy]

All this scares novice user like myself who have barely come to understand FITs.  I have a lot of $ invested in software whose final pathway is FITs... e.g. Maxim... which is bad enough.  I do not think I would have come to use PI if it were using a format I had not heard of [purchase is nonstarter]. 

[Philip de Louraille]

Roberto & Carlos: Thanks for the examples. I hadn't thought about radiotelescopy (and MRI) and the needs to store a phase.
Won't be long before PI can store files with a tensor-like format. ;-)

Jerryyyyy: no worries. Just because PI can work with formats you don't want to use does not mean you wasted your money. You can stay with what you know and use.

[rga218]

Hola Juan,

Hi Bob,

Our implementation of the FITS format has supported multi-extension FITS files for a long time. At least since 2006 if I can remember well; probably even 2005. Just to put a well-known example for everyone, if you use the BatchPreprocessing script and select ".fit" as the output file suffix, all generated master files will be multi-extension FITS files that you can use in PixInsight without problems. Each of these masters store three or four images in a single FITS file: the integrated master frame (science, bias, dark, or flat) and the low and high rejection map images, plus the slope map image if you select linear fit clipping. It has always worked that way.

Great to learn you guys are using CFITSIO and I'm sorry I was being unclear about my need to access MEF files! I know you can open an MEF image in PixInsight, but what I mean is: can I extract data in a particular extension from the console? I'm probably still being unclear, so for example, let's say I have a file foo.fits and it has 5 channels and I want to access the 3rd extension, which is the science channel, from the process console. The FITSIO syntax to access this data is foo.fits[SCI] or foo.fits[3] or foo.fits [SCI,3]. But if I try:

"open foo.fits[3]"

In the PixInsight process console this doesn't work. Am I doing something stupid? (Probably).

Touchée. Believe it or not, we really do what we can with the resources that we have.

Oh, I believe it! You've accomplished an incredible amount with a really small team, and when it comes to providing documentation you do better than me... you should see what undocumented stuff I have to foist on my collaborators just because there are not enough hours in the day sometimes.

I really don't want/need to criticize FITS here—if you have read me on this forum, you probably know some of the problems that we have been forced to face as a result of FITS limitations. We prefer to look forward for efficient solutions now, and that's the main reason for starting this project. Now that you mention it, the lack of Unicode support, distributed storage, unsigned integer pixel data (the BZERO/BSCALE trick is, well, a trick), a formal description of floating point pixel sample ranges, color spaces, color profiles, image thumbnails, strongly typed properties, data compression, along with the obsolete file header organization, are important for us.

I have total respect for that. The thing about PixInsight is that it is true to itself... go for it! By any rational standard most professional software is much quirkier than PixInsight, though of course those quirks are well documented, standardized, etc etc. For example, the lack of a formal description of floating point pixel ranges is a non-issue in all professional software because everybody understands what is going on (in fact, even though it's pretty rational, PixInsight's scaling to the range [0,1] is #3 on the list of things people complain about when I try to sell them on PixInsight).

Anyway, when it comes to file formats, FITS in particular is just set in stone and in the interests of consistency and interchange with decades of data in the archives my professional colleagues and I just have to put up with any deficiencies. Just read the angst that went into getting everybody to agree on standardizing on FITS in the first couple of sections of the PDF maintained by the IAU working group:

http://fits.gsfc.nasa.gov/standard30/fits_standard30aa.pdf

Of course, you aren't aiming PixInsight at professional astronomers as your main market, so you needn't feel hidebound to adhere to things (like common file formats for even intermediate data exchange). IMHO moving away from FITS as a default will limit your reach in the professional community, but everything's a trade-off. If you feel a need to ditch FITS as the default file format because you find it limits your ability to innovate in other ways, well, that's clearly a reasonable choice to make. If I were writing the software and not you I wouldn't do that, but you know your market and I don't, and if I wrote PixInsight it would be a lot less interesting and compelling than you've made it.

Bob

[Juan Conejero]

Jerryyyyy: no worries. Just because PI can work with formats you don't want to use does not mean you wasted your money. You can stay with what you know and use.

Absolutely. The XISF project is here to make everything more efficient and versatile. You can always decide to write your data in other formats, including FITS, TIFF, PNG, JPEG and others, which are supported and will be supported in PixInsight. XISF is from now on our native format, and you probably will like it when you know what it's capable of, but it won't limit your use of our software; it will do just the opposite to that.

[Juan Conejero]

...can I extract data in a particular extension from the console?

That can't be done, mainly because our FITS support module does not expose that functionality of CFITSIO. However, you can access any FITS extension very easily with a relatively simple JavaScript script (about 50 lines I think). Scripts can be invoked from the console.

(in fact, even though it's pretty rational, PixInsight's scaling to the range [0,1] is #3 on the list of things people complain about when I try to sell them on PixInsight).

The problem here is the difference between images and data. As we understand the concept, an image is a data structure whose main purpose is to be visually representable. As a consequence of its visual role, an image is subject to a number of constraints such as color spaces, color transformations, representable ranges, etc, that in turn require auxiliary data structures and formal definitions. Generic data structures don't have any of these constraints a priori, so there's generally no need to know the limits of the representable range and similar properties. One of the most critical problems that we have with FITS is precisely here: it is a format designed to store generic data, but quite ironically, it does not help at all to transport images—at least not for the concept of image that an image processing application needs to work with real-world images. We have addressed this problem with XISF, where images and data objects (which we call properties) are formally supported by different structures, and images have all the associated elements required to represent and use them consistently.

[Jay W. Butler]

I must tell you that in my case, this new format is a solution looking for a problem. I am still learning to use PI, which has been a very steep uphill slog. I don't want PI to start saving my images in a new file format that I am unfamiliar with and that may have consequences or properties that I don't know to handle.
What advantage, if any, does this new file format provide for an intermediately skilled amateur who just wants to process LRGB images with occasional narrowband additions?
Can I just continue to save my processed subs as FITS files? Am I going to have to make that election every time, or can I set it as a default on my system?
Jay Butler

[Juan Conejero]

Hi Jay,

The default output file format can be set as a global preferences option, so don't worry about that. As for advantages and features, please read the posts above.

this new format is a solution looking for a problem.

It is actually a solution to a number of problems.

[rga218]

We have addressed this problem with XISF, where images and data objects (which we call properties) are formally supported by different structures, and images have all the associated elements required to represent and use them consistently.

It sounds like PixInsight's approach will resemble that of Mathematica and MATLAB, which treat images as distinct from other forms of 2D data. I certainly understand your point of view and do see the logic of it, and furthermore see how what you're doing with XISF will be helpful if it is a component in this plan.

If I understand your description properly, then at least for my sorts of data flows 90% of the time I'm going to manipulating 'properties' and not 'images'. I'm actually totally fine with that.... seems like a good idea. Most of the time I'm manipulating generic 2D data that has proper units like mJy/pixel or electron/pix/sec, and only as the final step when I need to generate the final picture is it helpful for me to think about how to rescale my property into an 'image' in a color space with channels made up of [0..255] RGB or grayscale in the range [0..1] via screen transfer function or whatever.

Bottom line: If I can read a FITS image into a property, and not have to rescale it when read it in, and then I can keep it from being rescaled throughout all the data flow (calibration and other manipulations such as stacking) and only have to convert it to an image at the end of a long chain of analysis, that would make me very happy! I guess I'm hoping most of the (relevant) modules (things like PixelMath, Resample, DynamicCrop, DynamicPSF, ImageIntegration) will work on properties as well as on images, and then as the last step I hit my property data with a ScreenTransferFunction to turn it into an image.  If that is the case then I'm an XISF believer.

If what I wrote above makes sense, perhaps you might considering renaming ImageIntegration to be PropertyIntegration or even ImagePropertyIntegration? (Or maybe not... perhaps that would break too many things).

Bob

[jkmorse]

Jay,

+1 to Juan's response.  FITS incompatibilities have been an issue for PI and other software for years and is a continuing issue due to the lack of a set convention for using FITS in various pieces of software.  This is definitely a problem in need of a solution and not the other way around.  Take a look at this post to see some of what I mean:

http://pixinsight.com/forum/index.php?topic=4880.0

This is only one of dozens of posts on the same subject.

Juan, thanks for the clarifications on the practical side of this update and for continuing to push the envelope and taking us out of our comfort zone to drive better processing results.

Best,

Jim

[Geoff]

Juan, thanks for the clarifications on the practical side of this update and for continuing to push the envelope and taking us out of our comfort zone to drive better processing results.

Best,

Jim

+1

Henry Ford, on innovation, once said “If I had asked my customers what they wanted, they would have said ‘faster horses’”.
Geoff

[Joe Perulero]

Well done Juan , finally a brave move which I believe will pay dividends for PI later on !

[edd]

Does XISF have a documented specification yet or is it only by a reference implementation so far? I recognise these are early days if a document is planned but isn't ready yet.