New File Format: XISF - Extensible Image Serialization Format

[Juan Conejero]

Hi everyone!

Today we are glad to announce a new development project: XISF. XISF stands for Extensible Image Serialization Format, and is from now on the file format used by default in PixInsight for storage and management of images and their associated data and properties. XISF effectively replaces the FITS format, which is now deprecated in PixInsight, and will be supported exclusively for compatibility with existing data and applications.

XISF is an open file format. This means that its formal specification is publicly available, it can be freely used and implemented by anyone, and is open to contributions from third parties, including all software developers involved in PixInsight development, as well as individuals and teams from other companies and groups. The official XISF development repository will be part of the official PixInsight GitHub repositories.

We have just released a new open-source XISF module for all supported platforms (FreeBSD, Linux, OS X and Windows). The XISF module is an initial implementation of the XISF version 1.0 specification. We have also released a new version 1.38 of the BatchPreprocessing script that uses XISF for all generated output by default. Both updates are now available for PixInsight Core 1.8.3.1123.


Main Format Features

The initial XISF version 1.0 specification has the following main features:

* Two storage models:

   - Monolithic model, where all image metadata, pixel sample data and image properties are stored in a single storage unit (e.g. a disk file). Monolithic XISF file names carry the .xisf suffix.

   - Distributed model, where image definitions and metadata are stored in a single header unit, and pixel sample data and image properties can be stored as separate units, including local and remote storage (for example, we can send you a XISF header that points to image contents and properties stored in a remote server). XISF header units carry the .xish file suffix, while XISF binary data blocks (known as 'XISF blobs') have the .xisb file suffix.

   The initial implementation of XISF that we have released as a PixInsight module only supports monolithic XISF files.

* Storage of multiple images, metadata and image properties.

   An XISF file can store an unlimited number of images with different geometries, color spaces, and associated metadata and properties.

* Multidimensional images.

   XISF supports arbitrary image dimensions, including one-dimensional, two-dimensional and three-dimensional images, as well as images of higher dimensions such as tesseracts and other hypercubes.

   The current XISF PixInsight module only supports two-dimensional images.

* Multiple image channels.

   XISF supports an unlimited number of image planes (not to be confused with the dimensionality of the image), such as nominal color channels and alpha channels, for each stored image.

* Sequential and random access to pixel data.

   Monolithic XISF files support sequential and random access to load individual pixel sample rows or sets of contiguous rows.

* Seven pixel sample data types supported:

   - Unsigned 8-bit integer real
   - Unsigned 16-bit integer real
   - Unsigned 32-bit integer real
   - 32-bit IEEE 754 floating point real
   - 64-bit IEEE 754 floating point real
   - 32-bit IEEE 754 floating point complex
   - 64-bit IEEE 754 floating point complex

* Seven color spaces supported:

   - Grayscale
   - RGB
   - CIE XYZ
   - CIE L*a*b*
   - CIE L*c*h*
   - HSV
   - HSI

* Support for RGB working spaces.

   XISF can serialize the parameters of colorimetrically defined RGB working spaces, which can be associated with stored images.

* Support for ICC color profiles.

   XISF can store standard ICC color profiles associated with stored images.

* Arbitrary floating point data ranges.

   Floating point images (both real and complex) can be stored using arbitrary values and ranges. However, an unambiguous specification of the numeric range to which floating point image data is referred—that is, the black and white points of the image—is mandatory. Specifying the numeric range of integer image data is optional (by default, integer images utilize the whole [0,2ⁿ-1] range, where n is the sample bit depth)..

* Data compression.

   XISF supports arbitrary data compression codecs, including lossless and lossy compression schemes.

   The current XISF PixInsight module supports zlib/defalte lossless compression.

* Digital signatures.

   XISF implements digitally signed files with standard X.509 certificates and XML signatures. Digital signatures protect both data integrity and authenticity.

   Note: The current PixInsight XISF module still does not support digital signatures; this feature will be available in an upcoming version.

* XML file headers.

   XISF headers (in both the monolithic and distributed models) are plain text in standard XML 1.0 format encoded as UTF-8.

* Full Unicode support.

   Image metadata and textual properties fully support Unicode. XISF headers are encoded as UTF-8 data.

* Strongly-typed image properties.

   An XISF property is a data block accessible through a unique identifier (name-value pairs). Properties can be associated with images, with the whole XISF unit, or be serialized as independent elements. XISF properties are strongly-typed objects of scalar, vector and matrix types:

   Scalar property types:

      Boolean (a one-byte integer with 0 or 1 value)
      8-bit signed integer
      8-bit unsigned integer
      16-bit signed integer
      16-bit unsigned integer
      32-bit signed integer
      32-bit unsigned integer
      64-bit signed integer
      64-bit unsigned integer
      32-bit IEEE 754 floating point (float)
      64-bit IEEE 754 floating point (double)

   Vector property types:

      ByteArray - a vector of unsigned 8-bit integers
      IVector - a vector of signed 32-bit integers
      UIVector - a vector of unsigned 32-bit integers
      I64Vector - a vector of signed 64-bit integers
      UI64Vector - a vector of unsigned 64-bit integers
      F32Vector - a vector of 32-bit IEEE 754 floating point real values
      F64Vector - a vector of 64-bit IEEE 754 floating point real values
      String16 - a null-terminated vector of 16-bit Unicode code points (UTF-16)
      String8 - a null-terminated vector of 8-bit ISO/IEC-8859-1 or Unicode code points (UTF-8)

   Matrix property types:

      ByteMatrix - a matrix of unsigned 8-bit integers
      IMatrix - a matrix of signed 32-bit integers
      UIMatrix - a matrix of unsigned 32-bit integers
      I64Matrix - a matrix of signed 64-bit integers
      UI64Matrix - a matrix of unsigned 64-bit integers
      F32Matrix - a matrix of 32-bit IEEE 754 floating point real values
      F64Matrix - a matrix of 64-bit IEEE 754 floating point real values

* FITS compatibility.

   XISF is fully compatible with the FITS format. FITS header keywords can be stored in XISF files and then retrieved just as if you were working with actual FITS files. In this way existing tools and processing pipelines can work with XISF files without modification in PixInsight and other applications.


XISF Roadmap

Today we have released an initial implementation of XISF on the PixInsight platform. The full source code of this implementation is also available on our GitHub repositories under PCL license.

However, this is not an official release of the XISF format, mainly because we still lack a formal definition of XISF. Along with the need of a formal definition, we've got a lot of work to do with this project. The current roadmap for the XISF project includes the following main goals:

• Release of a formal definition document for XISF version 1.0. This is a top-priority task for us, so this document will be available in the incoming weeks.
• Release of a reference, open-source, multiplatform implementation of XISF in the C++ language. This implementation will be available under a BSD license on the PixInsight GitHub repositories.
• Contact the developers of other software applications, especially image acquisition applications, asking them to support XISF. We'll assist all interested teams in their XISF implementations.
• Contact other developers and development teams to encourage them to participate in the development of XISF.
• Further development of the XISF implementation in PixInsight.

I hope you'll enjoy this new initiative. Thank you for your continued support.

[davehudsn]

Juan,

With all due respect, an open format generally means it is a format agreed upon by a relevant consortium and not a format derived by a company and them deemed open because it's format is declare and not proprietary.

The FITS format is the Astronomical standard and I am not aware of a deviation from that.  It concerns me that you state (imply) that you will support current FITS formats, but not future releases/standards of the FITS format.

Please accept my apologies if I am in error here, but it feels like I've made a major investment in a piece of software that is switching formats and not planning on support the community standard in the future.   Could you clarify this?


Thanks,
Dave

[William McLaughlin]

I would be shocked if .fits is no longer supported and suspect this will never be the case.

My guess is that this is intended to better support the amateur imaging crowd (for which .fits was
not really designed) and be offered as an additional format for capture programs to support, offering
added features for that audience. As such, it's success will be dependent on camera control software
supporting the new format so remains to be seen. Given the large percentage of imagers that use PI,
I think it has a good chance as imaging control software writers will be aware of this.

Please correct me if this is wrong.

FYI, I have posted a link to this thread over on the SGP (Sequence Generator Pro) forum...

[Starlight]

I'm also surprised and concerned by the announcement. Fits is a universally accepted format for astronomical imaging and to produce your own format strikes of protectionism - Pixinsight is a well respected piece of software but it is not the only one - I'm sure there are a number of people that like to use Pixinsight alongside other astro software during their processing workflow.
I hope that you will continue to fully support the fits format and allow people to choose what format they use at all stages.

Don

ps It seems a very strange business plan to make your product less universally acceptable

[pmesquita]

same here. I've been learning the long learning curve of imaging and of the PI tools and only recently have I standardized my .fits work. I hope I keep getting .fits and an option with PI.

best regards
Paulo Mesquita

[btracy]

Perhaps Juan can take the time to explain just what the implications are for the users.  I, for one, do not have the foggiest idea what the XISF format means to me but I do know (as a retired software developer) that Juan and crew wouldn't be investing a lot of programmer time to develop this without good reason.

BobT

[rga218]

In this particular case I guess I wish that instead of developing a new file format PixInsight had embraced Multi-Extension FITS (MEF) format to add much of the desired functionality in the XISF format.

http://www.stsci.edu/hst/HST_overview/documents/datahandbook/intro_ch23.html

Lack of support for Multi-Extension FITS files (the standard adopted by most large telescopes) has been the #2 issue in my attempt to get more of my professional astronomer colleagues to use PixInsight (#1 has been missing documentation). In the professional community FITS (and its gradual evolution to embrace extensions and tables) is generally viewed as a universal standard that all software really must adhere to in order to gain acceptance. Multi-Extension FITS is backwards compatible with vanilla FITS (at least when using CFITSIO, though I guess it might be quite a lot of work to implement in PixInsight as I don't believe you're using CFITSIO). And Multi-Extension FITS is used as the standard data format for pretty much every instrument on every major optical-IR telescope I'm aware of (HST, Gemini, VLT, CFHT, etc.) and many radio telescopes. Most aspects of the new XISF format are already implementable using multi-extension FITS, and in my opinion the exceptions (e.g. unicode, signatures, and the distributed access model) probably would be viewed by most of my colleagues as insufficiently important to require a new standard. If PixInsight used Multi-Extension FITS I believe you'd get most of the capability of the new XISF format you're implementing, retain compatibility with old FITS files, and open up the capability of having amateurs using PixInsight easily use data from the archives of professional telescopes. 

That said, I am probably being a bit parochial here, and I do appreciate the desire to innovate! The core audience here are amateur astronomers, who probably have very different priorities to my own. In any case, PixInsight is sufficiently wonderful that I'll stick with it regardless of the file format whenever I can, so please take the comments above as an expression of my enthusiasm for the software, and a desire to be able to use it with even more telescopes and datasets.

Bob

P.S. PixInsight was used to reduce much of the data for this recent paper: http://arxiv.org/abs/1410.8141

[jkmorse]

Juan,

I am more interested in the practical application of using the XISF format.  Since most image capture software uses FITS, are you recommending that we do a format conversion before doing any processing steps.  If so, do you recommend starting with unsigned 16 bit integers or jumping right to 32 bit floating point format.

Also, are all the other PI processes able to work with XISF formatted images or will we need to wait for tool by tool conversions.

Thanks,

Jim

[kayronjm]

I too am not keen on having FITS replaced by the new format, or any format. If the team would like to develop this new format to present new workflow features, then fair enough, but PixInsight should maintain 100% FITS compatibility, now and in all future releases.

[Juan Conejero]

With all due respect, an open format generally means it is a format agreed upon by a relevant consortium and not a format derived by a company and them deemed open because it's format is declare and not proprietary.

If the Linux Information Project can be considered as a reliable source:

http://www.linfo.org/free_file_format.html

So according to the above definition, XISF is actually not only an open format, but a free format:

A free file format is a file format that is both (1) published so that anyone can read and study it in its entirety and (2) not encumbered by any copyrights, patents, trademarks or other restrictions so that anyone may use it at no monetary cost for any desired purpose. Such specifications are usually maintained by a non-commercial standards organization.

XISF complies with requirements (1) and (2) of the above definition. The only bit that we don't have here is the "maintained by a non-commercial standards organization" for obvious reasons. But it says "usually", not "necessarily", and as I have said in the starting post, we'd be glad if XISF were maintained and developed by a group of developers beyond our team. But let us define the initial formal specification first, and write a first reference implementation. Then XISF will be all yours.

The FITS format is the Astronomical standard and I am not aware of a deviation from that.  It concerns me that you state (imply) that you will support current FITS formats, but not future releases/standards of the FITS format.

Don't be concerned at all. We will continue supporting FITS and future versions of the FITS standard. In fact, we have been implementing the FITS standard for a long time, when other competing applications have not (and still don't, in some cases).

XISF will be the native file format of the PixInsight platform when we release the next version of the PixInsight Core application, and as such it will replace FITS by default. But default option does not mean only option. Of course you'll be free to select the FITS format as default if you want. And you can always save an XISF file in FITS format. Because XISF stores FITS header keywords since the beginning of the process, no FITS metadata will be lost since the moment you load your raw FITS frames until the final processed product.

[jkmorse]

Personally I applaud the effort to keep improving.  FITS may be the standard but that does not mean it's ideal.  Just look at the number of posts related to FITS incompatibilities to understand that sticking with the old method, just because it is old, if hardly the way to advance the efforts of Juan and his team to continually bring us new and better image processing tools.  Even if it only makes the FITS incompatibility problems a thing of the past, it will be well worth it.

For what it's worth.

Jim

[Juan Conejero]

I would be shocked if .fits is no longer supported and suspect this will never be the case.

Rest assured about that. We'll continue supporting FITS in the foreseeable future.

My guess is that this is intended to better support the amateur imaging crowd

IMHO, XISF could support the professional crowd better than FITS. Actually, we started the XISF project some time ago partly because FITS has a number of important limitations and problems (for us) that we have been suffering for a long time.

...it's success will be dependent on camera control software supporting the new format so remains to be seen. Given the large percentage of imagers that use PI, I think it has a good chance as imaging control software writers will be aware of this.

That's indeed a realistic view of the topic. Our main intention with XISF is to improve the experience of our users on the PixInsight platform. In this sense XISF should be considered as the native file format of PixInsight. We will make efforts to convince the developers of other image acquisition applications to implement XISF, and will be glad to assist them in their implementations. The availability of a reference implementation is crucial to achieve this goal.

[edd]

As a professional astronomer too I welcome this. FITS is versatile and well supported by some languages, but has some nasty features that make it a pain to use as well as one might like. Headers in particular are nasty. Having partially implemented FITS parsers from scratch in the last it's been a less than wonderful experience.

I've considered writing my own specification with, for example, XML headers for such a file format but never got around to it. I'd hope this starts people thinking about using it in other situations too and push this format further forward.

Clearly FITS support isn't going away and a converter between the two is essential for now but I think a new more modern format with the same capabilities makes it easier to develop new code, and apply other tools without being so dependent on cfitsio and the like and to cover the core functions.

Good work and a big thumbs up from me.

[Juan Conejero]

Hi Jim,

Since most image capture software uses FITS, are you recommending that we do a format conversion before doing any processing steps.

Not really. Your raw data stored as FITS files will continue working as input data for the ImageCalibration tool for example. IC will store calibrated images as XISF files by default, and from that point subsequent preprocessing stages will work with XISF. Actually, you can do this right now by just replacing the default ".fit" with ".xisf" in the output extension parameter of IC. In the next version the default output extension will be ".xisf", that's all. Nothing will be lost because XISF can store all FITS header keywords and retrieve them transparently, just as if it was FITS.

do you recommend starting with unsigned 16 bit integers or jumping right to 32 bit floating point format.

Raw CCD data are unsigned 16-bit integers, and no file format specification can change this. Raw CCD frames should *always* be stored in unsigned 16-bit format, no matter the formats and tools used.

Also, are all the other PI processes able to work with XISF formatted images or will we need to wait for tool by tool conversions.

The current versions of all preprocessing tools (ImageCalibration, StarAlignment, ImageIntegration, etc) work with FITS by default. We'll make the transition to XISF when we release the next version of the PixInsight Core application. Today we have taken a first step with a new version of the BatchPreprocessing script that uses XISF for output by default. Again, since XISF is fully compatible with FITS metadata, there should be no problems.

[Philip de Louraille]

FITS is to FORTRAN as to XISF is to C++ ?? ;-)

In any case, what the heck is a "floating point complex" format for? Very curious to learn! What do you do with that?!