PixInsight Forum

PixInsight => Release Information => Topic started by: Juan Conejero on 2014 November 30 10:13:06

Title: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 November 30 10:13:06
Hi everyone!

Today we are glad to announce a new development project: XISF. XISF stands for Extensible Image Serialization Format, and is from now on the file format used by default in PixInsight for storage and management of images and their associated data and properties. XISF effectively replaces the FITS format, which is now deprecated in PixInsight, and will be supported exclusively for compatibility with existing data and applications.

XISF is an open file format. This means that its formal specification is publicly available, it can be freely used and implemented by anyone, and is open to contributions from third parties, including all software developers involved in PixInsight development, as well as individuals and teams from other companies and groups. The official XISF development repository will be part of the official PixInsight GitHub repositories (https://github.com/PixInsight).

We have just released a new open-source XISF module for all supported platforms (FreeBSD, Linux, OS X and Windows). The XISF module is an initial implementation of the XISF version 1.0 specification. We have also released a new version 1.38 of the BatchPreprocessing script that uses XISF for all generated output by default. Both updates are now available for PixInsight Core 1.8.3.1123.


Main Format Features

The initial XISF version 1.0 specification has the following main features:

* Two storage models:

   - Monolithic model, where all image metadata, pixel sample data and image properties are stored in a single storage unit (e.g. a disk file). Monolithic XISF file names carry the .xisf suffix.

   - Distributed model, where image definitions and metadata are stored in a single header unit, and pixel sample data and image properties can be stored as separate units, including local and remote storage (for example, we can send you a XISF header that points to image contents and properties stored in a remote server). XISF header units carry the .xish file suffix, while XISF binary data blocks (known as 'XISF blobs') have the .xisb file suffix.

   The initial implementation of XISF that we have released as a PixInsight module only supports monolithic XISF files.

* Storage of multiple images, metadata and image properties.

   An XISF file can store an unlimited number of images with different geometries, color spaces, and associated metadata and properties.

* Multidimensional images.

   XISF supports arbitrary image dimensions, including one-dimensional, two-dimensional and three-dimensional images, as well as images of higher dimensions such as tesseracts and other hypercubes.

   The current XISF PixInsight module only supports two-dimensional images.

* Multiple image channels.

   XISF supports an unlimited number of image planes (not to be confused with the dimensionality of the image), such as nominal color channels and alpha channels, for each stored image.

* Sequential and random access to pixel data.

   Monolithic XISF files support sequential and random access to load individual pixel sample rows or sets of contiguous rows.

* Seven pixel sample data types supported:

   - Unsigned 8-bit integer real
   - Unsigned 16-bit integer real
   - Unsigned 32-bit integer real
   - 32-bit IEEE 754 floating point real
   - 64-bit IEEE 754 floating point real
   - 32-bit IEEE 754 floating point complex
   - 64-bit IEEE 754 floating point complex

* Seven color spaces supported:

   - Grayscale
   - RGB
   - CIE XYZ
   - CIE L*a*b*
   - CIE L*c*h*
   - HSV
   - HSI

* Support for RGB working spaces.

   XISF can serialize the parameters of colorimetrically defined RGB working spaces, which can be associated with stored images.

* Support for ICC color profiles.

   XISF can store standard ICC color profiles associated with stored images.

* Arbitrary floating point data ranges.

   Floating point images (both real and complex) can be stored using arbitrary values and ranges. However, an unambiguous specification of the numeric range to which floating point image data is referred—that is, the black and white points of the image—is mandatory. Specifying the numeric range of integer image data is optional (by default, integer images utilize the whole [0,2n-1] range, where n is the sample bit depth)..

* Data compression.

   XISF supports arbitrary data compression codecs, including lossless and lossy compression schemes.

   The current XISF PixInsight module supports zlib/defalte lossless compression.

* Digital signatures.

   XISF implements digitally signed files with standard X.509 certificates and XML signatures (http://www.w3.org/TR/xmldsig-core/). Digital signatures protect both data integrity and authenticity.

   Note: The current PixInsight XISF module still does not support digital signatures; this feature will be available in an upcoming version.

* XML file headers.

   XISF headers (in both the monolithic and distributed models) are plain text in standard XML 1.0 format encoded as UTF-8.

* Full Unicode support.

   Image metadata and textual properties fully support Unicode. XISF headers are encoded as UTF-8 data.

* Strongly-typed image properties.

   An XISF property is a data block accessible through a unique identifier (name-value pairs). Properties can be associated with images, with the whole XISF unit, or be serialized as independent elements. XISF properties are strongly-typed objects of scalar, vector and matrix types:

   Scalar property types:

      Boolean (a one-byte integer with 0 or 1 value)
      8-bit signed integer
      8-bit unsigned integer
      16-bit signed integer
      16-bit unsigned integer
      32-bit signed integer
      32-bit unsigned integer
      64-bit signed integer
      64-bit unsigned integer
      32-bit IEEE 754 floating point (float)
      64-bit IEEE 754 floating point (double)

   Vector property types:

      ByteArray - a vector of unsigned 8-bit integers
      IVector - a vector of signed 32-bit integers
      UIVector - a vector of unsigned 32-bit integers
      I64Vector - a vector of signed 64-bit integers
      UI64Vector - a vector of unsigned 64-bit integers
      F32Vector - a vector of 32-bit IEEE 754 floating point real values
      F64Vector - a vector of 64-bit IEEE 754 floating point real values
      String16 - a null-terminated vector of 16-bit Unicode code points (UTF-16)
      String8 - a null-terminated vector of 8-bit ISO/IEC-8859-1 or Unicode code points (UTF-8)

   Matrix property types:

      ByteMatrix - a matrix of unsigned 8-bit integers
      IMatrix - a matrix of signed 32-bit integers
      UIMatrix - a matrix of unsigned 32-bit integers
      I64Matrix - a matrix of signed 64-bit integers
      UI64Matrix - a matrix of unsigned 64-bit integers
      F32Matrix - a matrix of 32-bit IEEE 754 floating point real values
      F64Matrix - a matrix of 64-bit IEEE 754 floating point real values

* FITS compatibility.

   XISF is fully compatible with the FITS format. FITS header keywords can be stored in XISF files and then retrieved just as if you were working with actual FITS files. In this way existing tools and processing pipelines can work with XISF files without modification in PixInsight and other applications.


XISF Roadmap

Today we have released an initial implementation of XISF on the PixInsight platform. The full source code of this implementation is also available on our GitHub repositories (https://github.com/PixInsight/PCL/tree/master/src/modules/file-formats/XISF) under PCL license (http://pixinsight.com/license/PCL_PJSR_1.0.html).

However, this is not an official release of the XISF format, mainly because we still lack a formal definition of XISF. Along with the need of a formal definition, we've got a lot of work to do with this project. The current roadmap for the XISF project includes the following main goals:



I hope you'll enjoy this new initiative. Thank you for your continued support.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: davehudsn on 2014 November 30 13:42:25
Juan,

With all due respect, an open format generally means it is a format agreed upon by a relevant consortium and not a format derived by a company and them deemed open because it's format is declare and not proprietary.

The FITS format is the Astronomical standard and I am not aware of a deviation from that.  It concerns me that you state (imply) that you will support current FITS formats, but not future releases/standards of the FITS format.

Please accept my apologies if I am in error here, but it feels like I've made a major investment in a piece of software that is switching formats and not planning on support the community standard in the future.   Could you clarify this?


Thanks,
Dave
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: William McLaughlin on 2014 November 30 13:49:20
I would be shocked if .fits is no longer supported and suspect this will never be the case.

My guess is that this is intended to better support the amateur imaging crowd (for which .fits was
not really designed) and be offered as an additional format for capture programs to support, offering
added features for that audience. As such, it's success will be dependent on camera control software
supporting the new format so remains to be seen. Given the large percentage of imagers that use PI,
I think it has a good chance as imaging control software writers will be aware of this.

Please correct me if this is wrong.

FYI, I have posted a link to this thread over on the SGP (Sequence Generator Pro) forum...
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Starlight on 2014 November 30 14:12:15
I'm also surprised and concerned by the announcement. Fits is a universally accepted format for astronomical imaging and to produce your own format strikes of protectionism - Pixinsight is a well respected piece of software but it is not the only one - I'm sure there are a number of people that like to use Pixinsight alongside other astro software during their processing workflow.
I hope that you will continue to fully support the fits format and allow people to choose what format they use at all stages.

Don

ps It seems a very strange business plan to make your product less universally acceptable
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: pmesquita on 2014 November 30 14:17:49
same here. I've been learning the long learning curve of imaging and of the PI tools and only recently have I standardized my .fits work. I hope I keep getting .fits and an option with PI.

best regards
Paulo Mesquita
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: btracy on 2014 November 30 14:28:55
Perhaps Juan can take the time to explain just what the implications are for the users.  I, for one, do not have the foggiest idea what the XISF format means to me but I do know (as a retired software developer) that Juan and crew wouldn't be investing a lot of programmer time to develop this without good reason.

BobT
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: rga218 on 2014 November 30 14:31:35
In this particular case I guess I wish that instead of developing a new file format PixInsight had embraced Multi-Extension FITS (MEF) format to add much of the desired functionality in the XISF format.

http://www.stsci.edu/hst/HST_overview/documents/datahandbook/intro_ch23.html (http://www.stsci.edu/hst/HST_overview/documents/datahandbook/intro_ch23.html)

Lack of support for Multi-Extension FITS files (the standard adopted by most large telescopes) has been the #2 issue in my attempt to get more of my professional astronomer colleagues to use PixInsight (#1 has been missing documentation). In the professional community FITS (and its gradual evolution to embrace extensions and tables) is generally viewed as a universal standard that all software really must adhere to in order to gain acceptance. Multi-Extension FITS is backwards compatible with vanilla FITS (at least when using CFITSIO, though I guess it might be quite a lot of work to implement in PixInsight as I don't believe you're using CFITSIO). And Multi-Extension FITS is used as the standard data format for pretty much every instrument on every major optical-IR telescope I'm aware of (HST, Gemini, VLT, CFHT, etc.) and many radio telescopes. Most aspects of the new XISF format are already implementable using multi-extension FITS, and in my opinion the exceptions (e.g. unicode, signatures, and the distributed access model) probably would be viewed by most of my colleagues as insufficiently important to require a new standard. If PixInsight used Multi-Extension FITS I believe you'd get most of the capability of the new XISF format you're implementing, retain compatibility with old FITS files, and open up the capability of having amateurs using PixInsight easily use data from the archives of professional telescopes. 

That said, I am probably being a bit parochial here, and I do appreciate the desire to innovate! The core audience here are amateur astronomers, who probably have very different priorities to my own. In any case, PixInsight is sufficiently wonderful that I'll stick with it regardless of the file format whenever I can, so please take the comments above as an expression of my enthusiasm for the software, and a desire to be able to use it with even more telescopes and datasets.

Bob

P.S. PixInsight was used to reduce much of the data for this recent paper: http://arxiv.org/abs/1410.8141 (http://arxiv.org/abs/1410.8141)
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: jkmorse on 2014 November 30 14:38:50
Juan,

I am more interested in the practical application of using the XISF format.  Since most image capture software uses FITS, are you recommending that we do a format conversion before doing any processing steps.  If so, do you recommend starting with unsigned 16 bit integers or jumping right to 32 bit floating point format.

Also, are all the other PI processes able to work with XISF formatted images or will we need to wait for tool by tool conversions.

Thanks,

Jim
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: kayronjm on 2014 November 30 15:00:33
I too am not keen on having FITS replaced by the new format, or any format. If the team would like to develop this new format to present new workflow features, then fair enough, but PixInsight should maintain 100% FITS compatibility, now and in all future releases.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 November 30 15:48:57
With all due respect, an open format generally means it is a format agreed upon by a relevant consortium and not a format derived by a company and them deemed open because it's format is declare and not proprietary.

If the Linux Information Project can be considered as a reliable source:

http://www.linfo.org/free_file_format.html

So according to the above definition, XISF is actually not only an open format, but a free format:

A free file format is a file format that is both (1) published so that anyone can read and study it in its entirety and (2) not encumbered by any copyrights, patents, trademarks or other restrictions so that anyone may use it at no monetary cost for any desired purpose. Such specifications are usually maintained by a non-commercial standards organization.

XISF complies with requirements (1) and (2) of the above definition. The only bit that we don't have here is the "maintained by a non-commercial standards organization" for obvious reasons. But it says "usually", not "necessarily", and as I have said in the starting post, we'd be glad if XISF were maintained and developed by a group of developers beyond our team. But let us define the initial formal specification first, and write a first reference implementation. Then XISF will be all yours.

Quote
The FITS format is the Astronomical standard and I am not aware of a deviation from that.  It concerns me that you state (imply) that you will support current FITS formats, but not future releases/standards of the FITS format.

Don't be concerned at all. We will continue supporting FITS and future versions of the FITS standard. In fact, we have been implementing the FITS standard for a long time, when other competing applications have not (and still don't, in some cases).

XISF will be the native file format of the PixInsight platform when we release the next version of the PixInsight Core application, and as such it will replace FITS by default. But default option does not mean only option. Of course you'll be free to select the FITS format as default if you want. And you can always save an XISF file in FITS format. Because XISF stores FITS header keywords since the beginning of the process, no FITS metadata will be lost since the moment you load your raw FITS frames until the final processed product.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: jkmorse on 2014 November 30 15:52:12
Personally I applaud the effort to keep improving.  FITS may be the standard but that does not mean it's ideal.  Just look at the number of posts related to FITS incompatibilities to understand that sticking with the old method, just because it is old, if hardly the way to advance the efforts of Juan and his team to continually bring us new and better image processing tools.  Even if it only makes the FITS incompatibility problems a thing of the past, it will be well worth it.

For what it's worth.

Jim

Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 November 30 16:10:49
I would be shocked if .fits is no longer supported and suspect this will never be the case.

Rest assured about that. We'll continue supporting FITS in the foreseeable future.

Quote
My guess is that this is intended to better support the amateur imaging crowd

IMHO, XISF could support the professional crowd better than FITS. Actually, we started the XISF project some time ago partly because FITS has a number of important limitations and problems (for us) that we have been suffering for a long time.

Quote
...it's success will be dependent on camera control software supporting the new format so remains to be seen. Given the large percentage of imagers that use PI, I think it has a good chance as imaging control software writers will be aware of this.

That's indeed a realistic view of the topic. Our main intention with XISF is to improve the experience of our users on the PixInsight platform. In this sense XISF should be considered as the native file format of PixInsight. We will make efforts to convince the developers of other image acquisition applications to implement XISF, and will be glad to assist them in their implementations. The availability of a reference implementation is crucial to achieve this goal.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: edd on 2014 November 30 16:24:43
As a professional astronomer too I welcome this. FITS is versatile and well supported by some languages, but has some nasty features that make it a pain to use as well as one might like. Headers in particular are nasty. Having partially implemented FITS parsers from scratch in the last it's been a less than wonderful experience.

I've considered writing my own specification with, for example, XML headers for such a file format but never got around to it. I'd hope this starts people thinking about using it in other situations too and push this format further forward.

Clearly FITS support isn't going away and a converter between the two is essential for now but I think a new more modern format with the same capabilities makes it easier to develop new code, and apply other tools without being so dependent on cfitsio and the like and to cover the core functions.

Good work and a big thumbs up from me.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 November 30 16:28:03
Hi Jim,

Since most image capture software uses FITS, are you recommending that we do a format conversion before doing any processing steps.

Not really. Your raw data stored as FITS files will continue working as input data for the ImageCalibration tool for example. IC will store calibrated images as XISF files by default, and from that point subsequent preprocessing stages will work with XISF. Actually, you can do this right now by just replacing the default ".fit" with ".xisf" in the output extension parameter of IC. In the next version the default output extension will be ".xisf", that's all. Nothing will be lost because XISF can store all FITS header keywords and retrieve them transparently, just as if it was FITS.

Quote
do you recommend starting with unsigned 16 bit integers or jumping right to 32 bit floating point format.

Raw CCD data are unsigned 16-bit integers, and no file format specification can change this. Raw CCD frames should *always* be stored in unsigned 16-bit format, no matter the formats and tools used.

Quote
Also, are all the other PI processes able to work with XISF formatted images or will we need to wait for tool by tool conversions.

The current versions of all preprocessing tools (ImageCalibration, StarAlignment, ImageIntegration, etc) work with FITS by default. We'll make the transition to XISF when we release the next version of the PixInsight Core application. Today we have taken a first step with a new version of the BatchPreprocessing script that uses XISF for output by default. Again, since XISF is fully compatible with FITS metadata, there should be no problems.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Philip de Louraille on 2014 November 30 16:31:56
FITS is to FORTRAN as to XISF is to C++ ?? ;-)

In any case, what the heck is a "floating point complex" format for? Very curious to learn! What do you do with that?!
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: rga218 on 2014 November 30 16:49:01
Philip: Storing images as complex numbers is useful in a number of domains, most notably for polarimetry and for radio astronomy (where you need to capture phase information).
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Carlos Milovic on 2014 November 30 17:03:07
Phillip,
Complex samples are those that have real and imaginary components. This sample format is used a lot when working with fourier transforms, since the fourier transform of a real image is a complex image. Also complex samples are used in the context of biomedical imaging. It's more important use is in Magnetic Resonance Imaging.

So, if anyone is working with theyr images in the fourier domain, or planning to do biomedical imaging, this is good news.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 November 30 17:08:53
Hi Bob,

Lack of support for Multi-Extension FITS files (the standard adopted by most large telescopes) has been the #2 issue in my attempt to get more of my professional astronomer colleagues to use PixInsight

Our implementation of the FITS format has supported multi-extension FITS files for a long time. At least since 2006 if I can remember well; probably even 2005. Just to put a well-known example for everyone, if you use the BatchPreprocessing script and select ".fit" as the output file suffix, all generated master files will be multi-extension FITS files that you can use in PixInsight without problems. Each of these masters store three or four images in a single FITS file: the integrated master frame (science, bias, dark, or flat) and the low and high rejection map images, plus the slope map image if you select linear fit clipping. It has always worked that way.

Another example: we use FITS extensions to store ICC color profiles, which are absolutely essential in PixInsight—well, they are essential for any image processing application that works with real-world images.

It is true that we could develop a better graphical interface to manage multiple images—it is also true that other tools commonly used by professionals lack any graphical interface at all—, but multi-extension FITS are definitely supported in PixInsight.

Quote
(#1 has been missing documentation).

Touchée. Believe it or not, we really do what we can with the resources that we have.

Quote
In the professional community FITS (and its gradual evolution to embrace extensions and tables) is generally viewed as a universal standard that all software really must adhere to in order to gain acceptance. Multi-Extension FITS is backwards compatible with vanilla FITS (at least when using CFITSIO, though I guess it might be quite a lot of work to implement in PixInsight as I don't believe you're using CFITSIO).

We use CFITSIO in our FITS support module. This is true since the first versions of PixInsight LE that we released back in 2003.

Quote
Most aspects of the new XISF format are already implementable using multi-extension FITS, and in my opinion the exceptions (e.g. unicode, signatures, and the distributed access model) probably would be viewed by most of my colleagues as insufficiently important to require a new standard.

I really don't want/need to criticize FITS here—if you have read me on this forum, you probably know some of the problems that we have been forced to face as a result of FITS limitations. We prefer to look forward for efficient solutions now, and that's the main reason for starting this project. Now that you mention it, the lack of Unicode support, distributed storage, unsigned integer pixel data (the BZERO/BSCALE trick is, well, a trick), a formal description of floating point pixel sample ranges, color spaces, color profiles, image thumbnails, strongly typed properties, data compression, along with the obsolete file header organization, are important for us.

Quote
...I do appreciate the desire to innovate!

Thank you so much for that!

Quote
P.S. PixInsight was used to reduce much of the data for this recent paper: http://arxiv.org/abs/1410.8141 (http://arxiv.org/abs/1410.8141)

And I am very proud of that ;)
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: jerryyyyy on 2014 November 30 17:14:30
All this scares novice user like myself who have barely come to understand FITs.  I have a lot of $ invested in software whose final pathway is FITs... e.g. Maxim... which is bad enough.  I do not think I would have come to use PI if it were using a format I had not heard of [purchase is nonstarter]. 
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Philip de Louraille on 2014 November 30 17:33:13
Roberto & Carlos: Thanks for the examples. I hadn't thought about radiotelescopy (and MRI) and the needs to store a phase.
Won't be long before PI can store files with a tensor-like format. ;-)

Jerryyyyy: no worries. Just because PI can work with formats you don't want to use does not mean you wasted your money. You can stay with what you know and use.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: rga218 on 2014 November 30 18:09:10
Hola Juan,

Hi Bob,

Our implementation of the FITS format has supported multi-extension FITS files for a long time. At least since 2006 if I can remember well; probably even 2005. Just to put a well-known example for everyone, if you use the BatchPreprocessing script and select ".fit" as the output file suffix, all generated master files will be multi-extension FITS files that you can use in PixInsight without problems. Each of these masters store three or four images in a single FITS file: the integrated master frame (science, bias, dark, or flat) and the low and high rejection map images, plus the slope map image if you select linear fit clipping. It has always worked that way.


Great to learn you guys are using CFITSIO and I'm sorry I was being unclear about my need to access MEF files! I know you can open an MEF image in PixInsight, but what I mean is: can I extract data in a particular extension from the console? I'm probably still being unclear, so for example, let's say I have a file foo.fits and it has 5 channels and I want to access the 3rd extension, which is the science channel, from the process console. The FITSIO syntax to access this data is foo.fits[SCI] or foo.fits[3] or foo.fits [SCI,3]. But if I try:

"open foo.fits[3]"

In the PixInsight process console this doesn't work. Am I doing something stupid? (Probably).

Quote

Touchée. Believe it or not, we really do what we can with the resources that we have.


Oh, I believe it! You've accomplished an incredible amount with a really small team, and when it comes to providing documentation you do better than me... you should see what undocumented stuff I have to foist on my collaborators just because there are not enough hours in the day sometimes.

Quote
I really don't want/need to criticize FITS here—if you have read me on this forum, you probably know some of the problems that we have been forced to face as a result of FITS limitations. We prefer to look forward for efficient solutions now, and that's the main reason for starting this project. Now that you mention it, the lack of Unicode support, distributed storage, unsigned integer pixel data (the BZERO/BSCALE trick is, well, a trick), a formal description of floating point pixel sample ranges, color spaces, color profiles, image thumbnails, strongly typed properties, data compression, along with the obsolete file header organization, are important for us.

I have total respect for that. The thing about PixInsight is that it is true to itself... go for it! By any rational standard most professional software is much quirkier than PixInsight, though of course those quirks are well documented, standardized, etc etc. For example, the lack of a formal description of floating point pixel ranges is a non-issue in all professional software because everybody understands what is going on (in fact, even though it's pretty rational, PixInsight's scaling to the range [0,1] is #3 on the list of things people complain about when I try to sell them on PixInsight).

Anyway, when it comes to file formats, FITS in particular is just set in stone and in the interests of consistency and interchange with decades of data in the archives my professional colleagues and I just have to put up with any deficiencies. Just read the angst that went into getting everybody to agree on standardizing on FITS in the first couple of sections of the PDF maintained by the IAU working group:

http://fits.gsfc.nasa.gov/standard30/fits_standard30aa.pdf (http://fits.gsfc.nasa.gov/standard30/fits_standard30aa.pdf)

Of course, you aren't aiming PixInsight at professional astronomers as your main market, so you needn't feel hidebound to adhere to things (like common file formats for even intermediate data exchange). IMHO moving away from FITS as a default will limit your reach in the professional community, but everything's a trade-off. If you feel a need to ditch FITS as the default file format because you find it limits your ability to innovate in other ways, well, that's clearly a reasonable choice to make. If I were writing the software and not you I wouldn't do that, but you know your market and I don't, and if I wrote PixInsight it would be a lot less interesting and compelling than you've made it.

Bob
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 November 30 18:10:35
Quote
Jerryyyyy: no worries. Just because PI can work with formats you don't want to use does not mean you wasted your money. You can stay with what you know and use.

Absolutely. The XISF project is here to make everything more efficient and versatile. You can always decide to write your data in other formats, including FITS, TIFF, PNG, JPEG and others, which are supported and will be supported in PixInsight. XISF is from now on our native format, and you probably will like it when you know what it's capable of, but it won't limit your use of our software; it will do just the opposite to that.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 November 30 18:47:49
...can I extract data in a particular extension from the console?

That can't be done, mainly because our FITS support module does not expose that functionality of CFITSIO. However, you can access any FITS extension very easily with a relatively simple JavaScript script (about 50 lines I think). Scripts can be invoked from the console.

Quote
(in fact, even though it's pretty rational, PixInsight's scaling to the range [0,1] is #3 on the list of things people complain about when I try to sell them on PixInsight).

The problem here is the difference between images and data. As we understand the concept, an image is a data structure whose main purpose is to be visually representable. As a consequence of its visual role, an image is subject to a number of constraints such as color spaces, color transformations, representable ranges, etc, that in turn require auxiliary data structures and formal definitions. Generic data structures don't have any of these constraints a priori, so there's generally no need to know the limits of the representable range and similar properties. One of the most critical problems that we have with FITS is precisely here: it is a format designed to store generic data, but quite ironically, it does not help at all to transport images—at least not for the concept of image that an image processing application needs to work with real-world images. We have addressed this problem with XISF, where images and data objects (which we call properties) are formally supported by different structures, and images have all the associated elements required to represent and use them consistently.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Jay W. Butler on 2014 November 30 18:56:31
I must tell you that in my case, this new format is a solution looking for a problem. I am still learning to use PI, which has been a very steep uphill slog. I don't want PI to start saving my images in a new file format that I am unfamiliar with and that may have consequences or properties that I don't know to handle.
What advantage, if any, does this new file format provide for an intermediately skilled amateur who just wants to process LRGB images with occasional narrowband additions?
Can I just continue to save my processed subs as FITS files? Am I going to have to make that election every time, or can I set it as a default on my system?
Jay Butler
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 November 30 19:01:23
Hi Jay,

The default output file format can be set as a global preferences option, so don't worry about that. As for advantages and features, please read the posts above.

Quote
this new format is a solution looking for a problem.

It is actually a solution to a number of problems.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: rga218 on 2014 November 30 19:32:10
We have addressed this problem with XISF, where images and data objects (which we call properties) are formally supported by different structures, and images have all the associated elements required to represent and use them consistently.

It sounds like PixInsight's approach will resemble that of Mathematica and MATLAB, which treat images as distinct from other forms of 2D data. I certainly understand your point of view and do see the logic of it, and furthermore see how what you're doing with XISF will be helpful if it is a component in this plan.

If I understand your description properly, then at least for my sorts of data flows 90% of the time I'm going to manipulating 'properties' and not 'images'. I'm actually totally fine with that.... seems like a good idea. Most of the time I'm manipulating generic 2D data that has proper units like mJy/pixel or electron/pix/sec, and only as the final step when I need to generate the final picture is it helpful for me to think about how to rescale my property into an 'image' in a color space with channels made up of [0..255] RGB or grayscale in the range [0..1] via screen transfer function or whatever.

Bottom line: If I can read a FITS image into a property, and not have to rescale it when read it in, and then I can keep it from being rescaled throughout all the data flow (calibration and other manipulations such as stacking) and only have to convert it to an image at the end of a long chain of analysis, that would make me very happy! I guess I'm hoping most of the (relevant) modules (things like PixelMath, Resample, DynamicCrop, DynamicPSF, ImageIntegration) will work on properties as well as on images, and then as the last step I hit my property data with a ScreenTransferFunction to turn it into an image.  If that is the case then I'm an XISF believer.

If what I wrote above makes sense, perhaps you might considering renaming ImageIntegration to be PropertyIntegration or even ImagePropertyIntegration? (Or maybe not... perhaps that would break too many things).

Bob
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: jkmorse on 2014 November 30 19:38:35
Jay,

+1 to Juan's response.  FITS incompatibilities have been an issue for PI and other software for years and is a continuing issue due to the lack of a set convention for using FITS in various pieces of software.  This is definitely a problem in need of a solution and not the other way around.  Take a look at this post to see some of what I mean:

http://pixinsight.com/forum/index.php?topic=4880.0

This is only one of dozens of posts on the same subject.

Juan, thanks for the clarifications on the practical side of this update and for continuing to push the envelope and taking us out of our comfort zone to drive better processing results.

Best,

Jim
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Geoff on 2014 November 30 20:10:08

Juan, thanks for the clarifications on the practical side of this update and for continuing to push the envelope and taking us out of our comfort zone to drive better processing results.

Best,

Jim
+1

Henry Ford, on innovation, once said “If I had asked my customers what they wanted, they would have said ‘faster horses’”.
Geoff
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Joe Perulero on 2014 December 01 00:22:29
Well done Juan , finally a brave move which I believe will pay dividends for PI later on ! :)
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: edd on 2014 December 01 09:19:03
Does XISF have a documented specification yet or is it only by a reference implementation so far? I recognise these are early days if a document is planned but isn't ready yet.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 December 01 09:58:38
Quote
Does XISF have a documented specification yet or is it only by a reference implementation so far?

I am working right now on a formal definition document.

Just as an informal example, this is the XISF header for a compressed raw DSLR image:

<?xml version="1.0" encoding="UTF-8"?>
<!--
Extensible Image Serialization Format - XISF version 1.0
Created with PixInsight - http://pixinsight.com/
-->
<xisf version="1.0" xmlns="http://www.pixinsight.com" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.pixinsight.com http://pixinsight.com/xsd/xisf-1.0.xsd">
   <Image geometry="5796:3870:1" sampleFormat="UInt16" colorSpace="Gray" cfaType="RGGB" compression="zlib:44861040" location="attachment:1889:35792262">
      <FITSKeyword name="COMMENT" value="" comment="Decoded with PixInsight 01.08.03.1123"/>
      <FITSKeyword name="COMMENT" value="" comment="Decoded with DSLR_RAW module 01.02.00.0232"/>
      <FITSKeyword name="COMMENT" value="" comment="Decoded with dcraw version 9.22"/>
      <FITSKeyword name="INSTRUME" value="Canon EOS 5D Mark III" comment="Camera model"/>
      <FITSKeyword name="DATE-OBS" value="2012-04-11T16:15:37" comment="Camera timestamp"/>
      <FITSKeyword name="EXPTIME" value="0.000977" comment="Exposure time in seconds"/>
      <FITSKeyword name="ISOSPEED" value="400" comment="ISO speed as specified in ISO 12232"/>
      <FITSKeyword name="FOCALLEN" value="100.00" comment="Focal length in mm"/>
      <FITSKeyword name="APTDIA" value="12.50" comment="Aperture diameter in mm"/>
      <Resolution horizontal="100" vertical="100" unit="inch"/>
   </Image>
   <Metadata>
      <Property id="CreationTime" type="String8">2014-12-01T17:41:42Z</Property>
      <Property id="CreatorApplication" type="String16">PixInsight 01.08.03.1123</Property>
      <Property id="CreatorModule" type="String16">XISF module version 01.00.00.0023</Property>
      <Property id="CreatorOS" type="String16">Linux</Property>
      <Property id="CompressionMethod" type="String8">zlib</Property>
      <Property id="CompressionLevel" type="Int32" value="6"/>
   </Metadata>
</xisf>


This is for a RGB color image in 32-bit floating point format:

<?xml version="1.0" encoding="UTF-8"?>
<!--
Extensible Image Serialization Format - XISF version 1.0
Created with PixInsight - http://pixinsight.com/
-->
<xisf version="1.0" xmlns="http://www.pixinsight.com" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.pixinsight.com http://pixinsight.com/xsd/xisf-1.0.xsd">
   <Image geometry="960:540:3" sampleFormat="Float32" bounds="0:1" colorSpace="RGB" location="attachment:4096:6220800">
      <Resolution horizontal="72" vertical="72" unit="inch"/>
      <ICCProfile location="inline:base64">AAAH8HNjcnMCEAAAbW50clJHQiBYWVogB9EABAAMABMAAgAOYWNzcE1TRlQBAAAAbm9uZW5vbmUAAAAAAAAAAAAAAAAAAPbWAAEAAAAA0y1zY3JzAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKZGVzYwAAAPwAAABfY3BydAAAAVwAAAAJd3RwdAAAAWgAAAAUYmtwdAAAAXwAAAAUclhZWgAAAZAAAAAUZ1hZWgAAAaQAAAAUYlhZWgAAAbgAAAAUclRSQwAAAcwAAAIMZ1RSQwAAA9gAAAIMYlRSQwAABeQAAAIMZGVzYwAAAAAAAAAFc1JHQgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAdGV4dAAAAAAAAAAAWFlaIAAAAAAAAPbWAAEAAAAA0y1YWVogAAAAAAAAAAAAAAAAAAAAAFhZWiAAAAAAAABpkwAANnAAAATzWFlaIAAAAAAAAFuLAAC3FgAAHoRYWVogAAAAAAAALjQAABJ8AADzV2N1cnYAAAAAAAABAAAAAAAAAgAEAAcACwARABgAIAAqADUAQQBPAF4AbwCBAJQAqQDAANgA8gEOASsBSgFqAYwBsAHVAfwCJQJPAnsCqQLZAwsDPgNzA6oD4wQdBFkElwTXBRkFXQWjBeoGMwZ/BswHGwdsB78IFAhrCMMJHgl7CdkKOgqdCwELaAvQDDsMqA0WDYcN+g5uDuUPXg/ZEFYQ1RFWEdoSXxLmE3AT+xSJFRkVqxY/FtUXbhgIGKUZRBnlGogbLRvUHH4dKh3YHogfOh/vIKYhXyIaItcjlyRZJR0l4yasJ3YoQykTKeQquCuOLGYtQS4eLv0v3jDCMagykDN7NGg1VzZINzw4MjkrOiU7IjwiPSQ+KD8uQDdBQkJPQ19EcUWGRp1HtkjSSfBLEEwzTVhOf0+pUNZSBFM1VGlVn1bXWBJZT1qOW9BdFV5cX6Vg8WI/Y49k4mY4Z5Bo6mpHa6ZtCG5sb9NxPHKndBV1hnb5eG555nthfN5+XX/fgWSC6oR0hgCHjokfirOMSY3hj3yRGpK6lF2WApepmVSbAJywnmKgFqHNo4alQqcBqMKqhqxMrhWv4bGvs3+1UrcouQC627y5vpnAe8JhxEnGM8ggyhDMAs33z+7R6NPl1eTX5tnr2/Ld/OAI4hfkKeY96FTqbuyK7qnwyvLu9RX3P/lr+5r9y///Y3VydgAAAAAAAAEAAAAAAAACAAQABwALABEAGAAgACoANQBBAE8AXgBvAIEAlACpAMAA2ADyAQ4BKwFKAWoBjAGwAdUB/AIlAk8CewKpAtkDCwM+A3MDqgPjBB0EWQSXBNcFGQVdBaMF6gYzBn8GzAcbB2wHvwgUCGsIwwkeCXsJ2Qo6Cp0LAQtoC9AMOwyoDRYNhw36Dm4O5Q9eD9kQVhDVEVYR2hJfEuYTcBP7FIkVGRWrFj8W1RduGAgYpRlEGeUaiBstG9Qcfh0qHdgeiB86H+8gpiFfIhoi1yOXJFklHSXjJqwndihDKRMp5Cq4K44sZi1BLh4u/S/eMMIxqDKQM3s0aDVXNkg3PDgyOSs6JTsiPCI9JD4oPy5AN0FCQk9DX0RxRYZGnUe2SNJJ8EsQTDNNWE5/T6lQ1lIEUzVUaVWfVtdYEllPWo5b0F0VXlxfpWDxYj9jj2TiZjhnkGjqakdrpm0Ibmxv03E8cqd0FXWGdvl4bnnme2F83n5df9+BZILqhHSGAIeOiR+Ks4xJjeGPfJEakrqUXZYCl6mZVJsAnLCeYqAWoc2jhqVCpwGowqqGrEyuFa/hsa+zf7VStyi5ALrbvLm+mcB7wmHEScYzyCDKEMwCzffP7tHo0+XV5Nfm2evb8t384AjiF+Qp5j3oVOpu7IruqfDK8u71Ffc/+Wv7mv3L//9jdXJ2AAAAAAAAAQAAAAAAAAIABAAHAAsAEQAYACAAKgA1AEEATwBeAG8AgQCUAKkAwADYAPIBDgErAUoBagGMAbAB1QH8AiUCTwJ7AqkC2QMLAz4DcwOqA+MEHQRZBJcE1wUZBV0FowXqBjMGfwbMBxsHbAe/CBQIawjDCR4JewnZCjoKnQsBC2gL0Aw7DKgNFg2HDfoObg7lD14P2RBWENURVhHaEl8S5hNwE/sUiRUZFasWPxbVF24YCBilGUQZ5RqIGy0b1Bx+HSod2B6IHzof7yCmIV8iGiLXI5ckWSUdJeMmrCd2KEMpEynkKrgrjixmLUEuHi79L94wwjGoMpAzezRoNVc2SDc8ODI5KzolOyI8Ij0kPig/LkA3QUJCT0NfRHFFhkadR7ZI0knwSxBMM01YTn9PqVDWUgRTNVRpVZ9W11gSWU9ajlvQXRVeXF+lYPFiP2OPZOJmOGeQaOpqR2umbQhubG/TcTxyp3QVdYZ2+XhueeZ7YXzefl1/34FkguqEdIYAh46JH4qzjEmN4Y98kRqSupRdlgKXqZlUmwCcsJ5ioBahzaOGpUKnAajCqoasTK4Vr+Gxr7N/tVK3KLkAutu8ub6ZwHvCYcRJxjPIIMoQzALN98/u0ejT5dXk1+bZ69vy3fzgCOIX5CnmPehU6m7siu6p8Mry7vUV9z/5a/ua/cv//w==</ICCProfile>
   </Image>
   <Metadata>
      <Property id="CreationTime" type="String8">2014-12-01T17:53:56Z</Property>
      <Property id="CreatorApplication" type="String16">PixInsight 01.08.03.1123</Property>
      <Property id="CreatorModule" type="String16">XISF module version 01.00.00.0023</Property>
      <Property id="CreatorOS" type="String16">Linux</Property>
      <Property id="BlockAlignmentSize" type="UInt16" value="4096"/>
      <Property id="MaxInlineBlockSize" type="UInt16" value="3072"/>
   </Metadata>
</xisf>


The XSD definition document still has to be created. After the XML header the data blocks are stored in the the same file. Just to give you an idea of how XISF files look like.

Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 December 01 10:11:17
And this one is for a tiny RGB image of 6x6 pixels, which is completely defined by the XISF header (note the embedded Data element, where the pixel data are encoded as base64). In this case there are no attached data blocks in the XISF file.

<?xml version="1.0" encoding="UTF-8"?>
<!--
Extensible Image Serialization Format - XISF version 1.0
Created with PixInsight - http://pixinsight.com/
-->
<xisf version="1.0" xmlns="http://www.pixinsight.com" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.pixinsight.com http://pixinsight.com/xsd/xisf-1.0.xsd">
   <Image geometry="6:6:3" sampleFormat="UInt8" colorSpace="RGB" location="embedded">
      <Data encoding="base64">AAAAAP8A/wD/AAAAAAAAAP8AAP8AAAAAAAAA/wD/AP8AAAAA/wD//wD/AP8AAP8A/wD//wD//wD//wD/AP8AAP8A/wD//wD/AP8AAAAAAAAA/wD/AP8AAAAAAAAAAP8A/wD/AAAAAAAAAP8A</Data>
      <Resolution horizontal="72" vertical="72" unit="inch"/>
   </Image>
   <Metadata>
      <Property id="CreationTime" type="String8">2014-12-01T18:07:54Z</Property>
      <Property id="CreatorApplication" type="String16">PixInsight 01.08.03.1123</Property>
      <Property id="CreatorModule" type="String16">XISF module version 01.00.00.0023</Property>
      <Property id="CreatorOS" type="String16">Linux</Property>
      <Property id="BlockAlignmentSize" type="UInt16" value="4096"/>
      <Property id="MaxInlineBlockSize" type="UInt16" value="3072"/>
   </Metadata>
</xisf>
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: goofisd on 2014 December 01 12:59:20
Can someone explain what this new format means for me in simple, plain language?

I have a saying, "I'm a lot dumber than I look" .... help me understand. Why do I care about this format?
What does it do for me?

I'm an amateur imager, using a CCD and pixinsight. Will this change my workflow, or the processes I use?
If your explanation includes "complex numbers" or "plane vectors" ... you've lost me. I'm an old man with either too much whiskey in my blood, or not enough ... will this change matter to me?
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: jcinpv on 2014 December 01 13:41:41
After reading the brief specs on XISF, it appears to be an upward growth of the FITS format, meaning that while it continues to contain all that FITS has and does, XISF offers a wide spectrum of features that FITS can't support. To me, that is a step toward an area of graphics manipulation and presentation far beyond what FITS can do.

By declaring it as public domain, this makes it possible to be extended by others in ways not foreseen by the authors. That, too, is a good thing.

FITS has not been left behind, it is being cradled into a new realm.

So, loosely speaking, XISF is to FITS what zip is to gzip. I hope the analogy can be seen.

John C.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 December 01 13:52:23
Will this change my workflow, or the processes I use?

Hopefully, nothing will change in this regard. We have made efforts to make XISF compatible with all existing data and tools, so that the new format integrates seamlessly with the current PixInsight platform. Basically, you don't have to change anything. If you want to export a XISF image to a different format such as FITS, TIFF, JPEG, PNG, etc, just open the .xisf file and save it in the desired format. It's just that simple.

The benefits of the new format will be more subtle, and will be arriving gradually. We'll have better image and data management resources. Developers will enjoy a more robust and versatile platform, and this will allow for better and more sophisticated image analysis and processing tools in the medium-long term. I believe we are making a step in the right direction with this initiative.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: bitli on 2014 December 01 14:02:31
+1 for rga128 on the need  to make some processing in other units than proportion of white.  I want to  be able  to consider data points as measurements. Currently the limitations in this area come from PI, not from FITS.  I hope that this will be solved as a part of this effort, including for data imported from fits.

I think that the original announcement of the new format was not very well expressed, the latest announcement is better.  I wish you all the best for your effort, but I strongly opposed (for what it is worth) to making a new unproven, undocumented,  unique format the default used by many unsuspecting users.

This must be an opt-in, not an opt-out, for the time being.

juan, this is not a critic of your work or proposed format, I am very interested in it. But my experience of working with occasional or less technical users or users not fluent in English or not having the time to rebuild the documentation from hundredth of posts is that they will be very confused by this situation. Whathever your technical expertise, effort and dedication, there will be issues in changing format, and this must be solved will the help of willing users.

--bitli

Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 December 01 15:05:05
Quote
the need  to make some processing in other units than proportion of white.
Quote
I want to  be able  to consider data points as measurements.

There is a conceptual mistake in these requests. The graphical interface of PixInsight has been designed and implemented to work with images, not with generic data. What you are asking for simply falls out of the scope of the current PixInsight GUI.

You can work with data structures and types very different from images in PixInsight. We are doing this all the time everywhere. But not on the graphical user interface directly, because the GUI has been designed to work with images consistently and efficiently, not with arbitrary data. You can work with floating point images outside the [0,1] range without any problems; in fact, most image processing tools and scripts necessarily work with other data ranges internally (quick example: wavelet layers contain positive and negative real values with zero mean). However, once you leave a floating point image on a visible image window, it *must* be in [0,1] because the GUI has been designed and optimized to work with this range. These are the rules of the game and they work very well to generate visual representations.

Now if you want to visualize non-image data, you have to write a specific tool. For example, suppose that you want to represent a data matrix that contains temperature values from -50C to +50C. You can do this easily by adapting the data to a visually representable range, either [0,1] if the visual data are floating point numbers, or [0,2n-1] for n-bit integers. The conversion is just a linear scaling of the data from [-50,+50] to the appropriate representable range. For example, we do this to represent wavelet layers and the components of Fourier transforms as images. Once the data are being visualized, you can design a dedicated readout tool to measure temperatures instead of pixel sample values, either by reading the original data, or by doing the inverse conversion.

That said, we have plans to support other objects besides images directly in the GUI. However, don't expect this before version 2.0, and we still have a *lot* of work to do before that,

Quote
I think that the original announcement of the new format was not very well expressed, the latest announcement is better.

I agree. I will try to be less harsh the next time.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Geoff on 2014 December 01 16:29:00

You can work with data structures and types very different from images in PixInsight. We are doing this all the time everywhere. But not on the graphical user interface directly, because the GUI has been designed to work with images consistently and efficiently, not with arbitrary data. You can work with floating point images outside the [0,1] range without any problems; in fact, most image processing tools and scripts necessarily work with other data ranges internally (quick example: wavelet layers contain positive and negative real values with zero mean). However, once you leave a floating point image on a visible image window, it *must* be in [0,1] because the GUI has been designed and optimized to work with this range. These are the rules of the game and they work very well to generate visual representations.

Now if you want to visualize non-image data, you have to write a specific tool. For example, suppose that you want to represent a data matrix that contains temperature values from -50C to +50C. You can do this easily by adapting the data to a visually representable range, either [0,1] if the visual data are floating point numbers, or [0,2n-1] for n-bit integers. The conversion is just a linear scaling of the data from [-50,+50] to the appropriate representable range. For example, we do this to represent wavelet layers and the components of Fourier transforms as images. Once the data are being visualized, you can design a dedicated readout tool to measure temperatures instead of pixel sample values, either by reading the original data, or by doing the inverse conversion.

This gives us some inkling of the power of the changes, but it also points to the need of lots of lucid examples to help most(?)  of us to understand and use the new resource.
Geoff
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Jay W. Butler on 2014 December 01 17:39:26
Juan,

Just as I feared would happen after downloading and installing the updates relating to the new XISF file format and related batch pre-preprocessing script, I now have a system that will not process my .fits images. Thank you very much for this gratuitous solution to a problem that I at least did not know that I had. As I said, for me it is a "solution" in search of a problem; and all you have done for me is create a previously non-existent problem, which I will now describe.

As you instructed, I went into Global Preferences and set the default file extension to “.fits”. In the new batch pre-processing script the output file suffix defaults to “.xsif” and I changed that as well to “.fits”. I then loaded into the batch pre-processing script interface master bias and dark files in .fits format from my library and the .fits subs captured by Software Bisque’s TheSkyX camera operating software during my last image capture session in the observatory. I then ran the script (with the settings I had previously used hundreds of times without any problem). As I got to the registration stage of the script, I received the following error message on the Process Console and the batch pre-processing script hung up:

PCL FITS Format Support: Unable to open FITS file:
D:/STT-8300M Images/TARGETS/NGC470/20141201/calibrated/light/cosmetized/NGC470.Luminance.Light.00000001_c_cc.fits
CFITSIO error message stack:
01: failed to find or open the following file: (ffopen)
02: D:\SST-8300M Images\TARGETS\NGC470\20141201\calibrated\light\cosmetized\NGC470.L
03: uminance.Light.00000001_c_cc.fits

I have also tried running the script returning to “.xsif” as the default file extension and as the script’s output file suffix default with the same result.

This really makes me very angry. Whatever it is that I need to do to now solve the problem will come in the form of instructions that I may or may not understand and that will now require more time (probably much more time) in addition to the hour I have already wasted.

I have been a very strong supporter of PixInsight, frequently recommending it to others, and had concluded that it was a waste of money to acquire or use any other image processing software. Now you have left me with a program I cannot use and caused me unnecessary frustration and grief, with no easy end in sight. This is not a way to generate customer good will or to build your customer base with ordinarily intelligent astrophotographers who do not possess special programming skills.

Is there a, hopefully simple, way out of this?

Jay Butler
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: seigell on 2014 December 01 18:03:03
You do realize that you are free to download any of several of the more recent older releases, and install in place of the v01.08.03.1123 which you are indicating as the cause of your troubles ??

You may well be frustrated, but its a bit harsh to complain rather vehemently - including asserting that the remedial instructions will be beyond your grasp - and then "ask" for a "simple way out of this"...
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Geoff on 2014 December 01 19:53:15
I have noticed one post one this thread and one on http://pixinsight.com/forum/index.php?topic=7820.0 (http://pixinsight.com/forum/index.php?topic=7820.0) which claimed that BPP now no longer worked, so I decided to test my installation. I did the following
1. Ran BPP with the default .xisf selected with fits files as input. Everything worked OK and ImageIntegration on the resulting registered .xisf files gave a decent looking result.
2. Ran BPP with the same input files but with the default .xisf changed to .fit.  I ran ImageIntegration of the resulting .fit files but with no other changes and again got a decent looking result.
3.  I would have expected both integrated images to be identical, but they were not.  Subtracting one from the other with PixelMath gave an image that was non-zero.
4. The Process console also showed differences: Integrating the registered fits files gave
Gaussian noise estimates  : 1.2427e-004
Scale estimates           : 1.4255e-004
Location estimates        : 5.4082e-003
SNR estimates             : 4.6267e+003
Reference noise reduction : 1.2291
Median noise reduction    : 1.3839

6.084 s
while integrating the xisf files gave
Gaussian noise estimates  : 1.1899e-004
Scale estimates           : 1.3848e-004
Location estimates        : 2.0517e-003
SNR estimates             : 3.1851e+003
Reference noise reduction : 0.9861
Median noise reduction    : 1.0701

6.006 s

Both applications of ImageIntegration rejected exactly the same number of pixels from each subframe.

So why were there the above-noted differences?
Geoff
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Jay W. Butler on 2014 December 01 20:14:00
Reply to Seigell:

1.I complain "somewhat vehemently" precisely because I had expressed concern that this would happen and was assured that it would not.
2. If I go back to an earlier version of PI, which I can of course do, then I risk being stuck in a time warp where my earlier version is frozen because later updates are likely designed as revisions to the latest update. Why should I be required to run that risk to solve a problem not of my making?
3. I do not see the inconsistency between expressing concern that the "fix" will be difficult to understand and implement for a non-programmer like me (particularly given the dense, technical-term laden and fragmentary "documentation" incorporated in the program) and a request that instructions for the "fix" be simple. It's called cause and effect, a relationship you can surely comprehend.

Jay
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: mschuster on 2014 December 01 20:38:59
Jay,

With respect:

It think it would be helpful if you would provide Juan with files and script settings for testing purposes.

Also, I suggesting going back to an old version, wait four months, and then upgrade.

Alternatively, you can keep both versions handy, do production work with old version, and switch to newer versions to test and provide Juan with feedback. IMO I think everyone would benefit.

Mike
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: jeffweiss9 on 2014 December 01 22:09:44
From comments and problems above, it seems to me that this new format should not have been sprung on users as the 'new default', however meritorious for future PI programming ease.  Users still need to efficiently use the program based on hard-earned knowledge (with not much documentation).  Although I haven't yet tried it myself (and will resist as long as possible after reading here), clearly there are compatibility problems in spite of all assurances that that would not be the case.  This should have been opt-in, not opt-out, until all compatibility problems were resolved.  Or, at least give the user a global opt-out option that really works before it is released.   I love PixInsight but this insular format change is testing me and has some of the 'purist' flavor of locking out people from the Gallery who happen to believe PI is a tool, like any other tool, and feel their freedom infringed when told they must be 100% pure to enjoy full community support in the PI forum. PI has the freedom to invent a new format and declare it the default, but they should take more seriously the goal of supporting continued use by the wider community of users who are not all computer programmers.
Just my opinion(s).
-Jeff
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: pfile on 2014 December 01 22:23:21
there should probably be a beta program for changes that affect the whole platform such as this one.

rob
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 December 02 07:12:28
This has been a small bug in the CosmeticCorrection tool. I have just released two updates that fix this problem on all platforms.

Sorry for the inconvenience.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 December 02 07:38:39
Quote
Although I haven't yet tried it myself (and will resist as long as possible after reading here), clearly there are compatibility problems in spite of all assurances that that would not be the case.
Quote
there should probably be a beta program for changes that affect the whole platform such as this one.

The XISF module is a production stable release. It has been tested for stability and compatibility with the platform. The problem that has arisen here with CosmeticCorrection is of marginal importance and has nothing to do with the new format. It is just a regression caused by some recent changes in the BatchPreprocessing script, mixed with a small mistake I made when I packed the latest update.

It has taken less than 13 hours to get this problem fixed with the corresponding updates, which I think is not so terrible, especially taking into account the time zone differences. And that's because I have taken the liberty to sleep more than usual today, as I was tired after the effort put in the latest releases.

Bugs happen! They happen everywhere. Frankly, I think that the assertions you have made here are unfair with the work we are carrying out day to day to deliver high quality, innovative software to our users.

As everybody knows we are absolutely perfect. Obviously we have to simulate bugs from time to time, just to look normal, so bear with us... :D
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 December 02 07:45:10
Jay,

Just as I feared would happen after downloading and installing the updates relating to the new XISF file format and related batch pre-preprocessing script, I now have a system that will not process my .fits images. Thank you very much for this gratuitous solution to a problem that I at least did not know that I had. As I said, for me it is a "solution" in search of a problem; and all you have done for me is create a previously non-existent problem, which I will now describe.

My answer to this paragraph is the two updates that I have released a while ago. Let me know if the updates fix your problem, and if they don't, we'll try to do our best to fix them as quickly as possible.

Please accept my apologies for having caused you so much trouble. It was not my intention.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: dayers on 2014 December 02 09:04:37
All,

I work with PI extensively virtually every day. I have several projects going, and the format change for me so far has been seamless. I think there have been some overreactions in this thread that I think are unfair to Juan and his team. Not everyone has had problems with the change from FITS to XISF format.

Dave
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: pfile on 2014 December 02 09:12:02
Quote
there should probably be a beta program for changes that affect the whole platform such as this one.

Bugs happen! They happen everywhere. Frankly, I think that the assertions you have made here are unfair with the work we are carrying out day to day to deliver high quality, innovative software to our users.


exactly. bugs happen. which is why our design team has a > 1:1 ratio of verification engineers to rtl designers. and we have a giant QA department. and we run nightly hardware and software regressions. and on and on and on. obviously the cost of a bug in silicon is orders of magnitude more costly than a software bug, but when you have a big system, even the software needs to be hammered on for quite a long time before it is ready for release.

since you don't have those resources, a logical alternative is to "hide" big releases from the bulk of your users. you will always have a subset of advanced users that want to try new things, and they can help you find bugs before those bugs hit the mainstream users who (as evidenced by this thread) don't have the patience or desire to deal with these bugs.

no one is saying you don't write good software, or you don't support your software. i'm just saying that the blow-up in this thread could have been avoided by letting those users who are willing to take risks do that "behind the scenes" to flush out the bugs that you didn't find. case in point - a bug that just popped up this morning where saving XISF with japanese characters in the filename did not work. are you going to test every localization on your own? impossible.

rob


Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 December 02 10:05:09
Hi Geoff,

3.  I would have expected both integrated images to be identical, but they were not.  Subtracting one from the other with PixelMath gave an image that was non-zero.

There are very small differences caused by the way each format module loads pixel data. The FITS module has to communicate with CFITSIO (the software library that provides FITS format support), and this involves an "adaptation layer". In particular, pixel samples are always read as 64-bit floating point values from FITS files in order to support all possible data types (from 8-bit integers to 64-bit floating point), and then the values are converted to the data type of the image being read. For a 32-bit floating point image, this involves a conversion from double precision (http://en.wikipedia.org/wiki/Double-precision_floating-point_format) to single precision (http://en.wikipedia.org/wiki/Single-precision_floating-point_format). In our XISF implementation pixel data are read and written directly without any additional conversions.

This conversion is not perfect. For example, 1.234567 in double precision gives 1.2345670461654663 when converted to single precision and then back to double precision. The tiny difference is caused by the different  binary representations of floating point types. Basically, this difference can be understood as a form of noise. However, this noise is limited to the least significant binary digit of the single precision type, which cannot be larger than 1.2x10-7 (the machine epsilon (http://en.wikipedia.org/wiki/Machine_epsilon) for the single precision type), so it is insignificant for all practical purposes.

Quote
Median noise reduction    : 1.3839
Median noise reduction    : 1.0701

Effective noise reduction (ENR) estimates given by the ImageIntegration tool are not comparable between different data sets. They are relative estimates that you should try to minimize for a particular set of images by tweaking II parameters. Since they are not absolute estimates, a comparison as above is in general meaningless. The ENR function is extremely sensitive to small differences in the distribution of pixel values over low-signal areas of the image, which is a very good property in this case.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 December 02 10:29:52
case in point - a bug that just popped up this morning where saving XISF with japanese characters in the filename did not work. are you going to test every localization on your own? impossible.

Do you think this problem would have arisen had I released XISF as a beta version? Based on my experience after more than 10 years in this market, I bet it wouldn't. Unfortunately, this does not work that way. We perform exhaustive stability tests for each new release, but we cannot, as you say, verify every possible scenario where a specific tool or new feature can fail. We release when we are sure that what we are releasing is stable and works with our test data sets and stress projects. Then we make mistakes, overlook things, and have limitations, so bugs happen. But I believe that our current release policy is the best option to keep moving forward. IMHO, the improvements that we achieve are beneficial for all of our users, even for those that experience some annoyances.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: pfile on 2014 December 02 11:10:19
case in point - a bug that just popped up this morning where saving XISF with japanese characters in the filename did not work. are you going to test every localization on your own? impossible.

Do you think this problem would have arisen had I released XISF as a beta version? Based on my experience after more than 10 years in this market, I bet it wouldn't. Unfortunately, this does not work that way. We perform exhaustive stability tests for each new release, but we cannot, as you say, verify every possible scenario where a specific tool or new feature can fail. We release when we are sure that what we are releasing is stable and works with our test data sets and stress projects. Then we make mistakes, overlook things, and have limitations, so bugs happen. But I believe that our current release policy is the best option to keep moving forward. IMHO, the improvements that we achieve are beneficial for all of our users, even for those that experience some annoyances.

who knows? if that japanese user had tried it, then yes. if you had put out a beta and said, "i didn't have time to test X, Y and Z" then someone might find the time to test those things. for instance, i speak a little japanese and i have 3 japanese input methods turned on on my mac. in theory i could have tested the localization for japanese even though i don't normally have those input methods turned on.

i'm just telling you you have a base of somewhat fanatic users that would be happy to test things out in advance of wide release. if you don't want to take advantage of that, it's your choice. if no one takes you up on it, then you were no worse off. if they do take you up and find some bugs, then less sophisticated users are spared some grief.

rob
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: jkmorse on 2014 December 02 15:20:31
Rob,

I respect what you are suggesting but tend to disagree with its application here.  We are already a relatively small community of users and, as this post has shown, even us non experts can help identify problems (though maybe with a bit less venom at times).  I think that is particularly true with sophisticated software like PI which has users with highly scientific/technical backgrounds and us liberal arts types who love the hobby but couldn't write code if our lives depended on it (that said, I love reading the discussions between the experts like yourself and Juan as you chart solutions for the uninitiated like me).

Second, I fear the impact of beta-testing time dilation.  From what I have seen, running that system can substantially slow the roll out of new developments.  True, when the update does arrive, it may be "cleaner" but I would rather gets the updates weeks or months earlier even if it means a few days of frustration as we hammer out the kinks.

For what it's worth, but as I have said before and as I am sure others feel as well, one of the best aspects of PI is its sophistication.  But the other is the amazing dedication of Juan and his team to keep making it better.  I have to say, i am thrilled every time I see a new announcement from Juan about a major new initiative even if it takes work to understand it and time for its full potential to be realized.  Never once have I thought, "oh god, what if this doesn't work right out of the box?"  If I wanted safe, I would have just kept painting my images with Photoshop.

Best,

Jim
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Jay W. Butler on 2014 December 02 16:46:05
Juan,

I have as of late today (Tuesday, Wednesday in Europe) downloaded the debugged versions of PI and the batch pre-processing script you posted earlier today and have now used them successfully to create and process an LRGB image using bias, dark, flat and light frames generated in FITS format using TheSkyX. I have been able successfully to generate both XISF and FITS outputs by making the appropriate alternative selections in Global Preferences and in the BPP script window and to further process that output using a variety of PI processes. The problem I experienced earlier has disappeared.

Many thanks to you, Juan, for making the necessary bug fix in a timely way. I continue to believe, as I said from the outset, that PI is by far the best astronomical image processing tool on the market and, for my money, makes anything else unnecessary.

Jay
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: jkmorse on 2014 December 02 17:11:18
Jay,

Sorry for the venom comment.  It was a cheap shot.  But your note does verify the point I was making in my response to Rob.  Beta testing would likely have denied your ability and that of the Japanese user to help get the kinks out quickly.  And rest assured, you can count on Juan to work issues as fast as he can (if he would just give up that whole sleep thing).

Best,

Jim
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: pfile on 2014 December 02 17:53:47
sorry you had to waste so many electrons, but i never said it should be a closed beta.

Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: cdesselles on 2014 December 02 18:36:14
Boy oh boy!  What a thread!  Even though I have sat out the all of the debate (pro or con) as I haven't had the chance to check out the new update yet, I applaud ALL of the posters here for one whale of an entertaining discussion. 

To Juan and all of the PI team:
 
Keep up the fantastic work!  Please... continue to drag the rest of us, kicking and screaming, into that unknown realm of advancement.   All some of us ask is that you remember that the vast majority of your user base are not PHD mathematicians.  Try to be a little less harsh in your descriptions that invariably go way over most of our heads.

To the user base:

Even as only a semi-technically trained professional, I could recognize the statement that XISF didn't replace FITS.  It enhances it with all of FITS functionality included within it.  As for file conversion, who among us has not made extensive use of converting our results from floating point FITS to integer Tiff or JPEG so we could share our results with other via Astrobin, Photobucket, or even Facebook.  This is a quick simple step that is of such little consequence I'm amazed it comes up at all.  On the other hand, we as a customer base should exercise our right to voice our opinions on features and user friendliness of such an amazing and powerful software package.  After all, we have paid for that right when we purchased a license.

Speaking for myself:

Part of what I paid for is not just the software itself, but the dedicated and very innovative team that keeps PI on the cutting edge.  There is no possible way that innovation can continue without allowing the innovators the leeway to innovate.  This is where a bit of trust comes to the fore.  I trust the PI team to keep this application at the forefront of astronomical digital image processing.  The PI team can trust in me to give them the benefit of the doubt when it comes to new updates, to let them know as fully and factually as I can when those updates don't quite meet expectations, and to support them as best I can to assist them in resolving such "bugs" when they inevitably arise.  Can either side ask for more?

     
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Buzz on 2014 December 04 01:59:08
As a comparative newcomer I welcome the forsight that the PI development team have to ensure that the image processing capabilities of PI (and other) applications are not limited by the file format.  It certainly is pushing the boundaries. My images have improved greatly since taking the blue pill and I take every opportunity to encourage others to use PI.

At the same time, many of us have personal boundaries, influenced by the fact that the detailed operation of several key tools remains undocumented and requires extensive browsing and research to move forward.

My own helicopter view (and I am know of others who think similarly) is the effective use of PI is hampered by limited documentation and the advancement of new tools needs equal attention to supporting information to be at its most effective.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Geoff on 2014 December 04 02:10:32
Totally agree Chris. I'm sure the lack of documentation deters a lot of potential buyers, but more important than documentation is a "How to get started in PI". Yes I know there is plenty of web and video info, but a pull down menu item in PI itself with a link to an official PI document outlining the basics would be a great asset to people using the trial program.
Geoff
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Buzz on 2014 December 04 04:11:09
For info, I'm currently finishing off my third book, this one on Astrophotography, to be published by Focal Press in the new year. The initial reviewers of the book proposal all asked for PI content.

It includes over 100 pages on PI processing, including an introduction, alternative PI workflows and how to use it on a number of example images and comparing it with other applications' tools. To achieve this I have had to evaluate and make sense of dozens of resources on web and forum and then consolidate this into a structured approach. The research and writing has propelled my learning and I'm glad they demanded it.

100 pages is not enough though - it deserves many more to fully do PI justice.   

Maybe there is a collaborative project out there - between the developers and the users, to put something back to the community?

regards
Chris Woodhouse


Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: MortenBalling on 2014 December 06 17:17:41
Since I've upgraded today, I get this strange error message:

(https://lh5.googleusercontent.com/-ZqsNGdxNPdw/VIOpwcS4yxI/AAAAAAAADmA/aVV_spvZ54I/s650/Screen%2520Shot%25202014-12-07%2520at%252002.12.23.png)

I was trying to Blink my Lum and my Ha stacks, and I'm pretty sure that worked yesterday with fits?

So who am I to push, and where?  ;)

Best regards

Morten

(And yeah, I know my paths are a mess)
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 December 07 05:03:26
Hi Morten,

This is a limitation of the Blink tool, but it has nothing to do with XISF. This happens because you are selecting a master image created by the BatchPreprocessing script with the "generate rejection maps" enabled (as it is by default). The file contains three images: the master frame and the high and low rejection map images. Note that this happens exactly the same with the FIST format.

This said, I've taken a look at Blink's source code and it would be very easy to change it to support multiple images, or at least to allow loading a multi-image file by simply ignoring the additional images. I'll try to make this change in the next version. You can also write a PM to Nikolay.
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: MortenBalling on 2014 December 07 08:50:58
Hi Juan

Thanks! I was thinking about something similar earlier today. I guess some of the things I do with PI isn't your normal out of the box routines. I use the software a lot, for combining a lot of different images, (typically nonlinear creative common licensed images), so I often run into problems, that might be relatively easy to fix, but I'm also aware, that it might not be the most urgent stuff for other users.

I actually have a few other suggestions (masks integrated as alpha etc.), but I'll try to sum that up, and make another thread.

BR

Morten :)
Title: Re: New File Format: XISF - Extensible Image Serialization Format
Post by: Juan Conejero on 2014 December 22 00:50:32
Just to keep the people informed, I have released a first draft XISF 1.0 specification document (http://pixinsight.com/forum/index.php?topic=7907.0).