New File Format: XISF - Extensible Image Serialization Format

[Juan Conejero]

Does XISF have a documented specification yet or is it only by a reference implementation so far?

I am working right now on a formal definition document.

Just as an informal example, this is the XISF header for a compressed raw DSLR image:

<?xml version="1.0" encoding="UTF-8"?>
<!--
Extensible Image Serialization Format - XISF version 1.0
Created with PixInsight - http://pixinsight.com/
-->
<xisf version="1.0" xmlns="http://www.pixinsight.com" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.pixinsight.com http://pixinsight.com/xsd/xisf-1.0.xsd">
   <Image geometry="5796:3870:1" sampleFormat="UInt16" colorSpace="Gray" cfaType="RGGB" compression="zlib:44861040" location="attachment:1889:35792262">
      <FITSKeyword name="COMMENT" value="" comment="Decoded with PixInsight 01.08.03.1123"/>
      <FITSKeyword name="COMMENT" value="" comment="Decoded with DSLR_RAW module 01.02.00.0232"/>
      <FITSKeyword name="COMMENT" value="" comment="Decoded with dcraw version 9.22"/>
      <FITSKeyword name="INSTRUME" value="Canon EOS 5D Mark III" comment="Camera model"/>
      <FITSKeyword name="DATE-OBS" value="2012-04-11T16:15:37" comment="Camera timestamp"/>
      <FITSKeyword name="EXPTIME" value="0.000977" comment="Exposure time in seconds"/>
      <FITSKeyword name="ISOSPEED" value="400" comment="ISO speed as specified in ISO 12232"/>
      <FITSKeyword name="FOCALLEN" value="100.00" comment="Focal length in mm"/>
      <FITSKeyword name="APTDIA" value="12.50" comment="Aperture diameter in mm"/>
      <Resolution horizontal="100" vertical="100" unit="inch"/>
   </Image>
   <Metadata>
      <Property id="CreationTime" type="String8">2014-12-01T17:41:42Z</Property>
      <Property id="CreatorApplication" type="String16">PixInsight 01.08.03.1123</Property>
      <Property id="CreatorModule" type="String16">XISF module version 01.00.00.0023</Property>
      <Property id="CreatorOS" type="String16">Linux</Property>
      <Property id="CompressionMethod" type="String8">zlib</Property>
      <Property id="CompressionLevel" type="Int32" value="6"/>
   </Metadata>
</xisf>

This is for a RGB color image in 32-bit floating point format:

<?xml version="1.0" encoding="UTF-8"?>
<!--
Extensible Image Serialization Format - XISF version 1.0
Created with PixInsight - http://pixinsight.com/
-->
<xisf version="1.0" xmlns="http://www.pixinsight.com" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.pixinsight.com http://pixinsight.com/xsd/xisf-1.0.xsd">
   <Image geometry="960:540:3" sampleFormat="Float32" bounds="0:1" colorSpace="RGB" location="attachment:4096:6220800">
      <Resolution horizontal="72" vertical="72" unit="inch"/>
      <ICCProfile location="inline:base64">AAAH8HNjcnMCEAAAbW50clJHQiBYWVogB9EABAAMABMAAgAOYWNzcE1TRlQBAAAAbm9uZW5vbmUAAAAAAAAAAAAAAAAAAPbWAAEAAAAA0y1zY3JzAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKZGVzYwAAAPwAAABfY3BydAAAAVwAAAAJd3RwdAAAAWgAAAAUYmtwdAAAAXwAAAAUclhZWgAAAZAAAAAUZ1hZWgAAAaQAAAAUYlhZWgAAAbgAAAAUclRSQwAAAcwAAAIMZ1RSQwAAA9gAAAIMYlRSQwAABeQAAAIMZGVzYwAAAAAAAAAFc1JHQgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAdGV4dAAAAAAAAAAAWFlaIAAAAAAAAPbWAAEAAAAA0y1YWVogAAAAAAAAAAAAAAAAAAAAAFhZWiAAAAAAAABpkwAANnAAAATzWFlaIAAAAAAAAFuLAAC3FgAAHoRYWVogAAAAAAAALjQAABJ8AADzV2N1cnYAAAAAAAABAAAAAAAAAgAEAAcACwARABgAIAAqADUAQQBPAF4AbwCBAJQAqQDAANgA8gEOASsBSgFqAYwBsAHVAfwCJQJPAnsCqQLZAwsDPgNzA6oD4wQdBFkElwTXBRkFXQWjBeoGMwZ/BswHGwdsB78IFAhrCMMJHgl7CdkKOgqdCwELaAvQDDsMqA0WDYcN+g5uDuUPXg/ZEFYQ1RFWEdoSXxLmE3AT+xSJFRkVqxY/FtUXbhgIGKUZRBnlGogbLRvUHH4dKh3YHogfOh/vIKYhXyIaItcjlyRZJR0l4yasJ3YoQykTKeQquCuOLGYtQS4eLv0v3jDCMagykDN7NGg1VzZINzw4MjkrOiU7IjwiPSQ+KD8uQDdBQkJPQ19EcUWGRp1HtkjSSfBLEEwzTVhOf0+pUNZSBFM1VGlVn1bXWBJZT1qOW9BdFV5cX6Vg8WI/Y49k4mY4Z5Bo6mpHa6ZtCG5sb9NxPHKndBV1hnb5eG555nthfN5+XX/fgWSC6oR0hgCHjokfirOMSY3hj3yRGpK6lF2WApepmVSbAJywnmKgFqHNo4alQqcBqMKqhqxMrhWv4bGvs3+1UrcouQC627y5vpnAe8JhxEnGM8ggyhDMAs33z+7R6NPl1eTX5tnr2/Ld/OAI4hfkKeY96FTqbuyK7qnwyvLu9RX3P/lr+5r9y///Y3VydgAAAAAAAAEAAAAAAAACAAQABwALABEAGAAgACoANQBBAE8AXgBvAIEAlACpAMAA2ADyAQ4BKwFKAWoBjAGwAdUB/AIlAk8CewKpAtkDCwM+A3MDqgPjBB0EWQSXBNcFGQVdBaMF6gYzBn8GzAcbB2wHvwgUCGsIwwkeCXsJ2Qo6Cp0LAQtoC9AMOwyoDRYNhw36Dm4O5Q9eD9kQVhDVEVYR2hJfEuYTcBP7FIkVGRWrFj8W1RduGAgYpRlEGeUaiBstG9Qcfh0qHdgeiB86H+8gpiFfIhoi1yOXJFklHSXjJqwndihDKRMp5Cq4K44sZi1BLh4u/S/eMMIxqDKQM3s0aDVXNkg3PDgyOSs6JTsiPCI9JD4oPy5AN0FCQk9DX0RxRYZGnUe2SNJJ8EsQTDNNWE5/T6lQ1lIEUzVUaVWfVtdYEllPWo5b0F0VXlxfpWDxYj9jj2TiZjhnkGjqakdrpm0Ibmxv03E8cqd0FXWGdvl4bnnme2F83n5df9+BZILqhHSGAIeOiR+Ks4xJjeGPfJEakrqUXZYCl6mZVJsAnLCeYqAWoc2jhqVCpwGowqqGrEyuFa/hsa+zf7VStyi5ALrbvLm+mcB7wmHEScYzyCDKEMwCzffP7tHo0+XV5Nfm2evb8t384AjiF+Qp5j3oVOpu7IruqfDK8u71Ffc/+Wv7mv3L//9jdXJ2AAAAAAAAAQAAAAAAAAIABAAHAAsAEQAYACAAKgA1AEEATwBeAG8AgQCUAKkAwADYAPIBDgErAUoBagGMAbAB1QH8AiUCTwJ7AqkC2QMLAz4DcwOqA+MEHQRZBJcE1wUZBV0FowXqBjMGfwbMBxsHbAe/CBQIawjDCR4JewnZCjoKnQsBC2gL0Aw7DKgNFg2HDfoObg7lD14P2RBWENURVhHaEl8S5hNwE/sUiRUZFasWPxbVF24YCBilGUQZ5RqIGy0b1Bx+HSod2B6IHzof7yCmIV8iGiLXI5ckWSUdJeMmrCd2KEMpEynkKrgrjixmLUEuHi79L94wwjGoMpAzezRoNVc2SDc8ODI5KzolOyI8Ij0kPig/LkA3QUJCT0NfRHFFhkadR7ZI0knwSxBMM01YTn9PqVDWUgRTNVRpVZ9W11gSWU9ajlvQXRVeXF+lYPFiP2OPZOJmOGeQaOpqR2umbQhubG/TcTxyp3QVdYZ2+XhueeZ7YXzefl1/34FkguqEdIYAh46JH4qzjEmN4Y98kRqSupRdlgKXqZlUmwCcsJ5ioBahzaOGpUKnAajCqoasTK4Vr+Gxr7N/tVK3KLkAutu8ub6ZwHvCYcRJxjPIIMoQzALN98/u0ejT5dXk1+bZ69vy3fzgCOIX5CnmPehU6m7siu6p8Mry7vUV9z/5a/ua/cv//w==</ICCProfile>
   </Image>
   <Metadata>
      <Property id="CreationTime" type="String8">2014-12-01T17:53:56Z</Property>
      <Property id="CreatorApplication" type="String16">PixInsight 01.08.03.1123</Property>
      <Property id="CreatorModule" type="String16">XISF module version 01.00.00.0023</Property>
      <Property id="CreatorOS" type="String16">Linux</Property>
      <Property id="BlockAlignmentSize" type="UInt16" value="4096"/>
      <Property id="MaxInlineBlockSize" type="UInt16" value="3072"/>
   </Metadata>
</xisf>

The XSD definition document still has to be created. After the XML header the data blocks are stored in the the same file. Just to give you an idea of how XISF files look like.

[Juan Conejero]

And this one is for a tiny RGB image of 6x6 pixels, which is completely defined by the XISF header (note the embedded Data element, where the pixel data are encoded as base64). In this case there are no attached data blocks in the XISF file.

<?xml version="1.0" encoding="UTF-8"?>
<!--
Extensible Image Serialization Format - XISF version 1.0
Created with PixInsight - http://pixinsight.com/
-->
<xisf version="1.0" xmlns="http://www.pixinsight.com" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.pixinsight.com http://pixinsight.com/xsd/xisf-1.0.xsd">
   <Image geometry="6:6:3" sampleFormat="UInt8" colorSpace="RGB" location="embedded">
      <Data encoding="base64">AAAAAP8A/wD/AAAAAAAAAP8AAP8AAAAAAAAA/wD/AP8AAAAA/wD//wD/AP8AAP8A/wD//wD//wD//wD/AP8AAP8A/wD//wD/AP8AAAAAAAAA/wD/AP8AAAAAAAAAAP8A/wD/AAAAAAAAAP8A</Data>
      <Resolution horizontal="72" vertical="72" unit="inch"/>
   </Image>
   <Metadata>
      <Property id="CreationTime" type="String8">2014-12-01T18:07:54Z</Property>
      <Property id="CreatorApplication" type="String16">PixInsight 01.08.03.1123</Property>
      <Property id="CreatorModule" type="String16">XISF module version 01.00.00.0023</Property>
      <Property id="CreatorOS" type="String16">Linux</Property>
      <Property id="BlockAlignmentSize" type="UInt16" value="4096"/>
      <Property id="MaxInlineBlockSize" type="UInt16" value="3072"/>
   </Metadata>
</xisf>

[goofisd]

Can someone explain what this new format means for me in simple, plain language?

I have a saying, "I'm a lot dumber than I look" .... help me understand. Why do I care about this format?
What does it do for me?

I'm an amateur imager, using a CCD and pixinsight. Will this change my workflow, or the processes I use?
If your explanation includes "complex numbers" or "plane vectors" ... you've lost me. I'm an old man with either too much whiskey in my blood, or not enough ... will this change matter to me?

[jcinpv]

After reading the brief specs on XISF, it appears to be an upward growth of the FITS format, meaning that while it continues to contain all that FITS has and does, XISF offers a wide spectrum of features that FITS can't support. To me, that is a step toward an area of graphics manipulation and presentation far beyond what FITS can do.

By declaring it as public domain, this makes it possible to be extended by others in ways not foreseen by the authors. That, too, is a good thing.

FITS has not been left behind, it is being cradled into a new realm.

So, loosely speaking, XISF is to FITS what zip is to gzip. I hope the analogy can be seen.

John C.

[Juan Conejero]

Will this change my workflow, or the processes I use?

Hopefully, nothing will change in this regard. We have made efforts to make XISF compatible with all existing data and tools, so that the new format integrates seamlessly with the current PixInsight platform. Basically, you don't have to change anything. If you want to export a XISF image to a different format such as FITS, TIFF, JPEG, PNG, etc, just open the .xisf file and save it in the desired format. It's just that simple.

The benefits of the new format will be more subtle, and will be arriving gradually. We'll have better image and data management resources. Developers will enjoy a more robust and versatile platform, and this will allow for better and more sophisticated image analysis and processing tools in the medium-long term. I believe we are making a step in the right direction with this initiative.

[bitli]

+1 for rga128 on the need  to make some processing in other units than proportion of white.  I want to  be able  to consider data points as measurements. Currently the limitations in this area come from PI, not from FITS.  I hope that this will be solved as a part of this effort, including for data imported from fits.

I think that the original announcement of the new format was not very well expressed, the latest announcement is better.  I wish you all the best for your effort, but I strongly opposed (for what it is worth) to making a new unproven, undocumented,  unique format the default used by many unsuspecting users.

This must be an opt-in, not an opt-out, for the time being.

juan, this is not a critic of your work or proposed format, I am very interested in it. But my experience of working with occasional or less technical users or users not fluent in English or not having the time to rebuild the documentation from hundredth of posts is that they will be very confused by this situation. Whathever your technical expertise, effort and dedication, there will be issues in changing format, and this must be solved will the help of willing users.

--bitli

[Juan Conejero]

the need  to make some processing in other units than proportion of white.

I want to  be able  to consider data points as measurements.

There is a conceptual mistake in these requests. The graphical interface of PixInsight has been designed and implemented to work with images, not with generic data. What you are asking for simply falls out of the scope of the current PixInsight GUI.

You can work with data structures and types very different from images in PixInsight. We are doing this all the time everywhere. But not on the graphical user interface directly, because the GUI has been designed to work with images consistently and efficiently, not with arbitrary data. You can work with floating point images outside the [0,1] range without any problems; in fact, most image processing tools and scripts necessarily work with other data ranges internally (quick example: wavelet layers contain positive and negative real values with zero mean). However, once you leave a floating point image on a visible image window, it *must* be in [0,1] because the GUI has been designed and optimized to work with this range. These are the rules of the game and they work very well to generate visual representations.

Now if you want to visualize non-image data, you have to write a specific tool. For example, suppose that you want to represent a data matrix that contains temperature values from -50C to +50C. You can do this easily by adapting the data to a visually representable range, either [0,1] if the visual data are floating point numbers, or [0,2ⁿ-1] for n-bit integers. The conversion is just a linear scaling of the data from [-50,+50] to the appropriate representable range. For example, we do this to represent wavelet layers and the components of Fourier transforms as images. Once the data are being visualized, you can design a dedicated readout tool to measure temperatures instead of pixel sample values, either by reading the original data, or by doing the inverse conversion.

That said, we have plans to support other objects besides images directly in the GUI. However, don't expect this before version 2.0, and we still have a *lot* of work to do before that,

I think that the original announcement of the new format was not very well expressed, the latest announcement is better.

I agree. I will try to be less harsh the next time.

[Geoff]

You can work with data structures and types very different from images in PixInsight. We are doing this all the time everywhere. But not on the graphical user interface directly, because the GUI has been designed to work with images consistently and efficiently, not with arbitrary data. You can work with floating point images outside the [0,1] range without any problems; in fact, most image processing tools and scripts necessarily work with other data ranges internally (quick example: wavelet layers contain positive and negative real values with zero mean). However, once you leave a floating point image on a visible image window, it *must* be in [0,1] because the GUI has been designed and optimized to work with this range. These are the rules of the game and they work very well to generate visual representations.

Now if you want to visualize non-image data, you have to write a specific tool. For example, suppose that you want to represent a data matrix that contains temperature values from -50C to +50C. You can do this easily by adapting the data to a visually representable range, either [0,1] if the visual data are floating point numbers, or [0,2ⁿ-1] for n-bit integers. The conversion is just a linear scaling of the data from [-50,+50] to the appropriate representable range. For example, we do this to represent wavelet layers and the components of Fourier transforms as images. Once the data are being visualized, you can design a dedicated readout tool to measure temperatures instead of pixel sample values, either by reading the original data, or by doing the inverse conversion.

This gives us some inkling of the power of the changes, but it also points to the need of lots of lucid examples to help most(?)  of us to understand and use the new resource.
Geoff

[Jay W. Butler]

Juan,

Just as I feared would happen after downloading and installing the updates relating to the new XISF file format and related batch pre-preprocessing script, I now have a system that will not process my .fits images. Thank you very much for this gratuitous solution to a problem that I at least did not know that I had. As I said, for me it is a "solution" in search of a problem; and all you have done for me is create a previously non-existent problem, which I will now describe.

As you instructed, I went into Global Preferences and set the default file extension to “.fits”. In the new batch pre-processing script the output file suffix defaults to “.xsif” and I changed that as well to “.fits”. I then loaded into the batch pre-processing script interface master bias and dark files in .fits format from my library and the .fits subs captured by Software Bisque’s TheSkyX camera operating software during my last image capture session in the observatory. I then ran the script (with the settings I had previously used hundreds of times without any problem). As I got to the registration stage of the script, I received the following error message on the Process Console and the batch pre-processing script hung up:

PCL FITS Format Support: Unable to open FITS file:
D:/STT-8300M Images/TARGETS/NGC470/20141201/calibrated/light/cosmetized/NGC470.Luminance.Light.00000001_c_cc.fits
CFITSIO error message stack:
01: failed to find or open the following file: (ffopen)
02: D:\SST-8300M Images\TARGETS\NGC470\20141201\calibrated\light\cosmetized\NGC470.L
03: uminance.Light.00000001_c_cc.fits

I have also tried running the script returning to “.xsif” as the default file extension and as the script’s output file suffix default with the same result.

This really makes me very angry. Whatever it is that I need to do to now solve the problem will come in the form of instructions that I may or may not understand and that will now require more time (probably much more time) in addition to the hour I have already wasted.

I have been a very strong supporter of PixInsight, frequently recommending it to others, and had concluded that it was a waste of money to acquire or use any other image processing software. Now you have left me with a program I cannot use and caused me unnecessary frustration and grief, with no easy end in sight. This is not a way to generate customer good will or to build your customer base with ordinarily intelligent astrophotographers who do not possess special programming skills.

Is there a, hopefully simple, way out of this?

Jay Butler

[seigell]

You do realize that you are free to download any of several of the more recent older releases, and install in place of the v01.08.03.1123 which you are indicating as the cause of your troubles ??

You may well be frustrated, but its a bit harsh to complain rather vehemently - including asserting that the remedial instructions will be beyond your grasp - and then "ask" for a "simple way out of this"...

[Geoff]

I have noticed one post one this thread and one on http://pixinsight.com/forum/index.php?topic=7820.0 which claimed that BPP now no longer worked, so I decided to test my installation. I did the following
1. Ran BPP with the default .xisf selected with fits files as input. Everything worked OK and ImageIntegration on the resulting registered .xisf files gave a decent looking result.
2. Ran BPP with the same input files but with the default .xisf changed to .fit.  I ran ImageIntegration of the resulting .fit files but with no other changes and again got a decent looking result.
3.  I would have expected both integrated images to be identical, but they were not.  Subtracting one from the other with PixelMath gave an image that was non-zero.
4. The Process console also showed differences: Integrating the registered fits files gave
Gaussian noise estimates  : 1.2427e-004
Scale estimates           : 1.4255e-004
Location estimates        : 5.4082e-003
SNR estimates             : 4.6267e+003
Reference noise reduction : 1.2291
Median noise reduction    : 1.3839

6.084 s
while integrating the xisf files gave
Gaussian noise estimates  : 1.1899e-004
Scale estimates           : 1.3848e-004
Location estimates        : 2.0517e-003
SNR estimates             : 3.1851e+003
Reference noise reduction : 0.9861
Median noise reduction    : 1.0701

6.006 s

Both applications of ImageIntegration rejected exactly the same number of pixels from each subframe.

So why were there the above-noted differences?
Geoff

[Jay W. Butler]

Reply to Seigell:

1.I complain "somewhat vehemently" precisely because I had expressed concern that this would happen and was assured that it would not.
2. If I go back to an earlier version of PI, which I can of course do, then I risk being stuck in a time warp where my earlier version is frozen because later updates are likely designed as revisions to the latest update. Why should I be required to run that risk to solve a problem not of my making?
3. I do not see the inconsistency between expressing concern that the "fix" will be difficult to understand and implement for a non-programmer like me (particularly given the dense, technical-term laden and fragmentary "documentation" incorporated in the program) and a request that instructions for the "fix" be simple. It's called cause and effect, a relationship you can surely comprehend.

Jay

[mschuster]

Jay,

With respect:

It think it would be helpful if you would provide Juan with files and script settings for testing purposes.

Also, I suggesting going back to an old version, wait four months, and then upgrade.

Alternatively, you can keep both versions handy, do production work with old version, and switch to newer versions to test and provide Juan with feedback. IMO I think everyone would benefit.

Mike

[jeffweiss9]

From comments and problems above, it seems to me that this new format should not have been sprung on users as the 'new default', however meritorious for future PI programming ease.  Users still need to efficiently use the program based on hard-earned knowledge (with not much documentation).  Although I haven't yet tried it myself (and will resist as long as possible after reading here), clearly there are compatibility problems in spite of all assurances that that would not be the case.  This should have been opt-in, not opt-out, until all compatibility problems were resolved.  Or, at least give the user a global opt-out option that really works before it is released.   I love PixInsight but this insular format change is testing me and has some of the 'purist' flavor of locking out people from the Gallery who happen to believe PI is a tool, like any other tool, and feel their freedom infringed when told they must be 100% pure to enjoy full community support in the PI forum. PI has the freedom to invent a new format and declare it the default, but they should take more seriously the goal of supporting continued use by the wider community of users who are not all computer programmers.
Just my opinion(s).
-Jeff

[pfile]

there should probably be a beta program for changes that affect the whole platform such as this one.

rob