Author Topic: XISF Version 1.0 Specification - DRAFT 6  (Read 6112 times)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6540
    • View Profile
    • http://pixinsight.com/
XISF Version 1.0 Specification - DRAFT 6
« on: 2015 February 15 12:32:12 »
A new version of the XISF Version 1.0 Specification (DRAFT 6) is now available online:

      http://pixinsight.com/doc/docs/XISF-1.0-spec/XISF-1.0-spec.html

You may need to refresh your browser.

The previous draft 3, 4 and 5 documents are also available for reference:

      http://pixinsight.com/doc/docs/XISF-1.0-spec-D3/XISF-1.0-spec.html
      http://pixinsight.com/doc/docs/XISF-1.0-spec-D4/XISF-1.0-spec.html
      http://pixinsight.com/doc/docs/XISF-1.0-spec-D5/XISF-1.0-spec.html

The entire PIDoc source code of the latest version is available on our GitHub open source repositories:

      https://github.com/PixInsight/XISF-specification

Main changes:

- Fixed a number of errors in the equations of section 8.5.5 Representable Range.

- New section: 8.5.7 Adaptive Display Function Algorithm.

- New figures 4 and 5 (display function / adaptive display function examples).

- Other bug fixes.


A big thanks to all the people who are helping us with this project.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Corries

  • Newcomer
  • Posts: 38
    • View Profile
Re: XISF Version 1.0 Specification - DRAFT 6
« Reply #1 on: 2015 February 16 12:26:29 »
Hi Juan,

I am very new to PixInsight (and astrophotography for that matter) and I have a some questions about the new XISF specification.

1.   why did you change the file format to XISF?
2.   what are the advantages (in simple language, please)?
3.   should I change (I am a real newcomer, remember)

and while we are the subject

4.   when I go to save a file there are seven options of file format to choose from (from 8-bit unsigned integer to 64-bit IEEE 754 floating point).  I always choose the default (32-bit IEEE 754 floating point).  Is this the correct one to use and if so why all the other options?

I would really appreciate your advice.

Many thanks.
« Last Edit: 2015 February 16 15:53:20 by Corries »

Offline jrista

  • Newcomer
  • Posts: 11
    • View Profile
Re: XISF Version 1.0 Specification - DRAFT 6
« Reply #2 on: 2015 February 16 13:56:28 »
I just did a quick run through the spec. At a high level it looks very well done. Very clean, thorough, well illustrated. I would need more time with it to say anything in detail, however there was one thing I noticed that I thought I would mention. The bayer array types currently supported. They do not seem to support any of the Kodak RGBW CFAs. I know of a couple members on the CN forums who still use some of the older SBIG cameras with sensors like the KAI-8050:

http://www.ccd.com/pdf/ccd_8050.pdf

I have also seen a number of images on AstroBin taken with this camera and other similar cameras using these kinds of sensors. Would it be prudent to support such bayer patterns out the gate with XISF? I assume the format could be extended to support them if absolutely necessary (the spec does not explicitly state whether custom codes can be used for the cfaType property, that might be a key area where extensibility should probably be added.)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6540
    • View Profile
    • http://pixinsight.com/
Re: XISF Version 1.0 Specification - DRAFT 6
« Reply #3 on: 2015 February 17 06:13:58 »
Quote
1.   why did you change the file format to XISF?
2.   what are the advantages (in simple language, please)?

Because we need to replace FITS with an efficient, open and free format. Here is a brief, non-comprehensive list of important problems that we are addressing with XISF:

* Lack of formal resources to define how image data have to be interpreted and represented. For example, application A stores floating point images in the range from -12345.123 to +6789.001, but this range is not declared publicly. No problem, as this can be done legally with FITS. Application B uses the range from 0 to 123456, also undeclared. B has no way to know how to interpret floating point pixel data written as FITS files by A, and vice-versa, so despite the fact that both applications implement the FITS standard, they have a serious interoperability problem. The developers of B are smart and, after wasting precious time that they should invest in creative development tasks, manage to discover by reverse engineering the black and white points of images written by A. That's great because now B can read FITS files written by A so B users using A data are happy. One day, the developers of A decide, for strictly technical reasons, to change image ranges in their next version, and again they don't publicize them. Suddenly, all B users who need to import images written by A are lost. This cannot happen---and will never happen---with XISF.

* Undefined physical disposition of pixel data. For example, a two-dimensional RGB color image can be stored as a three-dimensional image in a single header-data unit (HDU), with three contiguous blocks to store the red, green and blue channels. It can also be stored as three one-dimensional images in three HDUs, one for each channel. Or as four HDUs, one with all the metadata and no data and three with minimal metadata and one-dimensional channel data. It is unclear (to me at least) if it can also be stored as a one-dimensional sequence of groups of contiguous red, green and blue pixel samples in a single HDU. This "flexibility" can be nice, but FITS has no formal resources to define the organization of multichannel or vector-valued images. In XISF we have two pixel storage models (planar and normal models) formally defined without ambiguity.

* No color spaces. If an application stores a three-dimensional image (also known as a data cube), what is it? An RGB color image? A grayscale image with two alpha channels? HSV? CIE XYZ? CIE L*a*b*? We need colorimetrically defined color spaces to perform brightness/chromaticity separations and other essential tasks. The word "color" appears only once in the FITS Standard version 3.0 document to say this (section G.2.1):

FITS data arrays contain elements which typically represent the values of a physical quantity at some coordinate location. Consequently they need not contain any pixel rendering information in the form of transfer functions, and there is no mechanism for color look-up tables.

* Lack of essential metadata and auxiliary data structures such as ICC color profiles, CFA patterns, image resolution parameters, image thumbnails, optimal visualization parameters, etc.

* Obsolete metadata and lack of Unicode support. Punched cards are cool, but as of 2015, we really need more than 7-bit ASCII, 8-character property names, and 80-byte metadata rows. We need full Unicode support, structured property identifiers, unlimited length names and values, and much more fundamental data types and data structures than 'logical', 'numeric', and 'character'. For example, if my name were "Iván Martínez Añejo", I would have to store it as something like this in FITS:

   name="AUTHOR" value="'Ivan Martinez Anyejo'" comment="Not my real name!"

while in XISF:

   <Metadata>
      <Property id="XISF:Author" type="String">Iván Martínez Añejo</Property>
   </Metadata>


* Rigid header-data sequential organization. Magnetic tapes also are very cool, but definitely not contemporary devices. For example, imagine a FITS file storing three images as three consecutive HDUs, as required by the FITS standard. If the file is available on a local hard disk this is not a practical problem: to access the headers of the second and third HDUs we need two file seek operations, but hard disks are very fast so we don't notice. However, what happens if the file is in a remote server, for example, being accessed as "http://www.example.com/foobar.fits" ? We have to download the first HDU, including the whole first image (which we perhaps don't need), before starting to download the header of the second HDU, and the same for the third one. Special network interface applications can be created to improve this situation, but this can't be an optimal solution. With a monolithic XISF file, an application can download the entire XISF header, which contains all of the metadata including all image and property descriptions, in a single operation, without needing to read a single pixel. With a distributed XISF unit, we can download the header and then download just the required image data.

* Lack of a distributed storage model. Distributed XISF units store the header (metadata) and the data as separate files, including local and remote resources. This allows for flexible storage configurations that are not possible with FITS. Furthermore, an XISF data blocks file allows indexed access to images and data objects through symbolic identifiers. This means that XISF data blocks files can be reorganized and extended freely without invalidating existing XISF units that depend on them. Distributed XISF units are also relocatable, so one can transport them between file systems and machines.

* No effective data integrity and authentication protection. Digital signatures based on XML signatures, X.509 certificates and cryptographic checksums are formalized in the XISF specification. A digitally signed XISF unit is effectively sealed and cannot be modified (There is a registered FITS convention for checksum keywords, but it does not protect authenticity because checksums can be tampered).


Quote
3.   should I change (I am a real newcomer, remember)

If you are a PixInsight user, you'll be using XISF to store your images in a natural way because it is PixInsight's native file format. You should always keep your original XISF files because XISF provides resources that don't exist in other formats. Unfortunately, you'll have to export them in different formats (FITS, TIFF, etc.) to use them with other applications. Hopefully, more applications will adopt XISF in the near future, and we'll work hard for this to happen.


Quote
4.   when I go to save a file there are seven options of file format to choose from (from 8-bit unsigned integer to 64-bit IEEE 754 floating point).  I always choose the default (32-bit IEEE 754 floating point).  Is this the correct one to use and if so why all the other options?

The 32-bit floating point format is the most logical option in most cases. The 32-bit unsigned integer and 64-bit floating point formats can be necessary to encode images with very large dynamic ranges, such as very deep linear HDR images generated with our HDRComposition tool. For the rest of images, 32-bit floating point should normally be used.
« Last Edit: 2015 February 17 06:24:12 by Juan Conejero »
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6540
    • View Profile
    • http://pixinsight.com/
Re: XISF Version 1.0 Specification - DRAFT 6
« Reply #4 on: 2015 February 17 06:59:47 »
Quote
I just did a quick run through the spec. At a high level it looks very well done. Very clean, thorough, well illustrated.

Thank you so much!

Quote
however there was one thing I noticed that I thought I would mention. The bayer array types currently supported. They do not seem to support...

Okay, I agree completely: the current cfaType attribute has to be replaced with something more comprehensive and flexible.

How about this solution:

   cfaPattern="p1p2...pn"

where each pi is an ASCII character that represents an element of the CFA matrix, oriented as it occurs on the image, with elements ordered from top to bottom and left to right, and where a pi can be one of:

   0   For a nonexistent or undefined CFA element
   R   Red
   G   Green
   B   Blue
   W   White
   C   Cyan
   M   Magenta
   Y   Yellow


The number n of CFA elements must have an integer square root, that is, only square CFA matrices can be described with this attribute. Rectangular matrices shall be represented as the necessary duplicates; for example, a 2 rows per 4 columns matrix "GWRWBWGW" shall be represented as a 4x4 matrix by "GWRWBWGWGWRWBWGW".

Examples:

   <Image ... cfaPattern="RGBG" ... />
   <Image ... cfaPattern="WRBG" ... />
   <!-- Fuji X-Trans -->
   <Image ... cfaPattern="GBGGRGRGRBGBGBGGRGGRGGBGBGBRGRGRGGBG" ... />


The new cfaPattern attribute would replace the current cfaType. What do you think?

Thank you for pointing out this important flaw.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline mschuster

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1008
    • View Profile
Re: XISF Version 1.0 Specification - DRAFT 6
« Reply #5 on: 2015 February 17 11:38:00 »
Quote
The new cfaPattern attribute would replace the current cfaType. What do you think?

Hi Juan,

This looks good. I suggest adding additional (optional) metadata that describes the type of CFA, such as "Kodak Sparse CFA". Such information would help define the pattern and help decoders employ appropriate algorithms for both deCFA and splitCFA operations, color space transformations, etc.

Thanks,
Mike

Offline jrista

  • Newcomer
  • Posts: 11
    • View Profile
Re: XISF Version 1.0 Specification - DRAFT 6
« Reply #6 on: 2015 February 17 17:40:46 »
Hi Juan,

Thanks for taking the time to address my question. I think a cfaPattern property would be much more flexible, without question.

I am curious about pattern shape, though. Are there sensors out there that have rectangular matrices? If so, is just specifying a string of color codes sufficient? What about non-rectangular patterns, like some of Fuji's designs with extra white pixels embedded at the corners of standard pixels? Both Canon and Sony have sensor designs that use different kinds of green pixels...both green pixels are not the same color of green (the Canon 7D is a real-world existing example of this...two different kinds of green pixel, Adobe had to address that explicitly in their ACR/LR AHD-based debayering engine). Sony has patents for sensors with triangular pixels even, and I think Fuji has patents (and maybe even cameras?) with hexagonal or octagonal pixels.... O_o I don't know how flexible you want to make the cfaPattern, but, there is some food for thought.

Regarding pattern. Just running my mind through it, the pattern "GWRWBWGW" could be 2x4, but it could also be 4x2. The pattern "GWRWBWGWGWRWBWGW" could be 4x4, or 2x8, or 8x2. Does it matter if it's 2x4 vs. 4x2? Are such patterns likely? When it comes to non-rectangular patterns, if they are even addressed, some way to define the pattern would certainly be needed...but that well fall out of scope for XISF. I think some way to hint at the actual dimensions of the matrix in addition to just the matrix definition itself could at least reduce or eliminate potential issues debayering sensor's properly. How to do that I'll leave up to you, but in my time as a software developer myself, I've learned that lax specifications can lead to issues, and often be abused. (Just look at the cross-browser issues that many major W3C specs, which very often left things "up to the user agent" (i.e. browser) to do as they please...and that lead to often significant incompatibilities with other standards, cross-browser inconsistencies, etc.)

I also like Mike's idea of allowing an optional name for the cfaPattern.

Thanks again for your time,
Jon

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2127
    • View Profile
Re: XISF Version 1.0 Specification - DRAFT 6
« Reply #7 on: 2015 February 18 03:06:31 »
It is nice to be able to describe virtually all possible CFA patterns. But what are we supposed to do with them:
- from a programmer's point of view, it is virtually impossible to support "all" patterns
- the bayer sequence alone may not be sufficient if the sensor grid has offsets between line (no idea if there are any sensors with this characteristics, but I remember experiments with hexagonal grids)
- assuming that I have a good algorithm for debayering a certain pattern: what is the exact meaning of green or blue. With this information only it is impossible to reconstruct an image with its "true" colors. What would be needed are the exact characteristics of the green or blue bayer filter.
- does it make any sense to consider "colors" here at all, or should we rather just think in terms of channels (channel 0,1,2). The usually RGGB pattern would then be encoded as "0112".

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6540
    • View Profile
    • http://pixinsight.com/
Re: XISF Version 1.0 Specification - DRAFT 6
« Reply #8 on: 2015 February 18 03:43:53 »
Hi Jon,

Quote
Thanks for taking the time to address my question.

Thank *you* for your contribution.

Quote
I am curious about pattern shape, though. Are there sensors out there that have rectangular matrices? If so, is just specifying a string of color codes sufficient? What about non-rectangular patterns, like some of Fuji's designs with extra white pixels embedded at the corners of standard pixels? Both Canon and Sony have sensor designs that use different kinds of green pixels...both green pixels are not the same color of green (the Canon 7D is a real-world existing example of this...two different kinds of green pixel, Adobe had to address that explicitly in their ACR/LR AHD-based debayering engine). Sony has patents for sensors with triangular pixels even, and I think Fuji has patents (and maybe even cameras?) with hexagonal or octagonal pixels.... O_o I don't know how flexible you want to make the cfaPattern, but, there is some food for thought.

Non-rectangular patterns or patterns with holes: that's why I have included a '0' element type for undefined or nonexistent CFA elements. This allows for arbitrary pattern shapes defined as rectangular matrices.

As for different green components, etc: Being an abstract image serialization format, XISF should not try to support or describe low-level hardware implementation details. IMO, describing CFA patterns in terms of RGB primary colors (plus WCMY to support a small subset of non-RGB cameras) is sufficient to interpret the physical organization of pixel components. If a decoder needs more hardware-related details, it should derive them on its own from image metadata. For example, the cfaPattern attribute, along with a CFA name (Mike's suggestion) and an Instrument:Camera:Model property, should be sufficient in such cases. On the other hand, XISF is not designed or intended to replace proprietary raw file formats such as .cr2 or .nef.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6540
    • View Profile
    • http://pixinsight.com/
Re: XISF Version 1.0 Specification - DRAFT 6
« Reply #9 on: 2015 February 18 04:10:02 »
I suggest adding additional (optional) metadata that describes the type of CFA, such as "Kodak Sparse CFA". Such information would help define the pattern and help decoders employ appropriate algorithms for both deCFA and splitCFA operations, color space transformations, etc.

OK, I think this is getting complex enough as to define a new XML core element, instead of adding more attributes to <Image>. How about this:

<Image ... >
   <CFA name="Kodak TRUESENSE Sparse Color Filter"
        pattern="GWRWWGWRBWGWWBWG" width="4" height="4" />
</Image>


If we really want to be exhaustive, we could define something like:

<Image ... >
   <CFA name="Example RGBG CFA" pattern="RgBG" width="2" height="2">
      <Component id="R" center="620" amplitude="150" />
      <Component id="g" center="525" amplitude="80" />
      <Component id="G" center="575" amplitude="90" />
      <Component id="B" center="475" amplitude="120" />
   </CFA>
</Image>


with central wavelength and amplitude in nm.

But, is this really necessary? I am not sure. And if it becomes necessary in the future, we can always extend the specification, since both options are not mutually exclusive if <Component> child elements are made optional (remember: the X is for extensible).

What do you think, folks?
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6540
    • View Profile
    • http://pixinsight.com/
Re: XISF Version 1.0 Specification - DRAFT 6
« Reply #10 on: 2015 February 18 11:57:29 »
Hi Georg,

It is nice to be able to describe virtually all possible CFA patterns. But what are we supposed to do with them:

Good questions.

Quote
- assuming that I have a good algorithm for debayering a certain pattern: what is the exact meaning of green or blue. With this information only it is impossible to reconstruct an image with its "true" colors. What would be needed are the exact characteristics of the green or blue bayer filter.

AFAIK, what decoders do, including dcraw, is to apply a mix of a priori knowledge and data available in some raw files. See this interesting document about Canon raw formats:

http://lclevy.free.fr/cr2/

and specifically section 5 of the document, where it describes color space conversions, dark level subtraction and white balancing. I think all of this stuff is well out of the scope of XISF. As I have said, XISF is not intended to replace proprietary camera raw formats.

In my opinion, we should not go too far with hardware-related properties at this stage. I think that the CFA element without the Component child element (see my previous post) is enough for now, unless there were consensus that we have to include it. We can always extend the specification to be more comprehensive, when necessary.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2127
    • View Profile
Re: XISF Version 1.0 Specification - DRAFT 6
« Reply #11 on: 2015 February 18 13:31:31 »
Hi Juan,

the point I wanted to make is that knowing the CFA pattern is probably less useful than it appears at first glance. To properly decode a CFA image, you need much more information - such as camera model, filter characteristics, sensor characteristics, etc. . The CFA pattern is only useful for a rather coarse approximation of real good debayering, everything else needs much more (such as camera model etc). So if you include the CFa pattern as an attribute in XISF, we should be aware of the limited usefulness this provides. Better than nothing, but...

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6540
    • View Profile
    • http://pixinsight.com/
Re: XISF Version 1.0 Specification - DRAFT 6
« Reply #12 on: 2015 February 20 09:52:38 »
I have just uploaded a working copy of the next DRAFT 7 of the XISF specification document:

http://pixinsight.com/doc/docs/XISF-1.0-spec-D7/XISF-1.0-spec.html

Changes:

* Removed the cfaType attribute of the Image core element.

* New ColorFilterArray core element (section 11.8).

* The RGBWorkingSpace element replaces the previous RGBWS (section 11.6, renamed for coherency).

* Minor bug fixes in some equations (typesetting bugs).

Let me know if you like ColorFilterArray as it is formalized now.

The actual DRAFT 7 will include more changes:

* XISF data blocks files (section 9.4) will include a descriptive XISF header. This is necessary because an orphan XISF blocks file (e.g., if all header files referencing it are lost) would be impossible or very difficult to recover otherwise, since we can't know what a data block stores without a header describing it.

* More standard compression algorithms.


All your contributions, bug reports, suggestions and constructive criticism will be most definitely welcome.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline mschuster

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1008
    • View Profile
Re: XISF Version 1.0 Specification - DRAFT 6
« Reply #13 on: 2015 February 20 16:01:26 »
Hi Juan,

Re ColorFilterArray. Sometimes CFA image size downloaded ends up not being a multiple of pattern size (don't know why). SplitCFA has trouble with this last time I tried (bug?). Should a mention of this issue be made in your document?

When integrating raw CFA bias and darks to generate masters, will IImageIntegration preserve ColorFilterArray? Seems like it should. What if there is a frame to frame mismatch?

I agree with Georg re more information needed for advanced deCFA. But usually proprietary and IMO probably best not addressed now.

Mike

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6540
    • View Profile
    • http://pixinsight.com/
Re: XISF Version 1.0 Specification - DRAFT 6
« Reply #14 on: 2015 February 21 02:13:42 »
Hi Mike,

Quote
Sometimes CFA image size downloaded ends up not being a multiple of pattern size (don't know why). SplitCFA has trouble with this last time I tried (bug?). Should a mention of this issue be made in your document?

Yes, this happens. For example, dcraw loads raw images from my EOS 7D as 5202x3465 pixels, so the last filter patterns at the bottom are halved. I'll check what you say about SplitCFA.

This is a low-level hardware-related issue that needs not be mentioned in the specification IMO.

Quote
When integrating raw CFA bias and darks to generate masters, will IImageIntegration preserve ColorFilterArray? Seems like it should.

Yes, ImageIntegration should define a CFA property for all mosaiced master images. The same is true for ImageCalibration with output calibrated mosaiced frames.

In the specification, section 11.8, I have written:

Encoders should associate ColorFilterArray elements with images mosaiced with CFAs.

Instead of should, I normally would have written shall, but this would exclude OSC raw images stored as FITS files, where there is no way to know the CFA pattern (or even if the image is mosaiced). I'm thinking on rewriting this sentence as:

Encoders shall associate ColorFilterArray elements with images known to be mosaiced with CFAs.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/