Author Topic: Questions about XISF format as used by PI  (Read 815 times)

Offline johnw3d

  • Newcomer
  • Posts: 19
    • View Profile
    • Flickr Astrophoto set
Questions about XISF format as used by PI
« on: 2016 November 27 20:01:53 »
Hi.  I'm writing a Python-based decoder for XISF files (as property dicts and Numpy arrays for use in the many Python-based astronomy tools out there) and have a few questions about the format as it is being currently  used in P.   

First, let me say the format is an extremely good one having now studied it carefully, quite easy to process in Python; I truly hope it will become a widely-accepted standard.

1) There are some regularly used properties that I don't see documented in draft 7.1 of the 1.0 spec; can you define them for me?  They are: XISF:BlockAlignmentSize, XISF:MaxInlineBlockSize, and XISF:OutputHints?

2) The image attribute 'imageType' does not seem to be being used, is there a plan to do so?   I think it would also be useful to add types for various intermediate preprocessing images such as calibrated & registered frames.

3) The astronomy-related and processing history properties appear to be being mapped to and included only as FITSKeyword nodes, not Property nodes.  Is that intentional and expected to be continued?

I'm writing this primarily for an image-cataloging app I'm developing, but plan to contribute it to Astropy (http://www.astropy.org) or scikit-mage (http://scikit-image.org) at some point; I'll post it here as well.

Thanks,
John.
Tak FSQ-106EDX refractor, Optec focuser, 10Micron GM1000HPS mount, QHY9M + 2" filter wheel, Alnitak FlipFlat, Baader LRGB & Astrodon 3nm narrowband filters; Per's ModelMaker, SGP, PixInsight, LRCS, Photoshop

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6040
    • View Profile
    • http://pixinsight.com/
Re: Questions about XISF format as used by PI
« Reply #1 on: 2016 November 29 08:58:08 »
Hi John,

Thank you so much for your appreciation of the XISF initiative. I am glad to know you are working on a Python-based decoder; that's great news!

Quote
1) There are some regularly used properties that I don't see documented in draft 7.1 of the 1.0 spec; can you define them for me?  They are: XISF:BlockAlignmentSize, XISF:MaxInlineBlockSize, and XISF:OutputHints?

These properties are not part of the standard specification. They reflect some particularities of the current implementation of XISF in PixInsight. XISF:BlockAlignmentSize is the minimum size in bytes of an (uncompressed) attached XISF data block. When a block has less bytes than this value, the remaining space is filled with zeros and creates a unused space gap in the XISF structure. This allows us to optimize uncompressed XISF files for fast random block access by storing each block at the beginning of a physical disk sector. The default value is obviously 4096 bytes, assuming an Advanced Format in use. The XISF:MaxInlineBlockSize is the maximum size in bytes of an inline (or embedded) data block. Blocks larger than this value always have attachment location type. The default value is 3072 bytes (which is the length of a block of 4096 bytes when represented in Base64). Finally, XISF:OutputHints stores the sequence of output format hints used when the file was generated. Note that all of these properties reflect idiosyncrasies of the PixInsight platform, so they are not and won't be part of the standard specification. Their names should be respected and not used, however, by third-party encoders for obvious practical reasons, unless they replicate exactly the same functionality that these properties represent in PixInsight.

Quote
2) The image attribute 'imageType' does not seem to be being used, is there a plan to do so?   I think it would also be useful to add types for various intermediate preprocessing images such as calibrated & registered frames.

The current implementation of XISF in PixInsight has several limitations, mainly reflecting limitations in the PixInsight Core application and the PCL development framework. The lack of imageType Image element attributes is one of them. This, along with many more lacking features, will be fixed in incoming versions. *Please* don't hesitate to contribute with any suggestions and ideas you have to improve the XISF specification. I plan on releasing a final 1.0 XISF specification document before the end of the year, if possible, so all contributions will be very welcomed.

Quote
3) The astronomy-related and processing history properties appear to be being mapped to and included only as FITSKeyword nodes, not Property nodes.  Is that intentional and expected to be continued?

This is another limitation of the current XISF implementation, and undoubtedly the most important one. Unfortunately, the entire PixInsight platform still has a strong dependency on FITS header keywords. This includes hundreds of essential properties required and propagated by countless tools and scripts, such as acquisition conditions, astrometric properties, noise estimates, image weights, etc. Overcoming this dependency is one of my priorities. Hopefully, somewhere in 2017 we won't depend on FITS keywords anymore. All image metadata will be generated as XISF properties in XISF files. Generation of FITS keywords will be optional and of course disabled by default. My goal is achieving a FITS-free platform as soon as possible. Fortunately, an automatic FITS-keyword-to-XISF-property converter is not very difficult to implement and would ease the transition, although it requires a lot of work.

Quote
I'm writing this primarily for an image-cataloging app I'm developing, but plan to contribute it to Astropy (http://www.astropy.org) or scikit-mage (http://scikit-image.org) at some point; I'll post it here as well.

That's exciting news! Please keep us informed, and let me know if you need further assistance.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline johnw3d

  • Newcomer
  • Posts: 19
    • View Profile
    • Flickr Astrophoto set
Re: Questions about XISF format as used by PI
« Reply #2 on: 2016 November 29 21:31:20 »
Thanks very much for the response, Juan.

Understand now about the blocksize & alignment properties - I presume the length field in an attachment location does not include the filler gap, right?   If so, I can safely ignore them for now.

Regarding metadata, I'll make my app look for XISF astronomy properties but otherwise honor FITS properties if they are present.  As part of my cataloging app, I am trying to build a canonical metadata hierarchy, similar to that which you have in the XISF spec, so I'll make a comparison soon and send along any suggestions.   I had trouble finding a good, comprehensive public hierarchy to model mine on, perhaps your spec is a worthy instance.

BTW, I've got a first version of the decoder going, it wound up being quite easy in Python, a couple of pages of code.  I need to fill it out a bit and put a decent API around it for public consumption, but will hopefully do that soon (particularly if the cloudy nights continue here in Silicon Valley ;=]).

Thanks again,
John.

Tak FSQ-106EDX refractor, Optec focuser, 10Micron GM1000HPS mount, QHY9M + 2" filter wheel, Alnitak FlipFlat, Baader LRGB & Astrodon 3nm narrowband filters; Per's ModelMaker, SGP, PixInsight, LRCS, Photoshop

Offline jv

  • Newcomer
  • Posts: 14
    • View Profile
Re: Questions about XISF format as used by PI
« Reply #3 on: 2017 September 17 12:36:50 »
Hello.

I am very interested in an XISF decoder for Python 3. 

Has there been any progress on this effort?

Thanks.

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2110
    • View Profile
Re: Questions about XISF format as used by PI
« Reply #4 on: 2017 September 17 18:57:25 »
I usually let PI convert to FITS, do the python work, then PI convert back to XISF.
Not ideal.
Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline jv

  • Newcomer
  • Posts: 14
    • View Profile
Re: Questions about XISF format as used by PI
« Reply #5 on: 2017 September 18 00:47:31 »
I usually let PI convert to FITS, do the python work, then PI convert back to XISF.

Thanks for the suggestion; however, I am interested in being able to read XISF files from Python 3.

__ jim __