Author Topic: XISF decompression failed on lz4  (Read 1384 times)

Offline hvb356

  • Member
  • *
  • Posts: 51
    • View Profile
XISF decompression failed on lz4
« on: 2017 April 02 05:35:42 »
Dear Juan,

for some reasons, I need to process XISF formatted files from a Windows forms application. Reading uncompressed data as well as Zlib compressed data with or without byte shuffling works as expected.

My attempts to get lz4 or lz4-HC decompression to work failed. lz4 or lz4-HC are in fact plain copies without a compression attribute. The metadata compression method is correctly stored.

lz4+sh works as promised.   

Regards

Hartmut
TOA-150, ML8300, CFW2-7 w. Astronomik HaLRGB, PDF, EM-400, guiding FS-60 + ST402, ASCOM bsd. image acquisition

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6619
    • View Profile
    • http://pixinsight.com/
Re: XISF decompression failed on lz4
« Reply #1 on: 2017 April 04 01:20:59 »
Hi Hartmut,

Using the current version of the XISF support module, I can't reproduce any errors with LZ4 and LZ4-HC compression. Can you be more specific? How are you using XISF files from Windows forms?
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline hvb356

  • Member
  • *
  • Posts: 51
    • View Profile
Re: XISF decompression failed on lz4
« Reply #2 on: 2017 April 04 03:01:55 »

Hi Juan,

>  How are you using XISF files from Windows forms?

by reading the file with .net XmlReader

I try to give you one example:
 
PixInsight (fit) file copy:

FileSize   33,318,720 bytes

Save as XISF, LZ4, unchecked Byte shuffling
===========================================

FileSize   33,317,968 bytes (248 bytes more compared to the uncompressed file!)

Reading this file returned

Attributes

bounds   0:1
checksum   nothing
colorSpace   Gray
compression   nothing
geometry   3326:2504:1
location   attachment:4752:33313216
sampleFormat   Float32

Metadata

CreationTime   2017-04-04T09:22:10Z
CreatorApplication   PixInsight 01.08.04.1195
CreatorModule   XISF module version 01.00.06.0107
CreatorOS   Windows
BlockAlignmentSize   nothing
MaxInlineBlockSize   nothing
CompressionMethod   lz4
CompressionLevel   64
AbstractCompressionLevel   0
ChecksumMethod   nothing
 

Save as XISF, LZ4+shuffling
===========================

FileSize   25,371,861

Attributes

bounds   0:1
checksum   nothing
colorSpace   Gray
compression   lz4+sh:33313216:4
geometry   3326:2504:1
location   attachment:4787:25367074
sampleFormat   Float32

Metadata

CreationTime   2017-04-04T09:26:08Z
CreatorApplication   PixInsight 01.08.04.1195
CreatorModule   XISF module version 01.00.06.0107
CreatorOS   Windows
BlockAlignmentSize   nothing
MaxInlineBlockSize   nothing
CompressionMethod   lz4+sh
CompressionLevel   64
AbstractCompressionLevel   0
ChecksumMethod   nothing

This is, what I see.

Best regards

Hartmut

TOA-150, ML8300, CFW2-7 w. Astronomik HaLRGB, PDF, EM-400, guiding FS-60 + ST402, ASCOM bsd. image acquisition

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 6619
    • View Profile
    • http://pixinsight.com/
Re: XISF decompression failed on lz4
« Reply #3 on: 2017 April 04 07:12:08 »
This is a perfectly normal result. A lossless data compression algorithm attempts to reduce the required storage space for a given data set, but that goal is not necessarily achievable [1]. Sometimes a data block is not compressible with a particular algorithm. When this happens, the XISF support module ignores compression parameters and just stores the uncompressed block.

The Zlib codec implements a lossless compression algorithm based on entropy analysis and reduction techniques. Zlib is extremely efficient and rarely fails to compress a data block. LZ4, on the other hand, works by detecting data redundancies directly on the data stream. This is much faster than entropy analysis, but is also less efficient in compression ratio terms. With highly entropic data (high noise contents, for example), plain LZ4 compression can fail to compress a block with relative ease. LZ4-HC is the "high compression" variant of LZ4, which usually achieves somewhat smaller compression ratios than Zlib, but is faster, especially for decompression.

The byte shuffling algorithm increments data locality, that is, data items are redistributed so that similar values tend to be placed close together. This greatly improves compressibility, so all compression codecs, including LZ4, can usually work much better with byte-shuffled data.


[1] Pedantic mode on - The proof of this is done by reduction to the absurd. If a data compression algorithm could guarantee that any data set can be compressed, even by reducing its size by one bit, then any data set could be compressed ad infinitum, that is, by compressing the result of a previous compression. Such data set would eventually require zero bytes of storage space - pedantic mode off.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline hvb356

  • Member
  • *
  • Posts: 51
    • View Profile
Re: XISF decompression failed on lz4
« Reply #4 on: 2017 April 04 09:15:25 »
Hi Juan,

that's a perfect answer. So, if the attributes say compression=nothing, I can simply take the buffer 'as is'.

Best regards and thank's for the help

Hartmut

TOA-150, ML8300, CFW2-7 w. Astronomik HaLRGB, PDF, EM-400, guiding FS-60 + ST402, ASCOM bsd. image acquisition