XISF decompression failed on lz4

[hvb356]

Dear Juan,

for some reasons, I need to process XISF formatted files from a Windows forms application. Reading uncompressed data as well as Zlib compressed data with or without byte shuffling works as expected.

My attempts to get lz4 or lz4-HC decompression to work failed. lz4 or lz4-HC are in fact plain copies without a compression attribute. The metadata compression method is correctly stored.

lz4+sh works as promised.

Regards

Hartmut

 

[Juan Conejero]

Hi Hartmut,

Using the current version of the XISF support module, I can't reproduce any errors with LZ4 and LZ4-HC compression. Can you be more specific? How are you using XISF files from Windows forms?

 

[hvb356]

Hi Juan,

>  How are you using XISF files from Windows forms?

by reading the file with .net XmlReader

I try to give you one example:

PixInsight (fit) file copy:

FileSize 33,318,720 bytes

Save as XISF, LZ4, unchecked Byte shuffling
===========================================

FileSize 33,317,968 bytes (248 bytes more compared to the uncompressed file!)

Reading this file returned

Attributes

bounds 0:1
checksum nothing
colorSpace Gray
compression nothing
geometry 3326:2504:1
location attachment:4752:33313216
sampleFormat Float32

Metadata

CreationTime 2017-04-04T09:22:10Z
CreatorApplication PixInsight 01.08.04.1195
CreatorModule XISF module version 01.00.06.0107
CreatorOS Windows
BlockAlignmentSize nothing
MaxInlineBlockSize nothing
CompressionMethod lz4
CompressionLevel 64
AbstractCompressionLevel 0
ChecksumMethod nothing


Save as XISF, LZ4+shuffling
===========================

FileSize 25,371,861

Attributes

bounds 0:1
checksum nothing
colorSpace Gray
compression lz4+sh:33313216:4
geometry 3326:2504:1
location attachment:4787:25367074
sampleFormat Float32

Metadata

CreationTime 2017-04-04T09:26:08Z
CreatorApplication PixInsight 01.08.04.1195
CreatorModule XISF module version 01.00.06.0107
CreatorOS Windows
BlockAlignmentSize nothing
MaxInlineBlockSize nothing
CompressionMethod lz4+sh
CompressionLevel 64
AbstractCompressionLevel 0
ChecksumMethod nothing

This is, what I see.

Best regards

Hartmut

 

[Juan Conejero]

This is a perfectly normal result. A lossless data compression algorithm attempts to reduce the required storage space for a given data set, but that goal is not necessarily achievable [1]. Sometimes a data block is not compressible with a particular algorithm. When this happens, the XISF support module ignores compression parameters and just stores the uncompressed block.

The Zlib codec implements a lossless compression algorithm based on entropy analysis and reduction techniques. Zlib is extremely efficient and rarely fails to compress a data block. LZ4, on the other hand, works by detecting data redundancies directly on the data stream. This is much faster than entropy analysis, but is also less efficient in compression ratio terms. With highly entropic data (high noise contents, for example), plain LZ4 compression can fail to compress a block with relative ease. LZ4-HC is the "high compression" variant of LZ4, which usually achieves somewhat smaller compression ratios than Zlib, but is faster, especially for decompression.

The byte shuffling algorithm increments data locality, that is, data items are redistributed so that similar values tend to be placed close together. This greatly improves compressibility, so all compression codecs, including LZ4, can usually work much better with byte-shuffled data.


[1] Pedantic mode on - The proof of this is done by reduction to the absurd. If a data compression algorithm could guarantee that any data set can be compressed, even by reducing its size by one bit, then any data set could be compressed ad infinitum, that is, by compressing the result of a previous compression. Such data set would eventually require zero bytes of storage space - pedantic mode off.

 

[hvb356]

Hi Juan,

that's a perfect answer. So, if the attributes say compression=nothing, I can simply take the buffer 'as is'.

Best regards and thank's for the help

Hartmut