XISF compression from ImageCalibration

[niccoc1603]

Hi, is it possible to output compressed XISF from the image calibration module?

[Juan Conejero]

Sure it's possible. For example, to use Zlib compression, specify the following in the output hints field (Format Hints section):

compression-codec zlib+sh

LZ4HC is an alternative compression algorithm, much faster than Zlib for decompression with slightly lower compression ratios:

compression-codec lz4hc+sh

You can do the same with all tools that provide an output hints parameter. For a list of all format hints available, open the Format Explorer window, select the desired format, and double click the implementation tree item.

[niccoc1603]

Thanks Juan, this is fantastic, is the LZ4HC also lossless?

[Juan Conejero]

Just a word of caution: If you feed the ImageIntegration tool with compressed XISF files, be aware that image compression disables XISF's incremental file reading feature. In other words, all compressed images will have to be decompressed automatically prior to integration, and hence all of them will have to be loaded entirely in memory. This may increase memory consumption very significantly.

[niccoc1603]

Hmm ok, so you suggest not compressing them before image integration?

[Juan Conejero]

Yes, LZ4 and its HC variant are lossless compression algorithms. The main advantage of LZ4 codecs is super fast decompression. See LZ4's website to learn more about these fantastic algorithms by Yann Collet:

https://lz4.github.io/lz4/

With LZ4HC you'll get somewhat worse compression ratios than Zlib, but the huge improvement in decompression times usually more than compensates for that.

LZ4 is an essential component of the PixInsight platform. For example, it is key for real-time decompression in the new star database format that we are developing right now (which will allow us to plate solve images without the need for online catalogs, among other important things).

[Juan Conejero]

Hmm ok, so you suggest not compressing them before image integration?

In general, yes, especially if you don't have XXL-sized RAM installed on your machine.

If you have to save storage space, XISF compression can be a good option during the rest of the image preprocessing process, but I don't recommend it for integration. If you use drizzle to generate your final master light frames (and you should use it to get the most out of your data), then registered images are just temporary working images that you can delete and regenerate very easily if required.

[niccoc1603]

thank you Juan

for example I could store my original lights as compressed XISF, and then if I have to re-process them, I can simply not use compression in the intermediary files (CosmeticCorrection, StarAlign) before I get to Integration. If this makes sense

[pfile]

yes - the other day i accidentally created some absolutely enormous mosaic panels - like 30GB each. without thinking i loaded them into blink... half an hour later they were loaded but as they were loading i began to panic thinking that i would run out of backing store. eventually i realized that the files had just been memory-mapped into PIs address space, so even though the images were charged to PIs memory footprint there was no other effect on the system. i think this would have ended in tragedy if the files had compression.

rob

[niccoc1603]

Sorry I am a bit confused now, do you suggest storing the original light files in a compressed format or not?

[pfile]

i am only saying that what juan is warning against is true - trying to load huge compressed XISF files into PI may bring your system to its knees.

for averaged sized files (100-200MB) you are probably OK with the compressed files. i think your strategy of archiving with compression but leaving compression off for production of intermediate files is a good one.

rob

[Jason]

Just a word of caution: If you feed the ImageIntegration tool with compressed XISF files, be aware that image compression disables XISF's incremental file reading feature. In other words, all compressed images will have to be decompressed automatically prior to integration, and hence all of them will have to be loaded entirely in memory. This may increase memory consumption very significantly.

Nothing stops progressive or more intelligent loading with tiled image compression.  You still get pretty much the same compression ratios but you decode only the tiles you want with fixes the problem you are talking about.  These are not mutually exclusive.

[Juan Conejero]

I agree. The only problem here is that tiled compression has not been implemented yet in our XISF implementation (the XISF standard specification supports it via compression subblocks). This is already in our to-do list. This task has low priority though, since its practical importance is marginal in our preprocessing pipelines.