What can I compress without much impact (xisf)?

L

Linwood

Well-known member
Large frame camera (Sony A7Riv about 62mpix, raw file size 119MB fits or arw either).

If I compress all the library files - master bias, master dark, Master flat -- that are used as input to calibration runs, is Pixinsight always smart about decompressing, and does it only once?

It's not a huge savings but every little bit helps.

So original lights -- a FITS file is about 119MB -- If I convert to xisf (still at 16bit) and compress it's less than half of that. If I'm doing that for archival purposes, worst case is a single decompression cost when I start over with calibration (I'd probably skip blink as already having been done and discarded those).

Aside: I was excited to see NINA would do the compression, and was thinking the one-time price during calibration was acceptable to start compressed and let NINA spend time while otherwise waiting during exposure, but then found out NINA waits between exposures while compressing. Sigh. So not doing that.

I'm assuming all the intermediate steps (which I do not keep), debayer, registered, approved, etc. should never be compressed.

And the zlib/max/byte shuffle compression is lossless. Very slow, but best compression lossless.

So do I have that right so far?

The final piece of the puzzle really isn't a pixinsight question but curious if anyone knows.

The uncompressed raw file from the camera is the same size as the FIT file captured from APT or NINA. The FIT file has the 14 bit camera raw in 16 bit integers. The ARW raw from the camera I WOULD HAVE ASSUMED was in 14 bit chunks inside the file. But the ARW is almost exactly the same size as the FIT file. Does that mean Sony is also stuffing 14 bit words inside 16 bit storage, wasting that space on the cards (etc)?
 
generally speaking compression should be OK everywhere with one exception - the files you put into ImageIntegration. a compressed XISF file must be read into memory in its entirety and decompressed. an uncompressed XISF file can be incrementally read from disk.

this means that ImageIntegration can keep only the slice of the stack it is working on at the moment in memory when using uncompressed files. if you use compressed XISF files II has to read all the input files into memory, then decompress them into memory, which might exceed how much RAM you have.

as for your last question, i think since computers have been 8-bit byte oriented for a very long time that everyone just uses 16 bit integers to store 14-bit data. if you were to pack the data in the file as 14-bit quantities you'd have to unpack it back to 16-bit integers before trying to use the data which would have a huge performance penalty.

rob
 
Thanks. And wow, I compressed a directory of dark masters and it was about 5 minutes per file. The nice thing is decompression is almost instant. Not that masters take up all that much room, but every bit helps. And if I start keeping lights, I think it will be even more helpful.
 
I did an experiment in case it is of interest to anyone using N.I.N.A. which, for ASCOM at least, will allow you to capture in xisf format and set (or not) compression.

I did some test 1s shots with a A7Riv image (62 megapixels) to see how adding compression affected time. Byte shuffling was on.

No compression5s total time (w1s exposure) 119 megabytes
LZ45s total time (neg difference)74 megabytes (38%)
LZ4HC6s 67 megabytes (44%)
zlib76s49 megabytes (59%)

The download via USB3 through all the layers in the ASCOM driver takes about 3-4 seconds, that's the reason for a 1s sub to take 5s as a base.

This implies there is very little overhead in saving a fair amount of space during capture (essentially none for LZ4), which might be relevant if you are using a outside laptop that is not huge. For me it's even more interesting as I periodically grab a sub over wifi to inspect -- that's actually quite slow given I'm at the outer edge of my wifi outside, and this makes it about twice as fast to grab.

Note these were random shots of a wall indoors -- compression is a function of content so it might vary with lights, darks, etc. in real life.
 
I am having big problems with compressed xisf files "accidentally" getting into ImageIntegration and "bringing my system to its knees."

Trying to debug why this is happening.

Would somebody please explain how to uncompress thousands of compressed xisf files, I guess using the Batch Format Conversion script?

In particular, I am looking for the magic output format hint to convert my files to plain-jane-uncompressed xisf before running them into ImageIntegration.
 
You must log in or register to reply here.
Back
Top