Abstract base class of data compression algorithm implementations. More...

#include <Compression.h>

Inheritance diagram for pcl::Compression:

Classes
struct	Performance
	Compression/decompression performance measurements. More...

struct	Subblock
	Compression subblock data. More...

Public Types
using	subblock_list = Array< Subblock >

Public Member Functions
	Compression ()=default

	Compression (const Compression &)=default

	~Compression () override

virtual String	AlgorithmName () const =0

bool	ByteShufflingEnabled () const

bool	ChecksumsEnabled () const

template<class C >
subblock_list	Compress (const C &data, Performance *perf=nullptr) const

subblock_list	Compress (const void data, size_type size, Performance perf=nullptr) const

int	CompressionLevel () const

virtual int	DefaultCompressionLevel () const =0

void	DisableByteShuffling (bool disable=true)

void	DisableChecksums (bool disable=true)

void	EnableByteShuffling (bool enable=true)

void	EnableChecksums (bool enable=true)

size_type	ItemSize () const

virtual int	MaxCompressionLevel () const =0

virtual int	MinCompressionLevel () const

Compression &	operator= (const Compression &)=default

void	SetCompressionLevel (int level)

void	SetDefaultCompressionLevel ()

void	SetItemSize (size_type itemSize)

void	SetSubblockSize (size_type size)

size_type	SubblockSize () const

ByteArray	Uncompress (const ByteArray &compressedData, size_type uncompressedSize, Performance *perf=nullptr) const

ByteArray	Uncompress (const subblock_list &subblocks, Performance *perf=nullptr) const

size_type	Uncompress (void data, size_type maxSize, const subblock_list &subblocks, Performance perf=nullptr) const

Public Member Functions inherited from pcl::ParallelProcess
	ParallelProcess ()=default

	ParallelProcess (const ParallelProcess &)=default

virtual	~ParallelProcess ()

void	DisableParallelProcessing (bool disable=true) noexcept

void	EnableParallelProcessing (bool enable=true, int maxProcessors=0) noexcept

bool	IsParallelProcessingEnabled () const noexcept

int	MaxProcessors () const noexcept

ParallelProcess &	operator= (const ParallelProcess &)=default

void	SetMaxProcessors (int maxProcessors) noexcept

void	Swap (ParallelProcess &process) noexcept

Protected Member Functions
virtual size_type	CompressBlock (void outputData, size_type outputSize, const void inputData, size_type inputSize, int level) const =0

virtual size_type	MaxBlockSize () const =0

virtual size_type	MaxCompressedBlockSize (size_type size) const =0

virtual size_type	MinBlockSize () const =0

void	Throw (const String &errorMessage) const

virtual size_type	UncompressBlock (void outputData, size_type outputSize, const void inputData, size_type inputSize) const =0

Static Protected Member Functions
static void	InPlaceUnshuffle (uint8 *data, size_type size, size_type itemSize)

static ByteArray	Shuffle (const uint8 *data, size_type size, size_type itemSize)

static ByteArray	Unshuffle (const uint8 *data, size_type size, size_type itemSize)

Detailed Description

Compression defines a common interface for compression algorithms such as Zlib/deflate, LZ4, LZMA, etc.

See also: ZLibCompression, LZ4Compression, LZ4HCCompression, ZstdCompression

Definition at line 83 of file Compression.h.

Member Typedef Documentation

◆ subblock_list

using pcl::Compression::subblock_list = Array<Subblock>

A dynamic, ordered list of compression subblocks.

Definition at line 112 of file Compression.h.

Constructor & Destructor Documentation

◆ Compression() [1/2]

pcl::Compression::Compression ( )

default

Default constructor. This object will be initialized with the following parameters:

Compression level: Use an algorithm-dependent default level to achieve a good compromise between running speed and compression ratio.

Byte shuffling: Enabled.

Item size: One byte (which inhibits shuffling).

Sub-block checksums: Enabled.

Sub-block size: Use the maximum possible subblock size for each compression algorithm.

Parallel processing: Enabled to use the maximum possible number of processors.

◆ Compression() [2/2]

pcl::Compression::Compression ( const Compression & )

default

Copy constructor.

◆ ~Compression()

pcl::Compression::~Compression ( )

inlineoverride

Virtual destructor.

Definition at line 148 of file Compression.h.

Member Function Documentation

◆ AlgorithmName()

virtual String pcl::Compression::AlgorithmName ( ) const

pure virtual

Returns the name of this data compression algorithm.

Implemented in pcl::ZstdCompression, pcl::LZ4HCCompression, pcl::LZ4Compression, and pcl::ZLibCompression.

◆ ByteShufflingEnabled()

bool pcl::Compression::ByteShufflingEnabled ( ) const

inline

Returns true iff byte shuffling is currently enabled for this Compression object.

See EnableByteShuffling() for more information.

Definition at line 223 of file Compression.h.

◆ ChecksumsEnabled()

bool pcl::Compression::ChecksumsEnabled ( ) const

inline

Returns true iff subblock checksums are currently enabled for this Compression object.

See EnableChecksums() for more information.

Definition at line 310 of file Compression.h.

◆ Compress() [1/2]

template<class C >

subblock_list pcl::Compression::Compress	(	const C &	data,
		Performance *	perf = `nullptr`
	)		const

inline

Compression of a direct container with contiguous storage.

Parameters

data	Reference to a direct container with contiguous storage and PCL container semantics. Meaningful Begin() and Size() member functions are required. Typically, the ByteArray class is used, although any array-like direct container class is appropriate, including Array, String, Vector and Matrix, among others.
perf	If non-null, pointer to a Performance structure where performance data will be provided.

Returns a dynamic array of compressed subblocks. Each array element is a Subblock structure with the compressed data, the uncompressed length in bytes, and an optional checksum. If compression succeeds, the returned array will have at least one element.

If the specified container is empty, or if one or more subblocks are not compressible with this algorithm, this function returns an empty array.

This function can only throw an exception in the event of an out-of-memory condition.

Definition at line 392 of file Compression.h.

◆ Compress() [2/2]

subblock_list pcl::Compression::Compress	(	const void *	data,
		size_type	size,
		Performance *	perf = `nullptr`
	)		const

Compresses a data block.

Parameters

data	Starting address of the data block to be compressed.
size	Length in bytes of the data block to be compressed.
perf	If non-null, pointer to a Performance structure where performance data will be provided.

Returns a dynamic array of compressed subblocks. Each array element is a Subblock structure with the compressed data, the uncompressed length in bytes, and an optional checksum. If compression succeeds, the returned array will have at least one element.

If the specified size is zero, or if one or more subblocks are not compressible with this algorithm, this function returns an empty array.

This function can only throw an exception if the specified data pointer is null, or in the event of an out-of-memory condition.

◆ CompressBlock()

virtual size_type pcl::Compression::CompressBlock	(	void *	outputData,
		size_type	outputSize,
		const void *	inputData,
		size_type	inputSize,
		int	level
	)		const

protectedpure virtual

Compression of a contiguous block starting at inputData with inputSize length in bytes, using the specified compression level. The compressed data will be written as a contiguous block starting at outputData, with length smaller than or equal to outputSize. Returns the number of bytes written to outputData, or zero if the data are not compressible.

◆ CompressionLevel()

int pcl::Compression::CompressionLevel ( ) const

inline

Returns the current compression optimization level.

See SetCompressionLevel() for more information.

Definition at line 185 of file Compression.h.

◆ DefaultCompressionLevel()

virtual int pcl::Compression::DefaultCompressionLevel ( ) const

pure virtual

Returns the default compression optimization level for this algorithm. The returned value represents a good compromise between execution speed and compression ratio.

Implemented in pcl::ZstdCompression, pcl::LZ4HCCompression, pcl::LZ4Compression, and pcl::ZLibCompression.

◆ DisableByteShuffling()

void pcl::Compression::DisableByteShuffling ( bool disable = true )

inline

Disables byte shuffling for this Compression object.

Equivalent to EnableByteShuffling( !disable ).

Definition at line 251 of file Compression.h.

◆ DisableChecksums()

void pcl::Compression::DisableChecksums ( bool disable = true )

inline

Disables subblock checksums for this Compression object.

Equivalent to EnableChecksums( !disable ).

Definition at line 339 of file Compression.h.

◆ EnableByteShuffling()

void pcl::Compression::EnableByteShuffling ( bool enable = true )

inline

Enables byte shuffling for this Compression object.

Enable this option to pre-process the data with a byte shuffling routine before compression. The byte shuffle algorithm reorganizes the data in a clever way that can greatly improve compression ratios, especially for data with high locality (or data containing similar values close together), such as most scientific and observational data. This option can be particularly efficient for compression of data structured as contiguous sequences of 16-bit, 32-bit and 64-bit numbers, such as most images processed with PixInsight. For 8-bit data, byte shuffling is a no-op and is hence ignored. Byte shuffling is enabled by default.

Definition at line 241 of file Compression.h.

◆ EnableChecksums()

void pcl::Compression::EnableChecksums ( bool enable = true )

inline

Enables subblock checksums for this Compression object.

When this option is enabled, a non-cryptographic checksum is computed for each compressed subblock and stored along with the subblock data. These checksums are computed again before decompression and compared with the stored values. This guarantees that the compressed data have not been altered.

In current versions of PixInsight, the xxHash algorithm (64-bit version) is used for compressed subblock checksums. xxHash is an extremely fast hashing function, so its impact on compression/decompression speed is negligible. See the pcl::Hash64() function for details.

Definition at line 329 of file Compression.h.

◆ InPlaceUnshuffle()

static void pcl::Compression::InPlaceUnshuffle	(	uint8 *	data,
		size_type	size,
		size_type	itemSize
	)

inlinestaticprotected

In-place reverse byte shuffling algorithm (or unshuffling) applied to size bytes starting at data, with element length itemSize in bytes.

Definition at line 573 of file Compression.h.

References pcl::Array< T, A >::Begin().

◆ ItemSize()

size_type pcl::Compression::ItemSize ( ) const

inline

Returns the current item size for shuffling.

This is the length in bytes of an element in the uncompressed data, used by the byte shuffling algorithm. See EnableByteShuffling() for more information.

Definition at line 263 of file Compression.h.

◆ MaxBlockSize()

virtual size_type pcl::Compression::MaxBlockSize ( ) const

protectedpure virtual

Returns the length in bytes of the largest contiguous data block that can be compressed with this algorithm.

◆ MaxCompressedBlockSize()

virtual size_type pcl::Compression::MaxCompressedBlockSize ( size_type size ) const

protectedpure virtual

Returns the maximum length in bytes of the contiguous block necessary to compress a contiguous block of the specified size in bytes.

◆ MaxCompressionLevel()

virtual int pcl::Compression::MaxCompressionLevel ( ) const

pure virtual

Returns the maximum compression optimization level supported by this algorithm. Higher levels tend to provide higher compression ratios at the cost of more computational work.

Implemented in pcl::ZstdCompression, pcl::LZ4HCCompression, pcl::LZ4Compression, and pcl::ZLibCompression.

◆ MinBlockSize()

virtual size_type pcl::Compression::MinBlockSize ( ) const

protectedpure virtual

Returns the length in bytes of the smallest contiguous data block that can be compressed with this algorithm.

◆ MinCompressionLevel()

virtual int pcl::Compression::MinCompressionLevel ( ) const

inlinevirtual

Returns the minimum compression optimization level supported by this algorithm. The default implementation returns zero.

Reimplemented in pcl::ZstdCompression.

Definition at line 161 of file Compression.h.

◆ operator=()

Compression& pcl::Compression::operator= ( const Compression & )

default

Copy assignment operator. Returns a reference to this object.

◆ SetCompressionLevel()

void pcl::Compression::SetCompressionLevel ( int level )

inline

Sets an algorithm-dependent compression optimization level.

Higher levels tend to provide higher compression ratios at the cost of more computational work. If zero or a negative value is specified, the compression routines will select a default level automatically to achieve a good compromise between speed and compression ratio for each compression algorithm.

Definition at line 199 of file Compression.h.

References pcl::Range().

◆ SetDefaultCompressionLevel()

void pcl::Compression::SetDefaultCompressionLevel ( )

inline

Sets the default compression optimization level for this compression algorithm.

This function is a shortcut for SetCompressionLevel( 0 ). The default compression level attempts to achieve a good compromise between speed and compression ratio for each compression algorithm.

Definition at line 212 of file Compression.h.

◆ SetItemSize()

void pcl::Compression::SetItemSize ( size_type itemSize )

inline

Sets the item size for shuffling.

itemSize is the length in bytes of a data item for the byte shuffling algorithm. See EnableByteShuffling() for more information.

Definition at line 274 of file Compression.h.

References pcl::Max().

◆ SetSubblockSize()

void pcl::Compression::SetSubblockSize ( size_type size )

inline

Sets the compression subblock size.

The specified size is the length in bytes of a compression subblock. If a value of zero is specified, the effective block size will be the maximum possible for this compression algorithm (for example, slightly less than 4 GiB for zlib/deflate and slightly less than 2 GiB for LZ4 and BloscLZ). The number of elements in the arrays returned by the Compress() family of functions will depend on this value.

Definition at line 299 of file Compression.h.

References pcl::Range().

◆ Shuffle()

static ByteArray pcl::Compression::Shuffle	(	const uint8 *	data,
		size_type	size,
		size_type	itemSize
	)

inlinestaticprotected

Byte shuffling algorithm applied to size bytes starting at data, with element length itemSize in bytes. Returns the shuffled data as a ByteArray object.

Definition at line 527 of file Compression.h.

References pcl::Array< T, A >::Begin().

◆ SubblockSize()

size_type pcl::Compression::SubblockSize ( ) const

inline

Returns the current compression subblock size in bytes.

See SetSubblockSize() for more information.

Definition at line 284 of file Compression.h.

◆ Throw()

void pcl::Compression::Throw ( const String & errorMessage ) const

protected

Helper function to throw an error message with inclusion of the algorithm name.

◆ Uncompress() [1/3]

ByteArray pcl::Compression::Uncompress	(	const ByteArray &	compressedData,
		size_type	uncompressedSize,
		Performance *	perf = `nullptr`
	)		const

inline

Decompression of a compressed data block stored in a ByteArray object. Returns the uncompressed data as a ByteArray.

See the Uncompress( const subblock_list&, Performance* ) const member function for more information.

Definition at line 464 of file Compression.h.

References pcl::Compression::Subblock::compressedData, and pcl::Compression::Subblock::uncompressedSize.

◆ Uncompress() [2/3]

ByteArray pcl::Compression::Uncompress	(	const subblock_list &	subblocks,
		Performance *	perf = `nullptr`
	)		const

inline

Decompression of a set of compressed subblocks.

Parameters

subblocks	Reference to a dynamic array of compressed subblocks. The specified array should be the result of a previous call to one of the Compress() functions for this algorithm. This object must have coherent ItemSize() and ByteShufflingEnabled() properties to match the ones used when the data was compressed.
perf	If non-null, pointer to a Performance structure where performance data will be provided.

Returns the uncompressed data as a ByteArray object.

In the event of errors, such as invalid or corrupted data, uncompressed block or subblock size mismatch, or an out-of-memory condition, this function throws an Error exception.

Definition at line 447 of file Compression.h.

References pcl::Array< T, A >::Begin().

◆ Uncompress() [3/3]

size_type pcl::Compression::Uncompress	(	void *	data,
		size_type	maxSize,
		const subblock_list &	subblocks,
		Performance *	perf = `nullptr`
	)		const

Decompression of a set of compressed subblocks.

Parameters

data	Starting address of the output uncompressed data block.
maxSize	Maximum space in bytes available at data. Must be equal to or larger than the total uncompressed size.
subblocks	Reference to a dynamic array of compressed subblocks. The specified array should be the result of a previous call to one of the Compress() functions for this algorithm. This object must have coherent ItemSize() and ByteShufflingEnabled() properties to match the ones used when the data was compressed.
perf	If non-null, pointer to a Performance structure where performance data will be provided.

Upon successful completion, the uncompressed data will be available in the specified output data block, and the function will return the length in bytes of the uncompressed data (necessarily smaller than or equal to maxSize).

In the event of errors, such as invalid or corrupted data, uncompressed block or subblock size mismatch, or an out-of-memory condition, this function throws an Error exception. In such case the data possibly stored in the output data buffer will be invalid.

◆ UncompressBlock()

virtual size_type pcl::Compression::UncompressBlock	(	void *	outputData,
		size_type	outputSize,
		const void *	inputData,
		size_type	inputSize
	)		const

protectedpure virtual

Decompression of a contiguous compressed block starting at inputData with inputSize length in bytes. The uncompressed data will be written as a contiguous block starting at outputData, with length equal to outputSize. Returns the number of bytes written to outputData, which should be equal to outputSize, or smaller in the event of error.

◆ Unshuffle()

static ByteArray pcl::Compression::Unshuffle	(	const uint8 *	data,
		size_type	size,
		size_type	itemSize
	)

inlinestaticprotected

Reverse byte shuffling algorithm (or unshuffling) applied to size bytes starting at data, with element length itemSize in bytes. Returns the unshuffled data as a ByteArray object.

Definition at line 550 of file Compression.h.

References pcl::Array< T, A >::At().

The documentation for this class was generated from the following file:

Compression.h

Classes

Public Types

Public Member Functions

Protected Member Functions

Static Protected Member Functions

Detailed Description

Member Typedef Documentation

◆ subblock_list

Constructor & Destructor Documentation

◆ Compression() [1/2]

◆ Compression() [2/2]

◆ ~Compression()

Member Function Documentation

◆ AlgorithmName()

◆ ByteShufflingEnabled()

◆ ChecksumsEnabled()

◆ Compress() [1/2]

◆ Compress() [2/2]

◆ CompressBlock()

◆ CompressionLevel()

◆ DefaultCompressionLevel()

◆ DisableByteShuffling()

◆ DisableChecksums()

◆ EnableByteShuffling()

◆ EnableChecksums()

◆ InPlaceUnshuffle()

◆ ItemSize()

◆ MaxBlockSize()

◆ MaxCompressedBlockSize()

◆ MaxCompressionLevel()

◆ MinBlockSize()

◆ MinCompressionLevel()

◆ operator=()

◆ SetCompressionLevel()

◆ SetDefaultCompressionLevel()

◆ SetItemSize()

◆ SetSubblockSize()

◆ Shuffle()

◆ SubblockSize()

◆ Throw()

◆ Uncompress() [1/3]

◆ Uncompress() [2/3]

◆ Uncompress() [3/3]

◆ UncompressBlock()

◆ Unshuffle()