As noted earlier in this thread, swap files are used to store image states, as part of the processing history infrastructure implemented in PixInsight. As you know, PixInsight supports unlimited undo/redo operations (only limited by the available storage resources).
When I first designed and implemented the current processing history model in PixInsight (back in 2005), I decided to keep image states stored as external files instead of dynamic memory blocks. Only pixel data are stored externally; all of the metadata associated with image states, including serialized processes and most image properties, as well as many auxiliary data objects (such as ICC profiles, color spaces and other image data with predictable/fixed storage requirements) are stored in memory.
The basic design principles behind this decision are rather simple: (1) favor usage of free RAM for pure image processing tasks, and (2) keep the whole processing history architecture as simple as possible structurally. Some advantages of storing image states as external files are:
*
Preserve free RAM for processing tasks. Many image processing tasks and algorithms are very demanding in their memory requirements. For example, each execution of the TGVDenoise process requires internally 11 duplicates of a single channel of the target image. Multiscale analysis algorithms, such as the starlet transform, the multiscale median transform, etc, also need many duplicates of the target image (basically, one duplicate per multiscale layer, but of the whole image, not just a single channel). Most multiscale processing tools maintain a cache of previously performed transforms, which requires a lot of available memory. Future implementations of many tools will also use multiple cached data to accelerate generation of intermediate transforms and auxiliary structures. Expect memory requirements to increase dramatically as we improve and extend the current tool set.
*
Prevent memory fragmentation. Fragmentation is a serious problem inherent to all software implementations that perform continued allocations and deallocations of dynamic memory blocks of varying sizes. The whole PixInsight platform has been designed to minimize fragmentation, including internal memory pools and other allocation devices. External swap file storage plays a key role in this optimization.
*
Coexist well with other software, including other running instances of PixInsight. Image states can easily consume several gigabytes in normal working scenarios (much more for complex processing works). If all of these data blocks are maintained in RAM, this can cause problems to run other applications, the operating system kernel and system services, which in turn can have a severe impact on overall system performance.
*
Robustness. A swap file allows recovering an image state in the event of a process error, even after an application crash.
*
Modularity. External storage of swap files implement a
divide and conquer strategy, which facilitates the implementation of multiple tasks around the same data structures. For example, PixInsight projects use exactly the same swap file format to store image states. In fact, when you save a project all swap files are simply copied from one directory to another. The same happens, in the opposite direction, when you load a project.
PixInsight has been conceived and designed to handle huge image processing projects. The true fact is that, even with the reasonably large RAM sizes available on contemporary high-end workstation machines (32 to 128 GiB), RAM is a precious resource that has to be used judiciously. This not only involves availability of large contiguous blocks for image processing in PixInsight, but also availability of free RAM to run other demanding applications and tasks concurrently. Wasting RAM to keep unused data blocks 'simply stored' on a regular basis is not the most efficient implementation IMO.
With modern hardware, such as solid state disk drives, and considering the flexibility of our implementation, the performance penalty incurred by external swap files is negligible or virtually nonexistent. On the Linux reference implementation, for example, the system /tmp directory used by default for storage of swap files uses tmpfs (a standard file system with dynamic RAM storage) by default on all modern Linux distributions. Even when using external mass storage devices, the Linux kernel implements such an efficient disk caching system that generation of huge swap files is virtually transparent. See for example
this benchmark, where both tmpfs and an SSD drive achieve a respectable transfer rate of 5.5 GiB/s. On OS X and Windows file system performance is worse, but also reasonably efficient on modern machines.
The current processing history architecture, based on external storage of pixel data, is efficient and robust. It could be extended by allowing optional storage in RAM, at least partially (for example, just the latest image state for fast single undo/redo operations), but I don't see any urgency in this.
