Opening Large TIF Files

[Bob Novo]

I have been trying to open 2.8GB TIF files into PixInsight and get the following message: PCL TIFF Format Support Error reading TIFF File. Files smaller than 2GB open fine. Is there a fix? Thank you.

[Juan Conejero]

Hi,

TIFF files are limited to 4 GB because the TIFF format uses 32-bit internal pointers. However, to work with images larger than 2 - 2.5 GB you need a 64-bit version of PixInsight running on a 64-bit operating system. You need also a reasonably sized RAM (4 GB minimum). For example, I have just made a test with a very big TIFF file (a grayscale image with 40540x26483 pixels in 32-bit floating point format) and it can be written and read without problems on a 64-bit Linux laptop with 8 GB of RAM.

There is a proposal to extend the TIFF format to 64-bit pointers to allow working with TIFF files larger than 4 GB: BigTIFF. However it is still a nonstandard version and we haven't implemented it yet.

[Bob Novo]

Thank you for your reply. When I first posted my question I was attempting to open a 2.9GB file onto a 64-bit XP machine with 10 GB fo RAM. As I indicated the file failed to open. Later I tried it on a 64 bit Windows 7 Professional laptop with 4GB of RAM with the same result. In both cases I am running the 64 bit version of PixInsight. Please note that the origional images are film based which were scanned. Thank you.
Bobnovo

[Juan Conejero]

Hi Bobnovo,

I can reproduce this problem on Windows 7 x64. This is a Windows-specific issue: it does not happen on FreeBSD, Linux and Mac OS X, where TIFF files up to 4 GB can be written and read normally.

Right now I have no idea why this happens. I'll try to find and fix this bug as soon as possible. Thanks for reporting, and sorry for the inconvenience.

[georg.viehoever]

This may be caused by the fact that Windows uses the IL32P64 model (integer and long in C++ are 32 bits, pointers 64 bits), but Linux uses I32LP64 (integer 32 bits, long and pointer 64 bits) on 64 bit platforms. Quite a lot of Linux software compiles more or less well on Windows, but fails with data that need 64 bit longs (e.g. as offsets). See for example http://en.wikipedia.org/wiki/64-bit#Specific_C-language_data_models . These issues can be difficult to find (usually requires debugging or careful reading of compiler warnings) 

Georg

[Juan Conejero]

Hi Bob,

This bug has been fixed. The corresponding update has been released today for Windows x86 and x64 platforms.

Thanks for reporting, and sorry for the delay in fixing this problem. Please let me know if your large TIFF files can now be loaded correctly in PixInsight.

[georg.viehoever]

I am nosy: What was the root cause?

[Juan Conejero]

It was quite simple actually. LibTIFF calls the standard lseek() POSIX function to move file pointers:

off_t lseek(int fildes, off_t offset, int whence)

On all UNIX and Linux platforms, off_t is nowadays defined as a 64-bit signed integer, which provides support for 64-bit filesystems. However, on Windows off_t is still a 32-bit signed integer, which doesn't allow for file positions (and hence for file sizes) larger than 2 GB. This is a consequence of MS's attempt at deprecating all POSIX standard routines (another absurd attempt at imposing nonstandard rules to break the standards).

The obvious solution is using _lseeki64 in the Windows version, with a custom modification to LibTIFF. Very easy to identify and fix, but my workload has not allowed me to fix this problem as soon as I should...

[georg.viehoever]

Sometimes I could cry . This kind of stuff is causing so much trouble with getting open source stuff running properly on MS 

[Harry page]

Hi

I sometimes have a problem opening a Tiff file saved in photoshop ( sorry it slipped out ) I have uploaded it for you to look at 

Harry

BTW a T shirt would have stopped the P word