Big problem with GradientMergeMosaic

[RBA]

With a 16GB RAM PC I ran GradientMergeMosaic on a large 16 frames mosaic... It took well over 20 hours.

That's OK.

The results had a few small issues, but that's OK.

I then saved the 64float TIFF image. Ok...

Closed PixInsight, opened it again, opened the TIFF image, and it's ALL ZEROS.

Opening the TIFF file I get a rescaling message, not sure if that's related or if it happens regularly:

Reading image:
C:/00_Astro/PDC/MaxMos/MergeMosaic.tif
Reading TIFF: 64-bit floating point, 3 channel(s), 15327x14994 pixels, chunky: 100%
Rescaling sample values to [0.00000000e+000,1.00000000e+000]: 100%


Have I lost all the data? Is there a bug when saving 64float TIFF files perhaps?
I'm not very keen on the idea of starting another 20-hours GradientMergeMosaic process.

[georg.viehoever]

No idea. I think it is unlikely that it is a GradientMergeMosaic problem. Maybe the TIFF writer/reader? Did you try writing 64 bit tiff files with some smaller image?
Georg

[Nocturnal]

Yikes!

[RBA]

No idea. I think it is unlikely that it is a GradientMergeMosaic problem. Maybe the TIFF writer/reader? Did you try writing 64 bit tiff files with some smaller image?

Yes, a very reduced version (1500x1467 pixels, versus over 15000x14994 the original) at 32float - this one saved and opens fine,but of couse it's usueless.

[Juan Conejero]

Let's see...

15,327*14,994*3*8 = 5,517,352,152 bytes = 5.138 GB

Unfortunately, the TIFF format uses 32-bit file offsets internally, and hence is limited to slightly less than 4 GB of pixel data. There is an unofficial BigTIFF format out there but the current TIFF support module implements the TIFF version 6 standard (LibTIFF library).

Why a 64-bit format? Why not 32-bit floating point, which would allow you to stay far away from the 4 GB limit? And why not the FITS format (which does not have these size limitations) instead of TIFF?

Obviously, there is a bug in the fact that PI shouldn't allow you to generate a TIFF file larger than 4 GB. I'll fix this in the next version.

Sorry for the wasted (almost) day. On a side note, to work with such huge mosaics you need a beefy machine. I'd recommend a minimum of 24 or 32 GB of RAM and a latest-generation i7-extreme or Xeon processor.

[RBA]

I saved it as 64float simply because that's how it was generated, no special interest in working at that it depth, particularly in this case. FITS/TIFF... I work with either at certain stages of post-processing (usually shifting to TIFF when I know I have no use whatsoever of any if the FITS header info.

Regarding beefy machine... Clearly memory was the bottleneck here. Unfortunately, I have to process this data and that's the machine I have here. I could change the 4x4GB modules for 4x8GB ones (and upgrade from Win7x64 Home to the "pro" version, or build me a Linux box or something), but at this point I have to deal with what I have....

Yes, it'd be good if PixInsight would warn us when the file is about to save won't be usable afterwards 

I might just split the mosaic into two and re-do GradientMergeMosaic... then merge the two "subfinal" images. Might go faster that way -- it all depends on how much memory the module needs times the size of the images....

[Nocturnal]

Rogelio,

there is no (or no substantial) difference in how different W7-x64 versions deal with memory. In any case, while it's cranking away, is it running out of memory or is it CPU bound? Disk bound maybe? The key to performance improvements is to identify the bottleneck and addressing it. Adding memory or changing the OS when neither is the problem will only make you cry :-)

[georg.viehoever]

Rogelio,
there is no (or no substantial) difference in how different W7-x64 versions deal with memory. ....

Win7 Home Premium is limited to 16 GB, see http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778%28v=vs.85%29.aspx

Georg

[RBA]

there is no (or no substantial) difference in how different W7-x64 versions deal with memory. In any case, while it's cranking away, is it running out of memory or is it CPU bound? Disk bound maybe? The key to performance improvements is to identify the bottleneck and addressing it. Adding memory or changing the OS when neither is the problem will only make you cry :-)

I believed the "Home" version can only address 16GB of RAM, while the "pro" version (Ultimate or however the official name is) doesn't have this restriction. Isn't that correct?

This was a memory issue because the RAM was being at 100% usage all the time (and the hard drive humming like nuts - paging memory I suppose) while the 4 CPU cores (it's an Intel i5 @ 2.67GHz)  were pretty much unoccupied...

[Nocturnal]

Thanks for the correction guys. Looks like Pro and more memory would help in this case.