File size limit?

[cmassa]

I have been working on a 4 panel mosaic and am playing with drizzle integration. I have run the panels which were stacked with 2x drizzle. I used star alignment to create the mosaic and register the parts to be used with gradientmergemosaic. The files are about 2.5 GB in size, with the mosaic being about 18,000 pixels in the long dimension.

When I try to open the mosaic created by star alignment or open the registered files I get an error message stating (sorry, can't figure out how to post an image here)

PCL FITS Format Support: Unable to open FITS file: mosaic.fit
CFITSIO error mesage stack:
01: failed get the size of the following file: (ffopen) mosaic.fit

I shortened the file name to mosaic.fit for less typing.

The PC is running Windows 7 Professional and has 8GB of RAM.

Is there a limit to the file size which can be opened in PI?

Thanks,
Chris

[cmassa]

Well, I guess it did upload the image.

[Andres.Pozo]

I found this same problem last year and I reported it in this message:
http://pixinsight.com/forum/index.php?topic=6469

[cmassa]

PI saved the file just fine, now it just won't open it.

This would not allow the use of drizzle integration to make mosaics then because the files sizes get big very quickly.

CHris

[Juan Conejero]

This is a specific problem with CFITSIO on Windows. On FreeBSD, Linux and Mac OS X there are no practical file size limits. As an example on the Linux laptop where I'm writing this, I have just written a FITS image with 24000x24000 pixels and three channels in 32-bit floating point format. Then I've closed the image and reloaded the disk FITS file without problems. Note that this image is about 6.5 GB.

CFITSIO is a rather old software library. It has never been designed to be truly portable code in my opinion. I have never liked it for many reasons, but it is the "gold standard" for FITS format support and our FITS support module depends on it since the g'old times of PixInsight LE. The current FITS module uses CFITSIO version 3.35 (June 2013). I'll update it to version 3.37 (June 2014) on September, but I doubt this will fix this problem on Windows.

Ideally, it's time to forget CFITSIO and use something more contemporary. Implementing a FITS support library from scratch is not a realistic option in the short-middle term, so if anyone knows a better FITS support library in C, I am extremely interested in hearing his or her opinions.

Anyway, I'll try to dive into the black waters of CFITSIO code to see if I can hack it to solve this problem. On a parallel note, I strongly recommend using Linux or FreeBSD (and even Mac OS X) for complex and demanding tasks such as building these huge mosaics.

[cmassa]

Thanks for the explanation. Guess I'll try installing a dual boot of Linux with Windows 7.

Chris

[cmassa]

I have installed Ubuntu and have PI running. When I try to open the registered/aligned files of my mosaic I get a similar message to one I got in Windows: 

GradientMergeMosaic: Global context
Start: Gradient Domain Merge Mosaic
Start: Loading images
Start: Loading image /home/chris/PI  Images/Registered/Panel 1 _drizzle_integration_ABE_CC_r.fit 0
Reading FITS: 32-bit floating point, 3 channel(s), 18097x11951 pixels: done
*** PCL Exception: PCL FITS Format Support: FITS file read error:
/home/chris/PI  Images/Registered/Panel 1 _drizzle_integration_ABE_CC_r.fit
CFITSIO error message stack:
01 : Error reading data buffer from file:
02 : /home/chris/PI  Images/Registered/Panel 1 _drizzle_integration_ABE_CC_r.fit
03 : Error reading elements 14401 thru 18097 from image (ffgcld).
Stop: Loading image /home/chris/PI  Images/Registered/Panel 1 _drizzle_integration_ABE_CC_r.fit 0. Time used:  0:13:44.820
Stop: Loading images. Time used:  0:13:44.902
Stop: Gradient Domain Merge Mosaic. Time used:  0:13:44.924

Any ideas?

Thanks,
Chris

[Juan Conejero]

03 : Error reading elements 14401 thru 18097 from image (ffgcld).

Really not the same error. This time CFITSIO is telling you that it cannot read the final part of the image. So the file generated on Windows is not valid (even if CFITSIO gave no error upon writing it). I'm afraid you'll have to generate the whole mosaic again on Linux...

[cmassa]

Thanks. I have gotten more exposures so I need to redo it anyway.

One other question about Linux, when I want to open files in PI while in Ubuntu it can only see the drive where Ubuntu is installed. I cannot navigate to my storage drive where the images are. But the file navigator in Ubuntu shows all of the drives. I can drag files into the PI window to open them but I don't know that it will work to load a script. How do I get PI to see and navigate to all of the hard drives on the system? Windows and Ubuntu are installed on different drives in a dual boot configuration.
 
Chris

[NGC7789]

In the Open dialog choose "Computer" and navigate to the mount point of your storage drive. In my case my storage drive is at /media/<user name>. I'm not sure how to get it to appear in the sidebar.

-Josh

[cmassa]

There they are, thanks. Seems like a strange place for them to be but I really don't know or understand Linux yet.

Chris

[bhwolf]

Does the latest CFITSIO solve this issue?  In looking very briefly at the release notes:

http://heasarc.gsfc.nasa.gov/FTP/software/fitsio/c/docs/changes.txt

... there are a couple of references to handling large files but not sure if it would solve this problem.

[NGC7789]

I'm not sure how to get it to appear in the sidebar.

It turns out you can just drag and drop items from the right side of the Open dialog to the left side (what I called the sidebar). Then they are easily accessible on subsequent dialogs.

P.S. Sorry for the thread hijack

[cmassa]

Thanks for the info!

Chris