300th Image Curse!

[harist]

I'm opening a new thread for this since my last post went unoticed as part of the "ImageIntegration Problem" thread.


ImageIntegration process with more than 300 files:

Just had the same problem at the 300th file (using windows 7 pro 64 bit). Has a solution been found to this?
PCL FITS Format Support: Unable to open FITS file:
D:/Astrophotos/All M63/Al_BINT_C_m63b_t60_63_r.fit
CFITSIO error message stack:
01 : failed to find or open the following file: (ffopen)
02 : D:\Astrophotos\All M63\Al_BINT_C_m63b_t60_63_r.fit

Thanks,

Tasos

[georg.viehoever]

Seems to be similar to http://pixinsight.com/forum/index.php?topic=1869.msg11491#msg11491

[twade]

Tasos,

Here's a quote from the MSDN site:

C run-time I/O now supports many more open files on Win32 platforms than in previous versions. Up to 2,048 files can be open simultaneously at the lowio level (that is, opened and accessed by means of the _open, _read, _write, and so forth family of I/O functions). Up to 512 files can be open simultaneously at the stdio level  (that is, opened and accessed by means of the fopen, fgetc, fputc, and so forth family of functions). The limit of 512 open files at the stdio  level can be increased to a maximum of 2,048 by means of the _setmaxstdio function.

Juan may be able to tweak the _setmaxstdio value to 2048.  It appears it may be set to 512 currently.  Hopefully, Juan will chime in on this subject.

Wade

[Juan Conejero]

Hi Wade,

Thanks a lot! I didn't know the _setmaxstdio() trick. It indeed seems the solution to this problem.

What is unclear is if the new limit applies only to the calling application (including all dynamically loaded shared objects), or if it applies also to shared objects loaded via LoadLibrary(). I suspect that the latter isn't true, so I'm going to add a "_setmaxstdio(2048);" call to the internal PCL module installation routine. In this way all modules should be able to use the maximum possible number of simultaneously open files on Windows.

Thanks again!

[Nocturnal]

I have to admit I'm a bit mystified why PI would need all these files to be open at the same time. It seems rather resource inefficient. I don't know what the penalty is for keeping files open other than that there's clearly an OS imposed limit but still.

[Juan Conejero]

Hi Sander,

When you have to integrate (stack) a large number of images, you cannot have all of them loaded in RAM. Imagine for example 300 16 Mpx raw CCD images. That would require 2*4096*4096*300 = 9.375 GB of RAM allocated just to store the plain raw pixel data, not counting auxiliary structures, and neglecting wasted allocation space due to data alignment and malloc's control data and padding.

For this reason the ImageIntegration process keeps all images open simultaneously during the whole process, and uses a small buffer (whose size is user-definable) per image to load a strip of pixel rows at each integration step. This allows for integration of a very large number of images. It seems that the maximum allowable limit is 2048 on Windows, which seems sufficient anyway. This limit doesn't exist (or can be avoided system-wide) on UNIX/Linux platforms.

[Nocturnal]

Hi Juan,

Right, I wasn't suggesting reading all the images in RAM, clearly that would be a resource issue in itself. So I suppose it's an efficiency thing where you keep a file open rather than closing/opening it as needed.

[georg.viehoever]

...
When you have to integrate (stack) a large number of images, you cannot have all of them loaded in RAM. Imagine for example 300 16 Mpx raw CCD images. That would require 2*4096*4096*300 = 9.375 GB of RAM allocated just to store the plain raw pixel data, not counting auxiliary structures, and neglecting wasted allocation space due to data alignment and malloc's control data and padding.
...

Juan,

I understand that you often are frustrated with the way Windows works. But the times when Linux was limited to something like 512 file handles are not so far away (BTW, you can query the max number with  cat /proc/sys/fs/file-max ).

Anyway: From my point of few, a leading edge tool such as PI will often touch the limits of technology, but it (unfortunately) has to work with the machinery that is available. Software that my team is writing is full of such workarounds for the different quirks (or features ) of different platforms. I believe that software should automatically fall back to a less resource consuming strategy if some resources get low (memory, filehandles, disk space, ...), even at the cost of performance, and maybe output a short warning message. But it should continue to work.  I would prefer if PI would have automatisms like this instead of exposing the user to many effects that he just cannot understand without deep knowledge of C++, compilers or operating systems.

I wonder how DeepSkyStacker is doing this. I never met any memory or file handle issues with this tool, even on my "old" 2GB WinXP system.

My 2 cents.

Georg

[Juan Conejero]

I understand that you often are frustrated with the way Windows works. But the times when Linux was limited to something like 512 file handles are not so far away (BTW, you can query the max number with  cat /proc/sys/fs/file-max ).

Don't worry, I'm not frustrated at all; I don't use Windows 

By the way, my Linux workstation (Fedora 13 x86_64) has file-max = 1605985 by default

Let's clarify a bit about ImageIntegration. The ImageIntegration process in PixInsight has no resource problems (other than the file handles limit on Windows, but I hope it will be fixed in the next version). ImageIntegration is able to integrate thousands of images, precisely because I designed it applying quite severe resource availability constraints. The images are not loaded in RAM at the beginning of the process, as most other imaging applications do. Instead of that naive approach, ImageIntegration implements an incremental pixel rejection/integration process. The integration work is divided into several steps, and at each step a strip of pixel rows (which can be as small as just one row) is read from each file and all pixels with the same coordinates are rejected and integrated using concurrent threads.

The only abundantly consumed resources are file handles, since all files must remain open during the entire process. Obviously, opening/closing files at each step is a very bad idea because it would invalidate file caches, among other things.

I would prefer if PI would have automatisms like this instead of exposing the user to many effects that he just cannot understand without deep knowledge of C++, compilers or operating systems.

In fact PI has hundreds of OS-dependent exceptions and adaptations internally. I see no problem here requiring the user to be a computer geek; just a bug and (hopefully) a solution, although the solution is a bit geeky, certainly

By the way, while I've been writing this PI has integrated 220 4 Mpx images with noise evaluation and sigma clipping in roughly 1000 seconds

[Juan Conejero]

Last-minute update:

This is line #51 in fitsio2.h (an internal header file in the CFITSIO library), version 3.25:

#define NMAXFILES  300   /* maximum number of FITS files that can be opened */

and these are lines #31 to 40 in drvfile.c, same version:

static char file_outfile[FLEN_FILENAME];

typedef struct    /* structure containing disk file structure */
{
    FILE *fileptr;
    LONGLONG currentpos;
    int last_io_op;
} diskdriver;

static diskdriver handleTable[NMAXFILES]; /* allocate diskfile handle tables */

A really ugly, inefficient and overly simplistic way to implement a file table. This should be a dynamic list, not a static array. I definitely would not hire the programmer that has written this crap. Now you know why I don't like CFITSIO (I don't like FITS at all, anyway).

So besides the file handle limit on Windows, this is the actual problem, and it affects all platforms. Of course, this is now fixed (set to 4096, which I think is a more than reasonable upper limit). This is my first hack to CFITSIO, too.

Next time I'll test all processes with actual disk files, not with synthetic data generators, even if I need to create thousands of actual disk files and wait for hours. Promised! 

So this bug is fixed in 1.6.1. Thanks you all for your patience.

[harist]

The good news is that the problem is pinpointed and easily solvable. I don't really like integrating each night separately and then reintegrating the integrated nights!

Tasos

[georg.viehoever]

Last-minute update:

This is line #51 in fitsio2.h (an internal header file in the CFITSIO library), version 3.25:

If they are using the C standard library (which they seem to do considering the use of FILE), they/you should use FOPEN_MAX  (defined in stdio.h).

Georg

[Juan Conejero]

they/you should use FOPEN_MAX  (defined in stdio.h)

I don't think so. FOPEN_MAX is actually the *minimum* number of files that the system guarantees can be opened simultaneously. See for example:

http://opengroup.org/onlinepubs/007908775/xsh/stdio.h.html

For example, on my Fedora 13 with gcc 4.4.4 FOPEN_MAX is defined as 16. I don't know of a standard way to know the maximum number of files on all platforms (/sys/proc on Linux is great but is nonportable, the same with ioctl for FreeBSD, etc.). So I think a practical option is to set a very high limit (I've set 8192) and if the process goes beyond the allowed limits, let the OS report a runtime error. This only affects CFITSIO, anyway.

[NKV]

So this bug is fixed in 1.6.1.

I want to confirm: 659 bias files integrate successful

[Nocturnal]

Excellent!