ImageIntegration crashing when stacking large number of frames

[rga218]

Hi, ImageIntegration is crashing when I attempt to stack large numbers of images (several hundred). Here is the background information:

I am running on a Mac OS X machine running Mavericks 10.9.2 with 16 Gb of RAM.

The images are individually 3352 x 2532 and take up about 17 Mb on disk.

Crashes occur even when I define a region of interest of size 300 x 300.

I have adjusted the buffer size and stack size and crashes appear to occur regardless of settings (including when I zero out both the buffer and the stack, which I believe is the minimum memory footprint setting).

ImageIntegration completes successfully when I attempt to stack about 100 images. Attempting to stack 300 images results in a crash. In my final case I expect to stack up to about 1000 images.

The crash log is attached.

Thanks for any advice!

Bob

P.S. Juan: PixInsight is proving absolutely integral to my research projects. It has almost completely replaced IRAF and IDL for me.

[pfile]

unfortunately this is a bug, or maybe more accurately an overly conservative limit in OSX. the kernel is configured to allow only a really small number of open files per process, and a really small total number of open files, compared to linux anyway.

there are some threads here with some ideas to increase those limits, but when i tried changing the limits, my machine became pretty unstable.

rob

[georg.viehoever]

Try a hierarchical approach: to integrate 1000 images, you can integrate your images in groups of 200, and then, in a second step, stack those 5 images. The result should be the same -except for pixel rejection. But groups of 200 images should already have stable enough statistics for good pixel rejection.

Georg

[rga218]

Rob and Georg ---

Thanks a bunch, now that I know what the issue is I will do some experimentation and see what works best to get round this.

Thanks again,

Bob

[Juan Conejero]

Hi Bob,

Mac OS X limits the maximum number of open files to 256. See this thread on Stack Overflow for a possible solution (the first answer to the original post), although I'm not sure if it applies to OS X 10.9. For integration of large sets of files, my advice is to use Linux or Windows if possible.

Hopefully the next version of PixInsight (next week) will be more robust to these problems on OS X thanks to a new version of Qt. However, the default limit of 256 open files is something we can't control.

P.S. Juan: PixInsight is proving absolutely integral to my research projects. It has almost completely replaced IRAF and IDL for me.

I am glad to read that! Let me know how we can improve the platform to make your work easier and better on PixInsight.

[rga218]

Thanks Juan. I'll try adjusting the maxfiles parameter and if that gives no joy I will just run my scripts on a Linux box.

I am glad to read that! Let me know how we can improve the platform to make your work easier and better on PixInsight.

At this point I think the main improvements I'd like to see are already in your plan, namely additional PJSR documentation and some way of making PJSR scripts non-blocking so scripts can call other scripts. But there are only 24 hours in the day and I think what you've already achieved with PixInsight is incredible.