Increasing the Maximum Number of Open Files on Mac OS X

[edd]

Hi,
I'm experiencing crashes during a large ImageIntegration process. I'm running 64 bit in Mac OS X 10.8, and this affected PixInsight 1.7 and now 1.8 RC3. The integrations are large (~250 images in this case) but dropping stack and buffer sizes as suggested in a previous thread doesn't help (plus this is 64 bit rather than 32, which I thought should help).

This usually happens in the phase when images are still being read in.

Crash report attached. Thanks! (Everything else is working brilliantly!)

[pfile]

you might try running memtest on your machine to see if you have bad ram.

http://www.elsething.com/memtestosx/memtestosx.html

i've never had a crash even while integrating 100s of images.

rob

[pfile]

ha! i take it back. i just tried to make a superbias with 250 files under 1.7 and the following things happened:

1) after reading about 180 files, PCL started complaining that it could not open some of the bias files
2) i tried removing those files from the list one by one but the problem seems to be related to the # of files - there's nothing wrong with the files it can't open during II
3) i toggled the last 50 files and tried again and PI crashed.

ouch. took a bunch of unsaved work with it.

Code: [Select]

Thread 9 Crashed:
0   libsystem_kernel.dylib        0x00007fff922d4212 __pthread_kill + 10
1   libsystem_c.dylib              0x00007fff91e45af4 pthread_kill + 90
2   libsystem_c.dylib              0x00007fff91e89dce abort + 143
3   QtCore                        0x000000010465e765 qt_message_output(QtMsgType, char const*) + 117
4   QtCore                        0x000000010465e927 qt_message_output(QtMsgType, char const*) + 567
5   QtCore                        0x000000010465eaea qFatal(char const*, ...) + 170
6   QtCore                        0x00000001047c22d9 QEventDispatcherUNIXPrivate::QEventDispatcherUNIXPrivate() + 425
7   QtCore                        0x00000001047c23df QEventDispatcherUNIX::QEventDispatcherUNIX(QObject*) + 47
8   QtCore                        0x0000000104669dea QThread::idealThreadCount() + 58
9   QtCore                        0x000000010466af78 QThread::terminate() + 744
10  libsystem_c.dylib              0x00007fff91e44742 _pthread_start + 327
11  libsystem_c.dylib              0x00007fff91e31181 thread_start + 13

Thread 9 crashed with X86 Thread State (64-bit):
  rax: 0x0000000000000000  rbx: 0x0000000000000006  rcx: 0x0000000139200c38  rdx: 0x0000000000000000
  rdi: 0x000000000000d21f  rsi: 0x0000000000000006  rbp: 0x0000000139200c60  rsp: 0x0000000139200c38
   r8: 0x00007fff7a680278   r9: 0x0000000139200c50  r10: 0x0000000020000000  r11: 0x0000000000000206
  r12: 0x0000000000000003  r13: 0x0000000139200d40  r14: 0x0000000139201000  r15: 0x0000000127f8d8a0
  rip: 0x00007fff922d4212  rfl: 0x0000000000000206  cr2: 0x00007fff7a679fe8
Logical CPU: 0


[pfile]

200 images -> okay
210 images -> crash

not sure where the actual breaking point is.

[Juan Conejero]

Hi,

This seems related to the maximum number of open files per process in Mac OS X. Open a terminal window and enter this command:

ulimit -a

The standard on 64-bit Linux kernels is 1024 files. On Mac OS X the limit is 256 files. This explains the problem. I'm not sure about how to increase this number, but perhaps you can find useful information here:

http://stackoverflow.com/questions/5377450/maximum-number-of-open-filehandles-per-process-on-osx-and-how-to-increase

On Windows, I use the _setmaxstdio routine to set the maximum of 2048 open files that Microsoft's CRT allows. On Linux we haven't had this problem (the default limit of 1024 files has not been exceeded, AFAIK). I'll try to find a similar solution to solve this problem on Mac OS X. I've found some hints here:

http://stackoverflow.com/questions/3166783/how-to-increase-the-limit-of-maximum-open-files-in-c-on-mac-os-x

and unfortunately it seems there may be a problem with OS X's stdio that would make a solution very difficult. Anyway, there's a bug somewhere because the application should never crash, but "fail elegantly" in case the limit of simultaneous open files is exceeded.

[pfile]

that makes sense, i was going to say i'd expect to see a crash at 255 or 256 images. but of course PI's going to have dylibs and other images and stuff open all the time.

i wonder if the bug is somewhere in the FITSIO module. what happens is first FITSIO returns an error due to the file limit, and ImageIntegration stops. then a subsequent run of II crashes in the places shown here. which is weird because it almost looks like it's crashing while trying to put up a dialog box. or maybe memory corruption/corrupted pointer or something?

[georg.viehoever]

....and unfortunately it seems there may be a problem with OS X's stdio that would make a solution very difficult. Anyway, there's a bug somewhere because the application should never crash, but "fail elegantly" in case the limit of simultaneous open files is exceeded.

I would think that I should fall back to a less file handle intense approach in this case - even at the cost of some performance. Getting a result slowly is better than getting no result at all.
Georg

[Juan Conejero]

Morning Georg,

Getting a result slowly is better than getting no result at all.

Sure, but that would require rewriting the II tool, which is time-consuming and nontrivial. Definitely, I'll try to increase the open files limit programmatically on Mac OS X before resorting to that

[Juan Conejero]

I wonder if the bug is somewhere in the FITSIO module.

From what I've seen in some tests, I agree (and I'm not too surprised, actually).

[edd]

Great - I ran
  launchctl limit maxfiles 2000 2000
and it seems to be crunching happily now. Thanks everyone!

[pfile]

do you have to restart PI or start PI from the shell where you ran that command?

is it sticky or must it be done on every reboot?

[edd]

It isn't sticky, and PI doesn't need to be started from the same shell as the command. I'm not sure if it needs restarting to apply but I suspect not - not able to test that right now.

To make it sticky /etc/launchd.conf can apparently be used, but I've not tested that either.

[Juan Conejero]

Hi,

launchctl limit maxfiles 2000 2000

I have made this topic sticky. Thank you for sharing the solution to this problem.

[pfile]

well i finally tried this in an effort to make a superbias.

setting the max # files to 2000 changes the character of the failure for me, but it still fails. it does not crash anymore but even before it gets to 200 files it refuses to open any more.

i wonder if i have hit some other kind of limit, as these files are quite large (90MB each)