Crash in ImageIntegration trying to integrate 275 bias frames

[rgbtxus]

This crash is repeatable on my machine with my data with an intervening reboot.
After the reboot I tried integrating 4 frames which worked fine.
By eye and memory it died initially after reading about 70-80% of the files.
I then retried it using the last 200 files (includes the file it was reading when it crashed)
That worked properly
Out of memory or stack kind of issue?
Let me know if you need anymore information
I have attached a document with the OSX crash dump info for both crashes
Richard

My parameters for ImageIntegration were as follows:

Combination: Average
Normalization: No Normalization
Weights: Don’t Care (all weights =1)
Scale estimator: Median Absolute Deviation from the median (MAD)
only integrate checked
Pixel Rejection (1):
Winsorized Sigma Clipping
No Normalization
Check all boxes
Pixel Rejection (II):
Sigma Low 4.0
Sigma High 3.0
Range Low 0.0
Range High 0.98

[pfile]

unfortunately a long-known problem and a limitation of OSX. per-process open file limit is in the range of ~200 or so i guess.

really dumb for a unix-based 64 bit operating system...

rob

[rgbtxus]

Ok, but I'm surprised on two fronts: 1) I could swear I integrated 400 bias frames a few weeks ago, and 2) your answer seems to imply that PI keeps all the files open while processing them, that seems unlikely to me (but I'm just a newbie and fully expect the flaw is my expectations) and I reran it on exactly 200 files (a number I picked out of a hat) and it worked.  Are you sure that is what is happening here -- and could better error handling prevent  PI from crashing if that is the case?

Do you have a suggestion for how to combine the data piecewise?  I can think of things to do, but I'm likely to make some incorrect assumption.

Thanks,
Richard

btw:  I do run linux file servers in my house so I guess I can always do the work there.  I believe I read that the image files are stored in a machine independent (or at least byte order designated) form so you can swap the files around systems.

[rstranzenbach]

If this issue limit is caused by the open file limit, this limit can be extended on a per process basis on OS X like on any other unix like OS.

Thranduil:~ ralf$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4864
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited
Thranduil:~ ralf$ 

I would try to extend the limit (ulimit -n <n>) and execute PixInsight by hand (/Applications/PixInsight/PixInsight.app/Contents/MacOS/PixInsight) --- just to check if this is the root cause for the crash.

[rgbtxus]

thanks, I'll try that

[rgbtxus]

Well, I tried bumping open files to 1024, then your value of 4864.  That prevent a crash but it errored out with "Unable to open FITS file ...." on the 228th file.  So, I guess I'll have to look elsewhere.
The message was
PCL FITS Format Support: Unable to open FITS file:
/Users/rgb/AP wip/M81 - 2015-01-17/bias/B_M81__0s_2015-01-18_04-25-31.fit
CFITSIO error message stack:
01 : failed to find or open the following file: (ffopen)
02 : /Users/rgb/AP wip/M81 - 2015-01-17/bias/B_M81__0s_2015-01-18_04-25-31.fit

Of course I checked, the file is there and can be opened

Thanks for looking at this.  Looks like pfile may be dead on.  OSX may have some other open file limitation so raising ulimt -n may not be helping.

Richard

[pfile]

i should have mentioned in my reply that i did try changing the # open files and my system pretty quickly became unstable. so i stopped messing with it...

i can't remember what version of OSX i was on at the time, maybe things have improved. i am still on 10.8 currently.

rob

[pfile]

oh also i'm not sure ulimit works on OSX? i think you have to use sysctl.

http://stackoverflow.com/questions/5377450/maximum-number-of-open-filehandles-per-process-on-osx-and-how-to-increase

rob