Author Topic: 1087 Ripley crashes on registration of 100+ files  (Read 3780 times)

Offline oetie

  • Newcomer
  • Posts: 24
1087 Ripley crashes on registration of 100+ files
« on: 2014 February 08 06:10:33 »
Hiya,
Not sure if this is a bug or a user problem but I couldn't find a tech help board. I just installed the commercial version of PI on my Fedora Core 19 machine. It has a AMD Phenom II X4 955 Processor and 4GB of mem (and 1 GB of swap). I am trying to preprocess  a little over 100 BIAS files and PI eats up all my memory and is eventually killed by the oom killer. I attached the relevant dmesg output. Memory usage before staring PI is 1.5GB/
The directory with all the BIAS files is 1.2GB so not that huge. I guess I could try to fix this by creating more swap space or getting more memory (or maybe trying a smaller window manager) but I do suppose this could be handled more elegantly, especially for a little less computer savvy users.
Thanks
Oetie

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2172
  • Join the dark side... we have cookies
    • http://www.astrophoto.cl
Re: 1087 Ripley crashes on registration of 100+ files
« Reply #1 on: 2014 February 08 07:25:01 »
In which fileformat are your frames? If not FITS, try batch converting them to that format, and try to integrate again.
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline oetie

  • Newcomer
  • Posts: 24
Re: 1087 Ripley crashes on registration of 100+ files
« Reply #2 on: 2014 February 09 12:19:45 »
Thanks, I will try that (they were CR2).

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: 1087 Ripley crashes on registration of 100+ files
« Reply #3 on: 2014 February 10 09:56:58 »
I understand that you refer to integration rather than registration, mainly because registering bias frames doesn't make too much sense.

From what you say I also assume that you are working with raw DSLR images. To integrate 100 of these images, the ImageIntegration process has to load all of them in memory at once (this is because DSLR raw formats don't support incremental file reading). Let's assume that your raw images are something like 4000x3000 pixels, and let's assume also that you are loading them as single-plane CFA data (no deBayering). ImageIntegration works with 32-bit floating point data internally. So 100 of these images require:

4000 x 3000 x 4 = 4.47... GiB

which is more than the size of your physical RAM. Since you are running a 64-bit operating system, you could complete the task using virtual memory on disk (although very slowly). However, you only have 1 GB of swap space. Consider that your Linux kernel needs at least 1 GB to run, plus some extra space to run vital services and processes, plus the space consumed by your X11 server and desktop manager, among other things. This means that your system is facing a severe out-of-memory condition.

When this happens, the Linux Out-of-Memory (OOM) Killer starts killing processes, especially those processes that have caused an OOM condition. When a process is killed this way there's no room for elegance: it simply dies without any chance to fix anything.

So the problem here is twofold: your machine clearly lacks the necessary resources to accomplish this task, and your Linux swap partition is too small (not just for PixInsight; with just 4 GB of RAM, you should have at least 8 GB of swap space for a normal desktop Linux installation).

On a side note, if you pre-convert your DSLR raw frames to the FITS format, you should be able to integrate 100 bias frames. Take a look at the BatchFormatConversion script.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: 1087 Ripley crashes on registration of 100+ files
« Reply #4 on: 2014 February 10 10:30:11 »
Since this is a common problem for beginnner: maybe add a warning to ImageIntegration when the estimated required memory exceeds available RAM. Also maybe add a hint to the documentation.
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)