*** Error: Out of memory on 64-bit system with 32GB of RAM doing integration

[dnault42]

I'm building a very large stack of 414 images at 6868 x 6894.  I was actually intending to drizzle these as well (the data does support it), but I can't get past the ImageIntegration step at the moment due to the error in the title.  This is the last portion of the log file:
Integration of 414 images:
Pixel combination ......... average
Output normalization ...... additive + scaling
Weighting mode ............ custom keyword: SFSWEIGHT
Scale estimator ........... iterative k-sigma / BWMV
Range rejection ........... range_low=0.000000
Pixel rejection ........... Winsorized sigma clipping
Rejection normalization ... scale + zero offset
Rejection clippings ....... low=yes high=yes
Rejection parameters ...... sigma_low=4.000 sigma_high=3.000

* Using 23 concurrent pixel stack(s) = 998.48 MB
Integrating pixel rows:     0 ->   609: done
Integrating pixel rows:   610 ->  1219: done
Integrating pixel rows:  1220 ->  1829: done
Integrating pixel rows:  1830 ->  2439: done
Integrating pixel rows:  2440 ->  3049: done
Integrating pixel rows:  3050 ->  3659: done
Integrating pixel rows:  3660 ->  4269: done
Integrating pixel rows:  4270 ->  4879: done
Integrating pixel rows:  4880 ->  5489: done
Integrating pixel rows:  5490 ->  6099:  75%
*** Error: Out of memory

My understanding was that PixInsight will push data back out to swap space if it starts running out of memory and I do actually see this happening when while monitoring it, but the memory allocation keeps creeping back up until the available memory drops to zero and this error pops up.  I went back to using the 1123 build as I was seeing some of the same allocation errors others have posted about on this forum.

All that being said, I'm wondering what to do about this.  I could purchase more memory (bringing the total to 64GB), but would that actually solve the problem?  What about doing a dual boot with linux, is running PI on linux more robust in memory management and would it avoid this problem?

Regards,
David

[jkmorse]

David,

That is a lot of data!  In similar situations, I have seen people mention in the forum that they run two or more separate integrations, then ultimately combine those.  Beats investing in more memory until you see if Juan has a solution.

As to a dual boot, I have heard Juan and others sing the praises of the linux version of PI being much more stable so that may be another way forward.

All this assumes that you have identified appropriately sized swap cache files in the Directories and Network folder in Global Preferences.  The notes there state that for normal processing you should have cache files that are a minimum of 60Gbs.  You may want to try upping that substantially if you have the space for files the size you are talking about.

For what its worth,

Best,

Jim 

[msmythers]

David

Is it possible your swap file area is full or not large enough. I have done integration's with over 600 images though only 4608x3072 images. Now I have not tried anything of that size with this latest revision but I did do a 540 image stack with 1.8.3.1123 a few months ago without issues. I don't have an SSD so my swap directory is on a separate internal 1.5 TB drive. I never worry about swap space.

Did you try purging and clearing ImageIntegration cache?


Mike

[dnault42]

Jim, Mike,

There's going to be more data than this eventually.  It's part of a large collaboration.  I've used the trick to split it up into smaller stacks before (4096 bias frames just wouldn't work in one pass) and I'll do that again if I don't have another choice, but I was trying to accomplish several things with this approach:
1) Keep it to only one pass of interpolation
2) Utilize the area based interpolation method inherent in drizzling
3) Really push the rejection parameters to get the absolute maximum from the data in a single stack
4) Use SubframeSelector to adjust the weighting between the frames

As for the swap file, I've got it pointed to an SSD with ~180GB of free space.  I didn't check to see if it fills up during the integration.  As my next experiment I'll point it to my storage drive which has about ~1.8TB of free space left and give it another run.

I did purge/clear the II cache right before running.  I was actually going to turn the cache off, but from what I understand disabling it doesn't stop the cache from building up during the current sesstion.

Thanks for the suggestions guys.  If you can think of anything else for me to try, definitely let me know.

Regards,
David

[RickS]

Have you tried increasing the size of the Windows paging file, David?

Cheers,
Rick.

[NGC7789]

As you may know you can point the swap to BOTH your SSD and your hard drive. You will get twice the smaller one (2 x 180GB in this case) at higher throughput than the larger source alone. Plus you can declare multiple swap threads for increased performance (but not space).

Linux will manage memory better and give you more tweaking options but if the issue is running out of swap linux alone won't remedy that.

[dnault42]

Moving the swap space to the mechanical drive seemed to have no effect.  It still fails at the 75% mark on pixel rows 5490 -> 6099.  I don't see that PixInsight uses any of the swap space for ImageIntegration.  After changing the swap location to the other drive I got 2 entries in the folder I specified, a PIWebIcons folder (something like 20KB of space used for that) and a .psm file that is about 874KB and didn't change in size during the run.  II seems to use the windows paging file, which is what I think Rick was getting at.  Right now I have Windows set to dynamically increase the paging file as necessary.  Before starting PixInsight it was a little over 4GB, it got up to over 25GB by the time it failed.  Rick, do you think there's any benefit to setting a fixed pagefile size?  I could set it at something like 50GB.

Thanks for the tip NGC7789, I had seen that you could have multiple swap locations specified.  I hadn't bothered with doing this before because I thought I had plenty of storage space on the SSD drive.  Based on what I saw with the experiment above, this may still be true, but I will definitely start specifying the mechanical drive as a swap location as well.

Regards,
David

[RickS]

Moving the swap space to the mechanical drive seemed to have no effect.  It still fails at the 75% mark on pixel rows 5490 -> 6099.  I don't see that PixInsight uses any of the swap space for ImageIntegration.  After changing the swap location to the other drive I got 2 entries in the folder I specified, a PIWebIcons folder (something like 20KB of space used for that) and a .psm file that is about 874KB and didn't change in size during the run.  II seems to use the windows paging file, which is what I think Rick was getting at.  Right now I have Windows set to dynamically increase the paging file as necessary.  Before starting PixInsight it was a little over 4GB, it got up to over 25GB by the time it failed.  Rick, do you think there's any benefit to setting a fixed pagefile size?  I could set it at something like 50GB.

David,

AFAIK, PI uses its swap directory for saving copies of images as they are modified.  It is probably not relevant to ImageIntegration as you have discovered.

The size of the Windows paging file will determine how much virtual memory your system can allocate.  I'm not sure if a larger fixed size will help but it is fairly easy to try.  Another thing that might be worth trying is to reduce the Buffer Size parameter in ImageIntegration.  The tool tip text suggests this can help with out-of-memory errors during integration.

Cheers,
Rick

[dnault42]

Well, increasing the Windows pagefile did help somewhat.  It made it through the process of stacking the data and generated the integrated result, then moved on to updating the drizzle files.  I left that going over night, but it crashed part way through (it was a full system crash).  The last file updated was around 1am, which wasn't long after I went to sleep.  It wrote out about a 3rd of the drizzle files before crashing.  I'm going to try three things today.  First I will try dropping the buffer size as Rick suggested along with further increasing the page file.  If that works, I'll continue forward trying DrizzleIntegration and see if that breaks.  If either of those two things fails, then I will setup a dual boot config with Ubuntu (the distro of linux I'm most familiar with) and see if that works any better.

Regards,
David

[dnault42]

Rick,

Thanks for the tips.  I was able to get through a full pass of ImageIntegration and DrizzleIntegration.  I decreased the buffer (8MB) and stack size (512MB) and increased the page file size (160GB).  DrizzleIntegration is giving some interesting results, but I'll start a different thread for that.

Again, thanks for the help.

Regards,
David

[RickS]

Glad I could help, David.  I have done some large integrations but not quite on that scale!

[STAstro]

I just got this when trying to combine 3996 BIAS Frames, I am using x64 Windows 10 with 32GB RAM, Core i9 10 Core, my Page file size is 12089Mb, I might increase this since I am using an NVMe M.2 device for Windows, speed should not be an issue

[Juan Conejero]

If you are integrating FITS files directly, you cannot open more than about 2000 files simultaneously on Windows. This is because the CFITSIO library uses standard C library calls to open files. See this document is you are interested in knowing more about this limitation on Windows. If you are using XISF files, the limit should be much higher, something in the range from 8000 to 16000 files, depending on Windows versions and configurations, since our XISF implementation uses Win32 system API calls on Windows.

Anyway, provided that you have enough memory available to perform this task, you may have to tune ImageIntegration parameters to implement it, buffer size and stack size specifically.

[STAstro]

Thanks Juan

XISF files are all well and good (I save them as this the first opportunity I can), however when dealing with Flats, Darks and Especially BIAS frames then they are straight off the camera so they are in FITS format, I decided that 999 BIAS frames was good enough in the end rather than 4096

With regards to the Issue on Windows, I wonder if you run PI on say Windows 2012/2016 Server you may not hit the maximum open files limit since Windows Server is designed to have more files open, unless it is down to a single process can only have that number of files open

[Golfox2]

I would like to specially say thank you to you guys.
In fact, my PI was working great a few weeks before, I updated windows to the last update, then I cleaned my drives. While doing that, I removed the SSD pagefile.sys as I wanted to save space (30gb !).
Since, PI would not integrate my 24mpx pictures from my canon 6d.
I tried for days playing with the stack size and buffer but nothing.

Finally, by reading your posts, I set back the pagefile for 8gb on the ssd and 32gb on the hdd and now it works seamlessly !
I'll try to lower a bit those values as I'm not sure it's worth having it high as that.

Thanks guys !!