Combining 100 bias files (32MB each) fails with out of memory

[Nocturnal]

Hi,

I have a new ASI160MM-Cool which generates 32MB FITS images:

K:/astroimaging/captures/2017/2017-04-14/bias/B_2017-04-14_13-08-28_Bin1x1_0.001s__-10C.fit
29 FITS keywords extracted
Reading FITS image: 16-bit integers, 1 channel(s), 4656x3520 pixels: done

When I try to create a master bias I get out of memory errors even though I use Vicent's suggested settings. At the moment my PC has about 6 GB RAM available wit PI running. Total available RAM is 12 GB. The settings I used: Average, no weighting, no noise eval, no norm, iterative k-sigma (didn't try others here, is there a way to disable this?). Meanwhile DeepSkyStacker had no trouble creating a master bias for me.

I was considering using PI for my stacking with this new camera but this doesn't look very promising. Even if I can 'dumb down' the integration enough to make it work it seems unlikely PI can handle stacking 50-100 lights which is what I anticipate needing with this camera. Because of the 12b ADC you are stuck taking lots of relatively short exposures.

Any hints appreciated! I'd rather not thrown out the memory sticks I have an replace them to get to 24GB or more.

Thanks,

   Sander

[msmythers]

Sander

What type of computer are you using? Is it a laptop or desktop? Windows, Linux, OSX or BSD? Does it have a dedicated graphics card with memory or does does the display use shared memory with the main memory? Do you have an SSD and have optimum swap space defined. Are you running low on drive space possibly? Many possibilities.

I don't have 32mb subs but mine are 16mb. I have stacked as many as 524 without splitting the process. I normally stack between 200 and 300 bias subs also.

My computer is a Windows 7 desktop with 16GB of ram, 2 SSD drives for PI and OS. 'All' data is kept on standard rotational drives. I have a 2560x1440 monitor powered by graphics card with 4GB of dedicated ram. Attached is what my memory usage is when I just start PI without other software like web browsers running. I've never had an 'out of memory' warning with this setup.


Mike

[Nocturnal]

Mike,

It is a Windows 10 desktop PC with an i7-920 and 12 GB of RAM. Dedicated graphics car. It's a bit old but with 8 virtual cores it's no slouch. I have a standard SSD for the OS and an NVMe SSD for my datafiles. Since PI explicitly says it is out of memory I doubt disk space has anything to do with it but perhaps the error is wrong. I'll check empty space on the OS disk but I think it's 10 GB at least. If someone knows for sure that image integration requires a lot of temporary disk space I'll look at that.

Thanks,

   Sander

[msmythers]

Sander

I just finished a 300 bias stack. The physical memory ran out before I hit 240 images but the process never stopped and never gave any error. I also have been checking my 2 swap folders and no files have been placed in those files. I would assume that the Windows Page File is taking the memory over run at this point. So you might want to check your Windows Page file setup. I used to set a definitive amount but with heavy graphics and video processing I sometimes do I decided to leave the page file management up to windows. 

Here are 2 screen dumps of the stack and the memory. The first is just as the final stage of the integration was starting. The second shows the memory returning to normal and the 3 images from the integration.


Mike

[Nocturnal]

Mike,
I see that my swap files have been limited to 1GB, probably by a Samsung SSD settings utility that tries to minimize wear on the flash drive. Even when I let the system set the swap policies PI runs out of memory so I doubt it can be resolved with swap. Once you start swapping performance takes a dive so it's not really an option for making this work for me, especially when DSS can do these files without apparent trouble. I verified that my C: drive has 55 GB available. I have more drives but temporary files are on C: so that's irrelevant.

I appreciate you trying to help!

   Sander

[Niall Saunders]

Hi Sander,

Very confusing !

Your images actually work out smaller than mine (QHY10: 3896 x 2612 x 1 RAW plane x 16-bit => 19.4Mb per image versus yours, in MONO, at ~31Mb per image), but only once I have deBayered mine into three colour planes (taking my image sizes to 61Mb) and, in reality these actually become 116Mb once I have stored my data in 32-bit float format - versus yours at ~63Mb if stored in 32-bit format.

So, 'image for image' you should not be running out of memory (or any other resources) given that you actually have 50% more RAM than I have (12Gb vs. 8Gb) and 50% more processors than I have (on far more powerful silicon than me - I'm still using my 8y.o. PC which was built the week before it Intel i3//5/7 chips hit the market).

Have you tried running the built-in Windoze "PerfMon" utility side-by-side with PixInsight? I use it quite a lot (and I also use it if I am writing code that is 'multi-threading' - which I try to avoid like the plague). If you haven't used it, I can send you the file that I use to configure the basics of PerfMon, to get things up and running (you'll need to tweak it for six cores, etc.)

But - none of this actually answers your basic question of "why" it isn't working! Sorry 

[Niall Saunders]

Oh, and on a separate point: I haven't made any adjustments for swap files whatsoever (either at OS or at PI level). I have four SSDs that I can use, but need to deal with that thorny issue at some later date - and I am still using Win7 64-bit here in the office (and XP SP2 systems out in the observatories on 12y.o. steam-driven lap-bacus machines with 3Gb RAM and rare, treasured, copies of PixelInsight LE punched on their paper-tape drives)

[Nocturnal]

Hi Niall,

I will run PerfMon next to PI next time I try this and post the results here. Thanks!

   Sander

[Niall Saunders]

Hi Sander - I got myself up to speed with PerfMon using video tutorials on YouTube.

One thing that was not clear, however, was the means of savung/loading configuraton setups for PerfMon. Saving was easy enough, but loading wasn't quite so simple - it involves opening the stored file (in HTML format) and the performing a Copy-to-Clipboard out of the HTML file, followed by a Paste-from-Clipboard inside PerfMon.

I found it quite interesting to see how my system behaved when I ran ImageIntegration with various Stack and Buffer sizes being defined.

Let me know if you can't figure it out (I know, somewhat unlikely for a Jedi Council Member  8) )

[Nocturnal]

Interesting. I ran II with Task Manager running. Total consumed memory went up a little and then the 'out of memory' error showed up. There was still several GB available. I just verified I'm using the latest Windows x64 version with fully updated modules. I hope Juan will shine his light on this issue soon.

[Nocturnal]

I just integrated 100 bias frames on my work laptop which is rather anemic compared to my personal desktop. Very odd. The work laptop install is 'fresh' while my personal install is ancient. Did some flag or option get flipped the wrong way that is preventing this from working properly?

[pfile]

there was just a thread on CN similar to this - the guy had his files out on a NAS and II was not working so well. he dug in a little and found that there were a zillion file handles open (which i think is the proper behavior with all the new parallelism that juan has implemented.) he never received an "out of memory" error but the thing slowed to a crawl. my thesis was that with all the seeking in loads of files that the NAS with its many spindles and heads is not a good match, but he found that it worked out OK as long as he reduced the number of files under consideration, similar to your problem.

so i wonder if there are some OS parameters that are different between your machines that give performance problems when there are lots of file handles open?

rob

[Nocturnal]

My work laptop runs W7, my desktop W10. I don't recall ever touching open file maximums. My desktop has much faster NVMe storage for the data files, more memory and more cores. I'll start by refreshing the PI configuration to default and see if that does anything.

[Nocturnal]

I reset my PI configuration and now image integration works. I have to admit I'm not particularly impressed by this but at least it works now. Since the previous config was stashed I can upload it if Juan or someone else wants to figure out what parameter was out of whack.

[Nocturnal]

It works for perhaps 40 images but when I combine 160 it still fails with the same error:

*** Error:
*** Error: Out of memory
<* failed *>

I see no appreciable bump in memory consumption while this is happening. I increased the buffer and stack sizes but this makes no difference. It stops at the same number of files each time. I can tell because subsequent runs are very fast due to caching.