out of memory

[shurik]

this looks like a chronic issue, and I have not found a definitive answer on the forum to the "OUT OF MEMORY" error. here is my situation:

hardware: 32 bit windows xp PC 16GB RAM, 2TB HD

1. processing first registered image went without any glitches, could run any type of algorithms, most took under a minute to complete :-)
2. closed all the working cloned copies and purged all swap files
3. opened the image for the second project ...can not do star mask generation, can not do HDR Wavelets, etc., all because  "OUT OF MEMORY" error.

my question is if there is anything else left from previous project that is eating up that memory, if it is where to look for it , (I purged all cashes and closed all the working copies); command line "mem" returns nothing saying it cant' be run

is re-start the only "true solution" to get rid of previous project..seems a bit radical :-0

what is general memory allocation specs that have to be selected in preferences to optimize the speed and performance.

thank you, Alex

[Carlos Milovic]

This is a strange behavior. Perhaps a regression/bug. Let's wait for Juan to jump in.


PS: 16Gb ram in a 32bits OS?

[shurik]

here is current memory log after I just received "out of memory error":

Total physical memory .......   3326.945 MB
Available physical memory ...   1959.344 MB ( 58.89%)
Total paged memory ..........   4096.000 MB
Available paged memory ......   4096.000 MB (100.00%)



so....1.9GB is not enough to run HDRW and star mask? :-(
(the image is 50MB)

[georg.viehoever]

After processing 178 images, memory was also low on my 2 GB Fedora11-x64 machine. Response times were remarkably slow. The problem disappeared after restarting PI. Note that this is with PI 1.6. Early 1.5 versions failed after 15 images. So I think memory leaks in PI are much less a problem today, but they are still there...

Georg
 

[Niall Saunders]

Hi all,

I am running Vista64 with 8GB ram, and I certainly see v1.6.1 gradually 'slow down' over a long period of 'manual' calibrations. Certainly the interface is not as 'snappy' as it used to be, it can take up to 3 seconds just to get PI to change the ID of an image (2 seconds before the Console even pops up).

Cheers,

[shurik]

well , I just restarted PI, working on a single image, but having multiple copies of it open at the same time (not more than 10) because I run different processes and trying to compare the results and the dreaded "out of memory" error came up again as I was running ATW transform, I purged all the processes, but it still says out of memory...so I am totally frustrated by this...do I now have to close everything again and restart PI hopefully I can get help soon..

what exactly is clogging  it ?...so I can delete it and free the memory that it used up, because it is definitively fixing itself after restart.... I am completely lost the machine says it has enough resources at the moment:

Total physical memory .......   3326.945 MB
Available physical memory ...   1828.730 MB ( 54.97%)
Total paged memory ..........   4096.000 MB
Available paged memory ......   4096.000 MB (100.00%)

[Juan Conejero]

Hi Alex,

Total physical memory .......   3326.945 MB
Available physical memory ...   1959.344 MB ( 58.89%)
Total paged memory ..........   4096.000 MB
Available paged memory ......   4096.000 MB (100.00%)

so....1.9GB is not enough to run HDRW and star mask? :-(
(the image is 50MB)

The memory allocation problem has many aspects to consider. On one hand, Windows won't allow you to use all the available memory. Probably less than 1.2 GB, even if PixInsight is the only application running. On the other hand, the actual problem here is the amount of memory that can be allocated in a contiguous memory block. As available memory becomes scarce, the operating system allows only allocation of relatively small blocks, which may be insufficient to store a single channel of the image. Another problem is memory fragmentation, which increases with time and may also prevent allocation of large contiguous blocks. All of these problems are particularly bad in a 32-bit operating system, and Windows XP isn't famous for its memory management.

That said, I'd need to know the following in order to diagnose the problem:

- The geometry of your image: width, height, number of channels.
- The sample data type: 16-bit integers or 32-bit floating point, etc.
- How many images open?
- Other applications running?
- The memory stats reported by Windows.

Consider also that some of the processes you've mentioned are memory intensive. For example, StarMask may need to generate several working duplicates of the image in 32-bit floating point format.

[Juan Conejero]

Niall,

... and I certainly see v1.6.1 gradually 'slow down' ...

Either you're joking (and then you're a bad boy!), or the subconscious has betrayed you this time

... it can take up to 3 seconds just to get PI to change the ID of an image (2 seconds before the Console even pops up).

This is really weird. I haven't seen that behavior in any of the tests we have conducted. I have also Vista x64.

There is no reason for PI to slow down after a long period of activity, other than memory fragmentation, but this problem should be of marginal importance in a 64-bit OS with 8 GB of RAM. So if you're experiencing this behavior, then we have some kind of strange bug here.

To reproduce it, I'd need:

- Your images.
- A XPSM file with the exact processes you're applying.
- Any additional information I need to reproduce your procedure.

Thanks!

[shurik]

-The geometry of your image: width, height, number of channels.

w:6552;  h:4416, working only on Luminance image


- The sample data type: 16-bit integers or 32-bit floating point, etc.

16 bit

- How many images open?

6 images open now, closing them does not help , still getting the error

- Other applications running?

only firefox is running

- The memory stats reported by Windows.

not sure how to get it...

-On the other hand, the actual problem here is the amount of memory that can be allocated in a contiguous memory block. As available memory becomes scarce, the operating system allows only allocation of relatively small blocks, which may be insufficient to store a single channel of the image. Another problem is memory fragmentation, which increases with time and may also prevent allocation of large contiguous blocks.

cache drive is defragmented and has large blocks of unsused memory

bottom line is , the program is not purging all the processes correctly when asked to do so in the settings menu, something is clogging it..because with whatever resources I have now running on PC and its memory status, if I close PI and reopen it again, it will work perfectly fine before it reaches some magical limit of memory use, that I have no way of seeing or controlling to help cleaning it up, so I wish this could be addressed; otherwise it looks like I have to constantly close and re-open the program  after just about 20 minutes of work...

[Niall Saunders]

Hi Juan,

Sorry - I think I had a breakdown in my flux capacitor. I can understand how annoyed you must be when I am pointing out a bug in software you haven't even written yet

However, by way of apology, let me give you the numbers for next week's EuroMillions : 1, 2, 3, 5, 8, 13, ......

In seriousness though,

The process I was following was nothing 'unusual'. I had given the PC a fresh reboot, with a full Vista64 Service Pack update, having just installed a new graphics card (to drive my two auxilliaryy monitors).

I had several sub-folders, each containing up to 30 images - these being groups of either (mono) Lights, Darks, Flats and FlatDarks - all taken with a 748x577 pixel DSI-IIPro (so, no HUGE images here )

My work-flow involved having a second copy of PI running at the same time (initially( in order to be able to convert all the original 32-bit F data to 16-bit UI data. A sample of these converted iomages had been opened and examined - all were perfectly OK, and image file size had indeed dropped by 'half', as expected.

With the second copy of PI then shut-down, I used ImageIntegration to create a MasterDark and a MasterFlatDark. In each case I renamed the resultant image accordingly, and saved it as a 16-bit UI.

I then used PixelMath and an ImageContainer to calibrate (dark-subtract) my raw Flats, saving these all as 16-bit UI as well. Then, ImageIntegration was used again, this time to create a MasterFlatDark, which was renamed as usual, and which was also saved as a 16-bit UI.

Now I could use PixelMath and another ImageContainer to fully calibrate (dark-subtract, then flat-divide) my raw Lights, again saving everything as 16-bit UI. Next, the calibrated Lights were StarAligned (using the first image in the sequence as a reference), with the resultant images still being saved in 16-bit UI format. Finally a last implementation of ImageIntegration gave me the required MasterLight - which was renamed as such, but was then saved in 32-bit F mode for subsequent processing.

All the individual processes seemed to execute fast enough - even ImageIntegration during recursive implementations (when necessary) to fine-tune the Sigma-Clipping levels (I like to aim for no more than 0.5% clipping at each of the High- and Low- levels).

It was just that the renaming process seemed to take 'longer and longer' as the steps described above were completed. And, by 'rename', I mean "double-click the image ID tab at the top-left of the image window, and type in a new name". In fact - with the 'new name' still on-screen, I would then <Ctrl-C> the text so that I scould also use it to re-name the ProcessIcon, allowing me to save the 'individual' process icon (ImageIntegration, etc.) in the same directory as the 'source data' - giving me a 'natural method' of self-documenting the stages that I had used.

So, what would you like me to do? Let me look at the overall file size of my complete data set (perhaps trimmed down to, say, 10 images per file-type, and these having been already trimmed to 16-bit UI). Either that, or you could connect to my PC using TeamViewer - after all we are less interested in what the data 'looks like' on-screen, as to how long a process takes to complete.

I have some time available this week, but will have NO free time after this coming Friday, for at least a week.

Drop me an email off-group if you want to try anything.

Cheers,

[Juan Conejero]

Hi Alex,

cache drive is defragmented and has large blocks of unsused memory

We are talking about memory (RAM) fragmentation, not disk fragmentation. Suppose you have 1 GB of free RAM. Now you create four images of 250 MB each. Then you close the second and the fourth images. Although there are 500 MB free, you can only create a new image of 250 MB, because that is the largest contiguous block that can be allocated. This is memory fragmentation. This is inherent to all dynamic allocation systems and becomes a serious problem when you are running a 32-bit operating system close to the limit of the physical memory space. Although I have been very careful to avoid it in PixInsight, it eventually causes problems, especially on 32-bit Windows after large processing sequences and RAM-intensive processes.

To show you how well a 64-bit operating system can overcome all of these problems, I have conducted a small test with one of the development machines that we have here at our lab. It is a veteran —but nicely built— Core 2 Duo with just 2 GB of RAM and Windows Vista Business x64 up-to-date.

This is a screenshot of PI 1.6 with 16 copies of a 16-bit grayscale image of 7000x5000 pixels:

http://forum-images.pixinsight.com/outmem/20100421/01.jpg

We have 16x7000x5000x2 = 1120000000 bytes or 1.04 GB of RAM allocated in 16 contiguous blocks of 66.757 MB each. Add to this PixInsight's 64-bit memory footprint (some 120 MB), the operating system kernel (around 500 MB), all system services and other running application, and you'll see that nearly all the available physical RAM is being used.

The next screenshot shows PixInsight 1.6 during execution of StarMask. It also shows memory statistics before running StarMask. There were just 217 MB free.

http://forum-images.pixinsight.com/outmem/20100421/02.jpg

This screenshot also shows Windows Task Manager (sorry, my OS is in Spanish) during execution of StarMask. As you see both processor cores were being used at 100% load, and the whole physical RAM was used. You know this from the "Disponible = 0" label, which in English means "Available = 0".

Here is PixInsight 1.6 after StarMask execution (which was quite slow, due to the necessary disk swapping):

http://forum-images.pixinsight.com/outmem/20100421/03.jpg

If you look at the console, you'll notice something very interesting: the available physical memory after StarMask execution is 618 MB, nearly three times more than before StarMask. How is this possible? Because Vista x64 has been able to move some resident processes to swap file storage (probably some services and part of its own kernel), compacting available physical RAM to yield more free space available to PixInsight.

As you see, with a 64-bit operating system I've been able to work with much more pixel data (16 images larger than yours) in half the RAM space (2 GB instead of your 4 GB).

So the problem here is a combination of extremely poor memory management (Windows XP 32-bit) and memory-intensive processes (StarMask, ATrousWaveletTransform and HDRWaveletTransform require several temporary working images in 32-bit format). My recommendation is to install a 64-bit operating system and a 64-bit version of PixInsight, especially considering that your machine has more than 4 GB of RAM installed.