Combining 100 bias files (32MB each) fails with out of memory

[Niall Saunders]

Hi Sander,

You don't have a 'bad file' at the point in the process where the ImageIntegration fails, do you? Just a thought, although I am not sure what such a 'bad file' might look like, or why a particular file could cause a blow-out of your memory.

[Nocturnal]

Hi Niall,

Since I see this problem with two different data sets I don't think the chance of that is high. I suppose it's possible. I would exclude the last file before the crash but the II dialog doesn't show the entire file name and can't be resized so it's a bit tricky.

[Niall Saunders]

No - I agree Sander. I hadn't picked up on the fact that you had seen this with two different data sets. I was going to suggest creating a new data set based on a whole bunch of copies of just one image - but your current experience would suggest that this, now, isn't going to shed any light on the problem.

I don't really have any rational suggestions right now - as a last resort, you might be facing ripping out PI and re-installing, but I hate this kind of 'dynamite' approach (even if it solves your problem, you will never know what the actual issue was, and so you learn nothing!)

[Nocturnal]

Hi Niall,

You couldn't know because I just acquired a new set of data overnight :-)

Anyway, my work laptop just integrated 160 of these images. Juan please give me some things to try here! Clearly PI is up to the task and my computer should be as well. Something is not set up correctly.

[Juan Conejero]

Anyway, my work laptop just integrated 160 of these images.

That's really strange. As Niall, I have no clue about what can be happening here. Your desktop machine definitely should be able to handle this data set without problems. Admittedly, my practical knowledge about Windows is quite limited (I use it just for building and testing), but I'd bet this is a system configuration issue. Sorry for not being able to help. These things don't happen on Linux... well, not normally

[Nocturnal]

I think you should be a bit more worried when PI fails with 'out of memory' when the system has plenty of memory available. Even the script you wrote today without clearing image data consumed far more memory before finally running out. Can PI be run in a mode with more debug messages so it will tell me what memory allocation failed?

[Juan Conejero]

These two cases are completely different. The script may eat a lot of memory because the JavaScript engine does not perform garbage collection as aggressively as it should for some native objects. The main reason is that the engine has no way to know how much memory a native object, such as an Image object, is consuming internally. A simple call to Image.free(), as I have described above, solves this issue in practice. Obviously, freeing memory blocks manually is not something one would expect to have to do in a dynamic language like JS. However, the PJSR is not really as dynamic as we tend to think, mainly because it is being supported by many objects that live in the C++ side of the runtime. This is a very complex topic, where I am investing a lot of my development time for the next versions of PixInsight. The SpiderMonkey engine will probably be replaced with something more manageable in future versions. Perhaps the V8 engine. I am investigating also Qt's V4, which looks promising. In a future version (maybe this year) we'll have an integrated C++ compiler in PixInsight (clang). When that happens, the JavaScript engine will be less relevant because PCL-based C++ code will be compiled and linked on the fly. In fact, the idea is that most standard modules will be distributed directly as source code, which will be built automatically on the user's machine.

The other out of memory issue is not reproducible. Well, it can be reproduced in your machine, obviously, but I have no way to replicate it on my working and testing machines, including Windows 10. The fact that this error does not happen on your laptop with the same data is telling us that it could be related to a particular system configuration. I know this sounds weird, but have you tried disabling your virus protection software? Are there other running background processes, such as system utilities?

 

[msmythers]

Just a question for thought, is the out of memory error an actual PI error report or a passed on Windows error report? That's truly a Juan question. Could you have an actual windows memory issue? Have you run any other software that really taxes the full physical memory of the computer? Just questions.


Mike

[Nocturnal]

Hi Juan,

I realize the javascript runtime is very different, I meant to indicate that the script was able to consume resources visibly while PI does not actually consume resources when it gets the out of memory error. There is a tiny blip in the consumed memory line in performance monitor but it doesn't climb near the top. Some other restricted memory resource is running out and it would be helpful to throw a full exception in that case so at least we can see if it truly is memory or files or something else. Clearly this case shows what the error reporting needs improving or that a logging option needs to be added. Or maybe not and then you simply accept you'll have cases like this where you throw up your hands. I will have a look at Avast and see if it is reporting anything.

Mike,

The error as shown is printed in the console and shown in a small PI dialog box. Windows is not out of memory at all. I know this for sure because a) the resource monitor does not indicate memory distress and b) I *have* run out of memory with PI using javascript and in that case Windows tells me I'm in trouble. That doesn't happen in this case.

Sander