Error: not enough space and Error: Out of Memory

[johnhudson922]

When trying to do a basic image integration to create a Bias master, I am getting these errors. This is a fresh (new) install of PI, the newest version. I am on Win 10, 64 bit, plenty of room on the HDD, 16 GB ram, etc. No real reason I should get this error that i'm aware of. It was a smallish amount of files, less than 20 probably. Ideas?

[johnhudson922]

Here's another one...

[johnhudson922]

So i reduced the amount of bias images i was trying to integrate to 8 from 31 and it worked. Any ideas? I need to be able to integrate more than 8 especially for darks.

[Andres.Pozo]

I think that if you convert your CR2 images to XISF the integration should work for any number of images. You can use the script BatchFormatConversion.

[johnhudson922]

I think that if you convert your CR2 images to XISF the integration should work for any number of images. You can use the script BatchFormatConversion.

Ok, thanks, I'll give that a try. It is odd though that it works for less than 10 .cr2 files, but not more. In the past (few years ago) i never had this issue.

[johnhudson922]

What about lowering the buffer size?

[johnhudson922]

It *appears* to work after converting from .cr2 to .xisf, however i'd still be interested in knowing what the issue was to begin with?

[pfile]

apparently due to the lack of support for 'incremental reading' of CR2 files, pixinsight has to hold the entirety of a CR2 file in memory while integrating each pixel stack. CR2 files are generally enormous, so this puts a lot of pressure on the computer's memory system - all CR2 files participating in the integration must be held in memory concurrently. xisf and fits files can be read incrementally so less stuff is forced to be held in memory at any one time.

having said that of course if you have a lot of ram and lots of VM backing store, this should be OK. i guess one thing would be to compute how large your 20 CR2 files are together and see if you're anywhere near starting to need to page (keeping in mind all the other processes that are currently paged-in and how much DRAM they are using...)

rob

[johnhudson922]

apparently due to the lack of support for 'incremental reading' of CR2 files, pixinsight has to hold the entirety of a CR2 file in memory while integrating each pixel stack. CR2 files are generally enormous, so this puts a lot of pressure on the computer's memory system - all CR2 files participating in the integration must be held in memory concurrently. xisf and fits files can be read incrementally so less stuff is forced to be held in memory at any one time.

having said that of course if you have a lot of ram and lots of VM backing store, this should be OK. i guess one thing would be to compute how large your 20 CR2 files are together and see if you're anywhere near starting to need to page (keeping in mind all the other processes that are currently paged-in and how much DRAM they are using...)

rob

Well, 300 total 20mb raw files is easily going to be more RAM than I have if thats how it processes the .cr2 files. Is this new though? Last time I used Pixinsight (3 years ago) it worked fine. Same files.

[pfile]

well on the software side, obviously there have been a ton of windows updates during that time... and also DCRAW was orphaned at some point so juan had to switch to libRAW. i guess if libRAW is less memory efficient than DCRAW that could also have something to do with it.

it is possible that if you configure windows to have more backing store the 300 deep stack could work, but then again there's a good chance your system would be brought to its knees paging all that stuff on and off the disk.

rob

[johnhudson922]

Ok so going forward, conver to .xisf first. Got it

[msmythers]

rob

I think it might be that in the last 3 years PI has gone to parallel thread processing for integration if I remember correctly. The stand alone debayering process has the same issue with large number of Raw format files if I remember correctly from other postings. It would make since that integration would also. I don't remember if turning off the parallel processing in the global preferences fixes the issue but it something to try. Personally I would go with the batch conversion to XISF but if hard drive space was an issue for some reason then turning off parallel processing might be an option. 



Mike

[pfile]

that could be the case... if for some reason each thread needs its own copy of the file in memory. but usually with lightweight threads the address space of all threads is shared with the parent process. so in theory each thread could share the file handle for the CR2 files. but who knows.

rob