Out of memory when integrating

[Golfox2]

Hi guys,

I encounter a lot of Out of Memory when integrating Bias, Lights or Darks.
It happens either with my Canon 80Da (24mpx) or my Canon 6D mkii (27.1mpx) at the integration step (see attached).

I tried to lower the buffer from 16 to 8 and the problem disappeared with my 80Da images, but still remains with my 6D.
I tried to decrease the buffer to 4, 2 and 1, and increase the stack size from 1024 to 2048 and 4096, but the issue is still here :

Code: [Select]

* Waiting for running tasks to terminatedone
*** Error:
*** Error: Out of memory
<* failed *>
I am sure it is not a computer related problem, I have 16GB RAM, here are the specs of my computer :

• CPU : AMD Ryzen 1600x
• SSD (OS): Intel M.2 600p Series
• MB : Asus ROG Strix B350-I Gaming
• RAM : Corsair Vengeance RGB 16GB 2400MHz

Screenshot attached

Thank you !
Golfox2

[discardedastro]

I'm seeing a lot of this, too. Windows 10, 32G of RAM.

Stack of 91 5496x3672px images, linear fit w/ local norm, even with 0/1024MB set in the integration. Machine has 15+ gigs of RAM free.

I've also seen memory errors with more than a few parallel processes in other stages of processing, which is frustrating.

[pfile]

CR2 files need to be loaded into memory in their entirety before PI can work on them. thus the load on the memory system is pretty enormous.

XISF files can be "incrementally" read so that PI does not have to keep the entire image in memory while integrating. this reduces the memory footprint a lot.

at the very least you should be converting your CR2 files to XISF. however, if you were to calibrate your files first they would by default be converted to XISF by that process.

any reason you're not calibrating?

rob

[discardedastro]

For what it's worth I'm seeing this with XISF files.

Code: [Select]

Integration of 91 images:
Pixel combination .................. average
Output normalization ............... local
Weighting mode ..................... custom keyword: SSWEIGHT
Scale estimator .................... iterative k-sigma / BWMV
Range rejection .................... range_low=0.000000
Pixel rejection .................... linear fit clipping
Rejection normalization ............ local
Rejection clippings ................ low=yes high=yes
Rejection parameters ............... lfit_low=5.000 lfit_high=2.500

* Using 1 concurrent pixel stack(s) = 7.67 MiB
Integrating pixel rows:     0 ->     0:   0%Uncompressing block (lz4+sh): 57.499 MiB -> 76.986 MiB (25.31%)
*** Error: Out of memory
<* failed *>
PI seems to spin up to about 6GB of memory usage (the PC I should point out has 17+ GB "available" according to Windows) and then throws the out of memory error.

[pfile]

well one caveat to what i said is that compressed XISF files will behave like CR2 files in this respect. the entire thing needs to be loaded into memory to be decompressed... i think XISF compression is off by default so i didn't mention it.

its possible that the task manager does not sample fast enough for you to see the actual out of memory condition occur, which might account for the fact that it seems that there's no actual out of memory condition in windows.

rob

[discardedastro]

well one caveat to what i said is that compressed XISF files will behave like CR2 files in this respect. the entire thing needs to be loaded into memory to be decompressed... i think XISF compression is off by default so i didn't mention it.

its possible that the task manager does not sample fast enough for you to see the actual out of memory condition occur, which might account for the fact that it seems that there's no actual out of memory condition in windows.

rob

Doesn't seem disabled by default (I don't recall setting it, at least) but I've turned it off now - that'd make a lot of sense.

[discardedastro]

Just to confirm this was the cause - I'd suggest a nice improvement in future versions would be to check to see if any of the input files are compressed or otherwise require loading fully into memory, and if they would, warn in the logs prior to integrating (or if it would exhaust memory, abort loudly and point out that this is the issue). Otherwise it's not terribly obvious that that's what happening.

[Golfox2]

Hi guys,

pfile : I only calibrate lights and flats. So when I do the master dark and master bias, I directly use the .CR2 files. But I could convert them in xisf, no problem at all. Which process should I use to do that ?

discardedastro : +1 for the loudly pointing out of the issue

should I disable XISF compression then ?

I managed to get the issue somehow solved by increasing my pagefile.sys size on both my SSD (C:/) and HDD (D:/), now the problem is gone, but for how long ?

[pfile]

you can use the batchformatconversion script. and yes you should turn off cost compression.

if you solved it by increasing the windows pagefile it should work ok as long as you don’t increase the number of input frames too much.

rob

[Golfox2]

I'll give it a try, thanks !
PS : I don't find the xisf compression option...

[wvanreeven]

It is in the XISF section in the Format Explorer. Select XISF and then click Edit Preferences to the lower right to enable or disable compression.


HTH, Wouter