1.8.7 out of memory errors

[TinySpeck]

I'm stacking the largest collection of subs I've ever tried, and running into out-of-memory errors during ImageIntegration.  I'm closing all programs except PI.  I have the latest PI release and updates (including one from today), and the latest Windows 10 with updates.  I'm using the new automatic buffer sizes option (but also fixed buffer sizes).  I got through the initial 'no pixel rejection' integration after a couple out-of-memory failures, but now I can't proceed with pixel rejection.  After reading the subs in a half a minute or so from the cache, the console reports:

Integration of 1806 images:
Pixel combination .................. average
Output normalization ............... additive + scaling
Weighting mode ..................... custom keyword: SUB_WEIGHT
Scale estimator .................... iterative k-sigma / BWMV
Pixel rejection .................... Winsorized sigma clipping
Rejection normalization ............ scale + zero offset
Rejection clippings ................ low=yes high=yes
Rejection parameters ............... sigma_low=4.000 sigma_high=4.000 cutoff=4.000
Large-scale rejection clippings .... low=no high=yes
Large-scale rejection parameters ... lsr_layers_low=2 lsr_grow_low=2 lsr_layers_high=3 lsr_grow_high=4

* Available physical memory: 11.869 GiB
* Allocated pixel buffer: 319 rows, 8.894 GiB
* Using 12 concurrent pixel stack(s), 1.338 GiB


* Integrating channel 1 of 3:
  Analyzing pixel rows:     0 ->   318:   3%
*** Error: Out of memory
<* failed *>

There is a pause of a couple minutes after "analyzing pixel rows", and then the memory failure happens.  The amount of physical memory reported varies by a GiB or two each time I try this.  I have tried rebooting.  I had success once on the initial integration by setting fixed buffer sizes of 32 MiB and stack of 2048 MiB, but now that doesn't seem to help.

It seems like PI should never run out of memory; it should adjust its processes to use what's available.  Is this problem a function of the new automatic buffer sizes?

Can anyone suggest anything?

[Juan Conejero]

It seems like PI should never run out of memory; it should adjust its processes to use what's available.

Not at all. ImageIntegration requires a minimum amount of working space, whose size is proportional to the number of input images, and depends largely on the selected pixel rejection parameters. We still haven't achieved perpetual motion—although keep trying

One of the factors that make this task impossible in your case is the fact that you are using large-scale pixel rejection. This requires a lot of working data to support multiscale transforms. If you disable this option, maybe you'll be able to perform the integration of these 1806 images.

At any rate, if you plan on carrying out tasks like these, I would recommend a much more capable machine, including a powerful processor with many cores (such as one of the latest AMD Ryzen processors) and RAM in the range from 64 to 128 GiB. For performance and stability reasons, I would also recommend using Linux or FreeBSD for these highly demanding tasks.

[TinySpeck]

Thanks, Juan, I see what you mean.  I will turn off large-scale pixel rejection and try again.  It also occurred to me that I could break this job into two parts and then combine them like you'd do with HDR processing.

And good luck with that perpetual motion machine.