Pixinsight 1.8.7: new memory dynamics and problems with lots of RAM on OS X

[scottstirling]

Hi - I installed 1.8.7 and latest patches last night.  So far, working well, but I ran into two things with new memory usage dynamics today:

1. ImageIntegration - using over 20 GB RAM on my iMac at one point, while integrating 146 Canon 6D RGB subframes (LinearFit clipping) with Automatic buffering option enabled.

2. StarAlignment - Register/Union Mosaic w/Thin Plate Splines model and both "Distortion correction" and "Local distortion" enabled - uses up all the memory and swap on my 40 GB iMac while doing a mosaic alignment of any two images.  Mosaic alignment with Projective Transformation model works for the same images, but 2-D Surface Splines w/Distortion Correction uses up all the memory on the host and does not complete successfully.

These are definitely new behaviors. Pixinsight 1.8.6x never used more than a couple GBs of RAM no matter how much free the OS had.  I saw in the 1.8.7 release notes about the auto-sizing of buffering for integration, new linear fit algorithm implementation, and the mosaic registration improvements, (the mosaic  was 4 full frame subframes overlapping but I've seen a lot larger ones produced by Pixinsight in the past), and these are all great features, so, figured I'd share the report of my experience so far on the computer I bought mainly for astrophotography processing.

Some screenshots attached.

Host computer details:
iMac Retina 5k, 27-inch, 2017
Memory 40 GB 2400 MHz DDR4
Processor 3.5 GHz Intel Core i5
1 TB storage, about 60 GB free

How to limit the max amount of memory Pixinsight will use now, just in case?

Thank you,
Scott Stirling
Graham, NC

[pfile]

out of curiosity were you directly integrating CR2 files or were they xisf files, and if so did they have compression turned on?

rob

[scottstirling]

xisf files, no compression - each one is about 242 MB.

I've unchecked "Automatic buffer sizes" on ImageIntegration and selected Sigma Clipping and am trying that now. I'll try with Linear Fit again next.

I managed to create a mosaic from 4 integration masters, but only with Projective Transformation.  Every time I try to run with Thin Plate Splines, progress gets to 75 - 86% in the "Local distortion model" phase and then the process spins til it consumes remaining free memory and swap.

Scott

[pfile]

ok - yeah maybe the problem is with the automatic buffer size, after all that is a new feature.

it's not uncommon to run out of memory on prior versions when using file formats that don't support incremental reading, which is why i wondered about the file format.

rob

[johnpane]

1. ImageIntegration - using over 20 GB RAM on my iMac at one point, while integrating 146 Canon 6D RGB subframes (LinearFit clipping)

I'm not seeing the problem in this example (unlike the other you reported).

[scottstirling]

I am wondering if this from pre-1.8.7 is causing an issue: http://dslr-astrophotography.com/improve-performance-pixinsight-swap-file-storage/

I tweaked swap settings for 1.8.6 per that article, several months ago. I uninstalled Pixinsight 1.8.6 when installing 1.8.7. The settings are preserved nevertheless.

I will try without these. - Scott S

P.S. the "Swap file compression" flag in the screenshot, I just checked now to try it.  It was unchecked before now.

[scottstirling]

I removed those tmp dirs from preferences and restarted Pixinsight.

Register/Union Mosaic w/Thin Plate Splines still hangs at "Local distortion model" and then Pixinsight starts using more and more memory and never finishes:

From the process console:

* Local distortion model:
Matched stars : 30028
Outliers      : 7084 (19.09%)
delta_x_min   :  -0.39 px
delta_x_max   :  +0.52 px
delta_y_min   :  -0.37 px
delta_y_max   :  +0.53 px

"Building recursive surface subsplines:  75%"

I will try again with local distortion model unchecked.

Scott S

[scottstirling]

Unchecking the "Local distortion" option works around the memory issue with Register/Union - Mosaic w/Thin Plate Splines.

No issues using Thin Plate Splines w/Register Union-Mosaic as long as I do not check the "Local distortion" checkbox. Checked, somehow causes Pixinsight to go into a memory usage loop, it seems.

Thank you,
Scott S

[pfile]

while the whole dataset is too big, do you have a subset of files that would cause the memory consumption to rise while running ImageIntegration? juan might need this to reproduce the problem.

also i guess it was a red herring but i don't think the PI swap would be related to this. the swap area is (to my knowledge) not used by ImageIntegration.

rob

[scottstirling]

Yes, I will try to narrow down a repeatable test case with fewer files.

Scott

[Juan Conejero]

Hi Scott,

1. ImageIntegration - using over 20 GB RAM on my iMac at one point, while integrating 146 Canon 6D RGB subframes (LinearFit clipping) with Automatic buffering option enabled.

Please read the tooltip for the automatic buffer sizes parameter:

Usually this is the best option to optimize image integration performance, but be aware that for large data sets and low-memory machines (relative to the total size of the data) the process may use too many system resources. If you want finer control on system resources usage, disable this option and tweak the values of the buffer size and stack size parameters.

So this is not a bug. 146 DSLR frames is a huge data set for your machine, and the linear fit clipping algorithm consumes even more memory. Disable automatic buffer sizes and fine tune the buffer size and stack size parameters, or leave them with default values, which is clearly suboptimal but will work for sure.

That said, to prevent cases like this one I'll add a new parameter to control the maximum fraction of the available RAM that ImageIntegration is allowed to use. I'll release this as an update.

2. StarAlignment - Register/Union Mosaic w/Thin Plate Splines model and both "Distortion correction" and "Local distortion" enabled - uses up all the memory and swap on my 40 GB iMac while doing a mosaic alignment of any two images.

This should not happen under normal conditions. I need two of these mosaic frames where this problem can be reproduced to diagnose it.