Author Topic: Image Container running out of memory  (Read 11723 times)

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Image Container running out of memory
« on: 2009 April 11 12:59:33 »
Hi,

as suggested in another thread, I am now using an ImageContainer to process approx. 200 images with the same process (contained in a Process Container). When applying the process, Pixinsight consumes more and more memory, failing with "Out of memory" message after 20 or so pictures. The memory is released after I acknowledge those 180 or so "out of memory" dialog  boxes. I can see in the Windows Task Manager of memory gradually disappears and is release.

I know that processing 200 images in one run is rather unusual for Pixinsight, but still this indicates something in the Image Container is eating up memory.

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Image Container running out of memory
« Reply #1 on: 2009 April 13 11:48:43 »
Hi Georg,

I have processed more than 2000 images with ImageContainer without problems, applying a ProcessContainer with ACDNR and HistogramTransformation instances (if I remember well). As far as I know, there is no memory leak in ImageContainer routines. You might have found a memory leak somewhere else, of course, or something that I haven't seen, and in this case I am extremely interested in fixing it for version 1.5.

Does your ProcessContainer (or some instance(s) contained therein) generate temporary images?

I'm also concerned by the '180 or so "out of memory" dialog boxes' part. That shouldn't happen, of course...
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Image Container running out of memory
« Reply #2 on: 2009 April 13 12:03:03 »
Juan,

no, I dont generate temporary images (at least not that i know). The process I used is attached as a script below. During those 20 images it can do in one run, memory in the task manager goes from 0.9 to 1.3 GBytes. Each loaded image is a JPG, file size approx. 3.8 MBytes, 3888x2592 RGB pixel.

I got rid of those 120 out of memory messages by defining by disableing "continue on error" in the image container. Ultimately, all my 120 images were processed...

Hope that helps tracing the bug.

Georg


// Total time: 15.288 s
var p = new ProcessContainer;
with ( p )
{
   // Start time: 2009/04/11 20:21:33 UTC
   // Execution time: 1.295 s
   var p_0 = new PixelMath;
   with ( p_0 )
   {
      expression = "$T/vigntettierung";
      expression1 = "";
      expression2 = "";
      expression3 = "";
      useSingleExpression = true;
      symbols = "";
      use64BitWorkingImage = false;
      rescale = true;
      rescaleLower = 0.0000000000;
      rescaleUpper = 1.0000000000;
      truncate = true;
      truncateLower = 0.0000000000;
      truncateUpper = 1.0000000000;
      createNewImage = false;
      newImageId = "";
      newImageWidth = 0;
      newImageHeight = 0;
      newImageAlpha = false;
      newImageColorSpace = SameAsTarget;
      newImageSampleFormat = SameAsTarget;
   }

   add( p_0 );

   // Start time: 2009/04/11 20:21:36 UTC
   // Execution time: 13.743 s
   var p_1 = new MorphologicalTransformation;
   with ( p_1 )
   {
      operator = Dilation;
      interlacingDistance = 1;
      lowThreshold = 0.000000;
      highThreshold = 0.000000;
      numberOfIterations = 3;
      amount = 0.55;
      selectionPoint = 0.50;
      structureName = "5x5 Circular Structure";
      structureSize = 5;
      structureWayTable = // mask
      [
      [[0x00,0x01,0x01,0x01,0x00,
         0x01,0x01,0x01,0x01,0x01,
         0x01,0x01,0x01,0x01,0x01,
         0x01,0x01,0x01,0x01,0x01,
         0x00,0x01,0x01,0x01,0x00]]];
   }

   add( p_1 );
   setMask( 1, "maske" );

   // Start time: 2009/04/11 20:21:51 UTC
   // Execution time: 0.125 s
   var p_2 = new HistogramTransformation;
   with ( p_2 )
   {
      H = // c0, m, c1, r0, r1
      [
      [0.00000000, 0.50000000, 1.00000000, 0.00000000, 1.00000000],
      [0.00000000, 0.50000000, 1.00000000, 0.00000000, 1.00000000],
      [0.00000000, 0.50000000, 1.00000000, 0.00000000, 1.00000000],
      [0.07500000, 0.54150198, 1.00000000, 0.00000000, 1.00000000],
      [0.00000000, 0.50000000, 1.00000000, 0.00000000, 1.00000000]];
   }

   add( p_2 );
   setMask( 2, "maske" );

   // Start time: 2009/04/11 20:21:53 UTC
   // Execution time: 0.125 s
   var p_3 = new HistogramTransformation;
   with ( p_3 )
   {
      H = // c0, m, c1, r0, r1
      [
      [0.00000000, 0.50000000, 1.00000000, 0.00000000, 1.00000000],
      [0.00000000, 0.50000000, 1.00000000, 0.00000000, 1.00000000],
      [0.00000000, 0.50000000, 1.00000000, 0.00000000, 1.00000000],
      [0.13125000, 0.44687500, 1.00000000, 0.00000000, 1.00000000],
      [0.00000000, 0.50000000, 1.00000000, 0.00000000, 1.00000000]];
   }

   add( p_3 );
   setMask( 3, "maske" );
   invertMask( 3 );
}
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline NKV

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 677
Re: Image Container running out of memory
« Reply #3 on: 2009 May 13 21:51:06 »
Out of memory. PI1.5. XP32
Best regards,
Nikolay.

Offline NKV

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 677
Re: Image Container running out of memory
« Reply #4 on: 2009 June 08 20:16:51 »
PI1.5.2 on XP32
ImageContainer >ImageContainer > Out of memory.
See attachment.
Best regards,
Nikolay.

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Image Container running out of memory
« Reply #5 on: 2009 June 09 06:15:29 »
Juan,

I wonder if this may be related to issues with the garbarge collection in PJSR. In the script I recently wrote I had to add a
Code: [Select]
gc();
statement in the paint method to avoid memory overflow issues. Maybe the same problem affects process containers that essentially run a script on each of its images. Just an idea, I did not do any tests...

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Cheyenne

  • PixInsight Addict
  • ***
  • Posts: 146
    • Link to Picasa gallery of my astronomy photos
Re: Image Container running out of memory
« Reply #6 on: 2009 June 10 15:50:06 »
I'm going to toss out a "me too" on the out of memory condition, I was not using image containers.  I closed out all view, flushed everything, but the only way I could "resolve" it was to simply exit. 

I wasn't quite sure of the exact sequence of events, or what all I had "open" at the moment (I'm still exploring), but I do remember the last thing I ran before the problem was one of the scripts (I believe it might have been the noise evaluation).

If it does happen again, I will be more aware and try to grab logs and such.
Cheyenne Wills
Takahashi 130 TOA
Losmandy G11
SBIG STF8300M
Canon 20Da
SBIG ST-i + openPHD for autoguiding

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Image Container running out of memory
« Reply #7 on: 2009 June 11 15:21:28 »
Hi Nikolay, Georg, Cheyenne,

I am trying to reproduce this problem in order to fix it in 1.5.3. It seems it's a Windows-only issue, probably a memory fragmentation problem. I've been unable to reproduce it on all Linux and Mac OS X machines I have at hand, and I have tried it hard. I have Windows Vista x64 and XP SP3 x86 virtual machines with 6 GB and 4 GB of RAM, respectively, and ImageContainer runs fine on them with hundreds of images. I'll try to find a physical Windows box tomorrow.

Quote
I wonder if this may be related to issues with the garbarge collection in PJSR

Not at all. Deferred garbage collection problems are exclusive to scripts running on the JavaScript engine. PixInsight is pure C++ and uses deterministic garbage collection, immediate object destruction (for non-GUI objects) and deferred destruction synchronized with the event handler (for GUI objects). I always want to have full control, you know me.

Quote
I'm going to toss out a "me too" on the out of memory condition, I was not using image containers.

Of course this may happen. The JavaScript engine can eat up to 4 GB of RAM if necessary. And it will eat it, eventually. After severe RAM deprivation, available memory may become too fragmented and allocation of large blocks impossible. This means that even if you see enough free memory (use the mem command on PI's console), loading relatively large images may be impossible after an out-of-memory error. In theory, simply using the application (opening and closing images, etc.) should fix the problem "naturally", but one usually has to exit, as you did. We are always trying to improve on this, but the best way to prevent it is having plenty of RAM. This happens on all platforms, but is worse on Windows.

I know you're a Linux user (which is great! ;) ). How much RAM do you have?

Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Image Container running out of memory
« Reply #8 on: 2009 June 11 15:57:42 »
Juan,

I thought the Process Container, that you apply on an Image Container, essentially represents a script, and thus might inherit PJSR properties such as garbage collection issues when applied. But I am probably wrong.

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Image Container running out of memory
« Reply #9 on: 2009 June 11 16:46:13 »
Hi Georg,

No, both ProcessContainer and ImageContainer are 100% C++ code. I think this bug must be something relatively simple. I'll find it, of course, it's just a matter of time (but I have my fingers crossed, just in case) :)
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Cheyenne

  • PixInsight Addict
  • ***
  • Posts: 146
    • Link to Picasa gallery of my astronomy photos
Re: Image Container running out of memory
« Reply #10 on: 2009 June 11 17:55:35 »
Mine was in a Linux environment.

I will try to reproduce the problem

Cheyenne Wills
Takahashi 130 TOA
Losmandy G11
SBIG STF8300M
Canon 20Da
SBIG ST-i + openPHD for autoguiding

Offline Cheyenne

  • PixInsight Addict
  • ***
  • Posts: 146
    • Link to Picasa gallery of my astronomy photos
Re: Image Container running out of memory
« Reply #11 on: 2009 June 11 18:24:07 »
I was able to do some operations and get it to fail "on demand".  I will run using valgrind -- but in the meantime here is some info.


===== capture of linux top ===========
top - 19:09:09 up 4 days,  2:11,  4 users,  load average: 0.78, 0.39, 0.36
Tasks: 171 total,   2 running, 169 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us, 13.1%sy, 12.0%ni, 74.7%id,  0.2%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8235856k total,  4767936k used,  3467920k free,   513668k buffers
Swap:  9414040k total,        0k used,  9414040k free,  1218240k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                             
29083 cwills    21   1 2325m 2.2g  27m R  100 27.9   1:20.57 PixInsight

====================================================================
Here is my process console log:
================================
Initializing PixInsight...

JavaScript runtime initialized.

Installing 31 module(s):
/opt/PixInsight/bin/GREYCstoration-pxm.so
/opt/PixInsight/bin/ColorManagement-pxm.so
/opt/PixInsight/bin/IntensityTransformations-pxm.so
/opt/PixInsight/bin/ColorSpaces-pxm.so
/opt/PixInsight/bin/TIFF-pxm.so
/opt/PixInsight/bin/Geometry-pxm.so
/opt/PixInsight/bin/Deconvolution-pxm.so
/opt/PixInsight/bin/ImageIntegration-pxm.so
/opt/PixInsight/bin/NoiseGeneration-pxm.so
/opt/PixInsight/bin/Debayer-pxm.so
/opt/PixInsight/bin/JPEG-pxm.so
/opt/PixInsight/bin/Morphology-pxm.so
/opt/PixInsight/bin/Sandbox-pxm.so
/opt/PixInsight/bin/Global-pxm.so
/opt/PixInsight/bin/Compatibility-pxm.so
/opt/PixInsight/bin/MaskGeneration-pxm.so
/opt/PixInsight/bin/ImageRegistration-pxm.so
/opt/PixInsight/bin/ColorCalibration-pxm.so
/opt/PixInsight/bin/NoOperation-pxm.so
/opt/PixInsight/bin/BMP-pxm.so
/opt/PixInsight/bin/JPEG2000-pxm.so
/opt/PixInsight/bin/Wavelets-pxm.so
/opt/PixInsight/bin/DSLR_RAW-pxm.so
/opt/PixInsight/bin/Convolutions-pxm.so
/opt/PixInsight/bin/FITS-pxm.so
/opt/PixInsight/bin/PixelMath-pxm.so
/opt/PixInsight/bin/BackgroundModelization-pxm.so
/opt/PixInsight/bin/CloneStamp-pxm.so
/opt/PixInsight/bin/NoiseReduction-pxm.so
/opt/PixInsight/bin/Image-pxm.so
/opt/PixInsight/bin/RestorationFilters-pxm.so
* Scripting resources updated.
31 of 31 module(s) installed.

Processing script file: /opt/PixInsight/bin/startup.scp

PixInsight Core 01.05.02.0503 (x86)
Copyright © 2003-2009 Pleiades Astrophoto
----------------------------------------------------------------------
Welcome to PixInsight. Started 2009 Jun 12 01:21:04 J2454994.55630 UTC

* Parallel processing enabled: Using 4 logical processors.
* PSM AutoSave enabled. Auto-save period: 30 seconds.

Reading image:
/home/cwills/astro/Oct_23_2006/ic405x.fit
Reading FITS: 32-bit integers, 3 channel(s), 3522x2348 pixels: 100%

run -x=auto /opt/PixInsight/src/scripts/NoiseEvaluation.js

Processing script file: /opt/PixInsight/src/scripts/NoiseEvaluation.js

ic405x
Calculating noise standard deviation...

* Channel #0
Convergence reached after 7 iterations.
Noise standard deviation = 6.855e-04, N = 7836080 (94.76%)

* Channel #1
Convergence reached after 6 iterations.
Noise standard deviation = 4.539e-04, N = 7960043 (96.26%)

* Channel #2
Convergence reached after 7 iterations.
Noise standard deviation = 3.260e-04, N = 7944136 (96.06%)


run -x=auto /opt/PixInsight/src/scripts/DarkStructureEnhance.js

Processing script file: /opt/PixInsight/src/scripts/DarkStructureEnhance.js

RGBWorkingSpace: Processing view: Mask
0.010 s

ATrousWaveletTransformV1: Processing view: Mask1
Writing swap files...
520.07 MB/s
Processing channel #0
À trous wavelet transform: 100%
Processing wavelet layer #1
Applying layer bias/deringing: 100%
Processing wavelet layer #2
Applying layer bias/deringing: 100%
Processing wavelet layer #3
Applying layer bias/deringing: 100%
Processing wavelet layer #4
Applying layer bias/deringing: 100%
Processing wavelet layer #5
Applying layer bias/deringing: 100%
Processing wavelet layer #6
Applying layer bias/deringing: 100%
Inverse à trous wavelet transform: 100%
Normalizing sample values: 100%
Processing channel #1
À trous wavelet transform: 100%
Processing wavelet layer #1
Applying layer bias/deringing: 100%
Processing wavelet layer #2
Applying layer bias/deringing: 100%
Processing wavelet layer #3
Applying layer bias/deringing: 100%
Processing wavelet layer #4
Applying layer bias/deringing: 100%
Processing wavelet layer #5
Applying layer bias/deringing: 100%
Processing wavelet layer #6
Applying layer bias/deringing: 100%
Inverse à trous wavelet transform: 100%
Normalizing sample values: 100%
Processing channel #2
À trous wavelet transform: 100%
Processing wavelet layer #1
Applying layer bias/deringing: 100%
Processing wavelet layer #2
Applying layer bias/deringing: 100%
Processing wavelet layer #3
Applying layer bias/deringing: 100%
Processing wavelet layer #4
Applying layer bias/deringing: 100%
Processing wavelet layer #5
Applying layer bias/deringing: 100%
Processing wavelet layer #6
Applying layer bias/deringing: 100%
Inverse à trous wavelet transform: 100%
Normalizing sample values: 100%
8.780 s

PixelMath: Processing view: Mask
Writing swap files...
206.63 MB/s
Executing PixelMath expression: combined RGB/K channels:
$T -Mask1: 100%
Truncating to [0.0000000000,1.0000000000]: 100%
0.641 s
Writing swap files...
25.85 MB/s
Reading swap files...
1127.10 MB/s

ATrousWaveletTransformV1: Processing view: Mask
Writing swap files...
487.92 MB/s
Processing channel #0
À trous wavelet transform: 100%
Inverse à trous wavelet transform: 100%
Normalizing sample values: 100%
Processing channel #1
À trous wavelet transform: 100%
Inverse à trous wavelet transform: 100%
Normalizing sample values: 100%
Processing channel #2
À trous wavelet transform: 100%
Inverse à trous wavelet transform: 100%
Normalizing sample values: 100%
7.429 s

PixelMath: Processing view: Mask
Writing swap files...
93.61 MB/s
Executing PixelMath expression: combined RGB/K channels:
Mask-ic405x: 100%
Truncating to [0.0000000000,1.0000000000]: 100%
0.772 s

ConvertToGrayscale: Processing view: Mask
Writing swap files...
85.72 MB/s
In-place grayscale conversion: 100%
0.368 s

Rescale: Processing view: Mask
Writing swap files...
508.82 MB/s
Rescaling pixel values: 100%
0.113 s

ATrousWaveletTransformV1: Processing view: Mask
Writing swap files...
100.47 MB/s
À trous wavelet transform: 100%
Processing wavelet layer #2
Smoothing wavelet layer: 100%
Inverse à trous wavelet transform: 100%
Normalizing sample values: 100%
3.726 s

ProcessContainer: Processing view: ic405x
Writing swap files...
140.00 MB/s
HistogramTransformation
Processing channel #0: Histogram transformation: 100%
Processing channel #1: Histogram transformation: 100%
Processing channel #2: Histogram transformation: 100%
Mask: Masking from swap files...
1.975 s
Writing swap files...
534.48 MB/s
Reading swap files...
775.81 MB/s


run -x=auto /opt/PixInsight/src/scripts/NoiseEvaluation.js

Processing script file: /opt/PixInsight/src/scripts/NoiseEvaluation.js

ic405x
Calculating noise standard deviation...

* Channel #0
Convergence reached after 7 iterations.
Noise standard deviation = 6.866e-04, N = 7828044 (94.66%)

* Channel #1
Convergence reached after 6 iterations.
Noise standard deviation = 4.544e-04, N = 7950985 (96.15%)

* Channel #2
Convergence reached after 7 iterations.
Noise standard deviation = 3.268e-04, N = 7935281 (95.96%)


run -x=auto /opt/PixInsight/src/scripts/ExtractWaveletLayers.js

Processing script file: /opt/PixInsight/src/scripts/ExtractWaveletLayers.js


run -x=auto /opt/PixInsight/src/scripts/NoiseEvaluation.js

Processing script file: /opt/PixInsight/src/scripts/NoiseEvaluation.js

residual
Calculating noise standard deviation...

* Channel #0
Convergence reached after 9 iterations.
Noise standard deviation = 2.549e-05, N = 7598172 (91.88%)

* Channel #1
*** Error [137]: /opt/PixInsight/src/scripts/NoiseEvaluation.js, line 69: out of memory
*** Error [000]: /opt/PixInsight/src/scripts/NoiseEvaluation.js, line 113: Math.abs(): function call sequence aborted.
« Last Edit: 2009 June 11 18:32:37 by Cheyenne »
Cheyenne Wills
Takahashi 130 TOA
Losmandy G11
SBIG STF8300M
Canon 20Da
SBIG ST-i + openPHD for autoguiding

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Image Container running out of memory
« Reply #12 on: 2009 August 03 09:17:12 »
Hi,

I have managed to identify and fix this bug. It will be fixed in the next version, which is due in September, on all supported platforms.

This bug is actually related to a similar problem detected with the BatchFormatConversion script.

Thanks to all who found and confirmed this bug, and sorry for the inconveniences.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline David Serrano

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 503
Re: Image Container running out of memory
« Reply #13 on: 2009 August 03 11:52:45 »
But... you were on holidays, weren't you? :P
--
 David Serrano