PI1.55 BatchFormatConversion script eats memory

[georg.viehoever]

Hi,

I just tried to convert a batch of FITS images to Jpg using the BatchFormatConversion script. This fails after 6 images with a memory error. The task manager (Vista  ) clearly show that memory is consumed, which is released only when the script is canceled. Adding a gc() here does not help.

Code: [Select]

...
  this.convertFiles = function()
   {
      for ( var i = 0; i < this.inputFiles.length; ++i )
      {
         gc();
         var w = ImageWindow.open( this.inputFiles[i] );
...


Georg

Georg

[Niall Saunders]

Hi Georg,

Have another look at my Batch deBayer PJSR script - I had nasty 'garbage collection' issues as well but, between Juan and myself, we were able to 'tame' the problem.

Maybe using gc() at the equivalent points in the Batch Conversion script would be the solution?

Cheers,

[georg.viehoever]

Hi Niall,

Thanks for the hint. I think I already placed the gc() inside the central loop that apparently consumes the memory. BTW gc(true) also does not help, it just appears to be slower. So I still have no idea how to resolve this.

Georg

[Juan Conejero]

Hi Georg,

Try calling processEvents() and gc() inside the main loop:

Code: [Select]

         for ( var j = 0; j < w.length; ++j )
         {
            // Write the output image.
            // Force ImageWindow to disable all format and security features:
            // * Query format-specific options
            // * Warning messages on missing format features (icc profiles, etc)
            // * Strict image writing mode (ignore lossy image generation)
            // * Overwrite verification/protection
            w[j].saveAs( outputFilePath, false, false, false, false );

            // Close the image window.
            // Note that we haven't called w.show(), so it is hidden.
            w[j].close();

            // Empty PI's event queue
            processEvents();

            // Perform a hard garbage collection
            gc();
         }

Let's see if this forces the images to close immediately. The problem here seems to be that PI is scheduling destruction of ImageWindow's internal structures, for the sake of performance. With little free RAM this can cause out-of-memory problems.

[georg.viehoever]

Juan,

Thanks for looking into this.

I tried with PI 1.56. The patch does not help. BTW: The problem here is *not* caused by particularly low memory in Windows. I think Linux would run out of memory as well, only slightly later. I am starting with more than 1 GByte of RAM free. However, the images that need to be converted are large (each fit is 150 Mbytes). I dont think that low memory on the windows side is the isssue here. Rather, PI does not clean away something after it is done with the conversion of one file.

Georg

[Juan Conejero]

Hi Georg,

I've just made a test on a virtual Linux machine (Fedora Core 8 i386) downgraded to have 2 GB of RAM. I can convert 19 FITS files like yours (about 150 MB each) to JPEG format using the BatchFormatConversion script (out-of.the-box version, without modifications). While reading file # 20, I always get an out of memory error, so you're right. Indeed gc() neither processEvents() help at all. For some reason, PixInsight does not destroy all internal image structures when it closes an image during script execution.

There's no memory leak at all; it's just that some memory deallocations seem to be scheduled until the end of the script. The evolution of memory allocation during script execution is very interesting. PI steadily increases its memory consumption until it reaches about the 85% of the physical memory (yes, Linux can do that ). Then it decreases to about the 60%. Then it suddenly grows again to the 87%, and at this point the out of memory exception is thrown by internal PCL routines.

Be sure this problem will be fixed in the next version; I'm working on it right now. Thank you for discovering it!

[georg.viehoever]

Great! Always happy to help tracking down a problem!

Looking forward to PI1.5.7!

Georg

[Juan Conejero]

Hi again,

Since PI 1.5.7 will probably be delayed a bit (I plan on relaxing a little on August), here's a modified version of the BatchFormatConversion script that has no memory consumption problems:

Code: [Select]

/*
   BatchFormatConversion v1.0 - ### Special bugfix version

   A batch image format conversion utility.

   Copyright (C) 2009 Pleiades Astrophoto S.L.

   This program is free software: you can redistribute it and/or modify it
   under the terms of the GNU General Public License as published by the
   Free Software Foundation, version 3 of the License.

   This program is distributed in the hope that it will be useful, but WITHOUT
   ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
   FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
   more details.

   You should have received a copy of the GNU General Public License along with
   this program.  If not, see <http://www.gnu.org/licenses/>.
*/

#feature-id    Utilities > BatchFormatConversion

#feature-info  A batch image format conversion utility.<br/>\
   <br/> \
   This script allows you to select a set of input image files, an optional output \
   directory, and an output format. The script then iterates reading each input file \
   and saving it on the output directory with the specified output format.<br>\
   <br>\
   This script is very useful when you have to convert several images &mdash;no \
   matter if dozens or hundreds&mdash; from one format to another automatically. \
   It saves you the work of opening and saving each file manually one at a time.<br/>\
   <br/> \
   Written by Juan Conejero (PTeam)<br/>\
   Copyright &copy; 2009 Pleiades Astrophoto S.L.

#feature-icon  BatchFormatConversion.xpm

#include <pjsr/Sizer.jsh>
#include <pjsr/FrameStyle.jsh>
#include <pjsr/TextAlign.jsh>
#include <pjsr/StdButton.jsh>
#include <pjsr/StdIcon.jsh>
#include <pjsr/UndoFlag.jsh>  // ### Required by bugfix modifications

#define DEFAULT_EXTENSION  ".jpg"

#define WARN_ON_NO_OUTPUT_DIRECTORY 1

#define VERSION "1.0"
#define TITLE   "BatchFormatConversion"

/**
 * Batch Conversion Engine
 */
function BatchFormatConversionEngine()
{
   this.inputFiles = new Array;
   this.outputDirectory = "";
   this.outputExtension = DEFAULT_EXTENSION;
   this.overwriteExisting = false;

   this.convertFiles = function()
   {
      for ( var i = 0; i < this.inputFiles.length; ++i )
      {
         var w = ImageWindow.open( this.inputFiles[i] );

         if ( w.length == 0 && i+1 < this.inputFiles.length )
         {
            var msg = new MessageBox( "<p>Unable to load input image file:</p>" +
                                      "<p>" + this.inputFiles[i] + "</p>" +
                                      "<p><b>Continue batch format conversion ?</b></p>",
                                      TITLE, StdIcon_Error, StdButton_Yes, StdButton_No );
            if ( msg.execute() == StdButton_No )
               break;
         }

         // Build the output file path from its separate components:
         //    drive+directory+name+extension
         // Of course, drive is always an empty string under Linux/UNIX/OSX

         var fileDir = (this.outputDirectory.length != 0) ?
            this.outputDirectory : File.extractDrive( this.inputFiles[i] ) +
                                   File.extractDirectory( this.inputFiles[i] );

         // Ensure that our directory string ends with a slash separator.
         if ( fileDir.length != 0 && fileDir.charAt( fileDir.length-1 ) != '/' )
            fileDir += '/';

         var fileName = File.extractName( this.inputFiles[i] );

         var outputFilePath = fileDir + fileName + this.outputExtension;

         if ( !this.overwriteExisting && File.exists( outputFilePath ) )
         {
            // Obtain a nonexisting file name by appending an underscore and a
            // growing integer to the output file name.
            for ( var u = 1; ; ++u )
            {
               var tryFilePath = File.appendToName( outputFilePath, '_' + u.toString() );
               if ( !File.exists( tryFilePath ) )
               {
                  outputFilePath = tryFilePath;
                  break;
               }
            }
         }

         for ( var j = 0; j < w.length; ++j )
         {
            // Write the output image.
            // Force ImageWindow to disable all format and security features:
            // * Query format-specific options
            // * Warning messages on missing format features (icc profiles, etc)
            // * Strict image writing mode (ignore lossy image generation)
            // * Overwrite verification/protection
            w[j].saveAs( outputFilePath, false, false, false, false );

            /* ###
               Destroy the image in this window. This releases all pixel
               buffers and most internal control structures.
            */
            with ( w[j].mainView )
            {
               beginProcess( UndoFlag_NoSwapFile );
               image.free();
               endProcess();
            }

            /* ###
               Close the image window, unconditionally.
               Note that we haven't called w[j].show(), so it is hidden.
            */
            w[j].forceClose();

            /* ###
               Tell the garbage collector that w[j] is no longer required.
            */
            delete w[j];

            /* ###
               Perform a thorough (hard) garbage collection.
            */
            gc();
         }

         /* ###
            Destroy also the array of image windows.
         */
         delete w;
      }
   }
}

var engine = new BatchFormatConversionEngine;

/**
 * Batch Conversion Dialog
 */
function BatchFormatConversionDialog()
{
   // Add all properties and methods of the core Dialog object to this object.
   this.__base__ = Dialog;
   this.__base__();

   //

   this.helpLabel = new Label( this );
   this.helpLabel.frameStyle = FrameStyle_Box;
   this.helpLabel.margin = 4;
   this.helpLabel.wordWrapping = true;
   this.helpLabel.useRichText = true;
   this.helpLabel.text = "<p><b>" + TITLE + " v" + VERSION + "</b> &mdash; " +
                         "A batch image format conversion utility.</p>" +
                         "<p>Copyright &copy; 2009 Pleiades Astrophoto</p>";
   //

   this.files_TreeBox = new TreeBox( this );
   this.files_TreeBox.multipleSelection = true;
   this.files_TreeBox.rootDecoration = false;
   this.files_TreeBox.alternateRowColor = true;
   this.files_TreeBox.setMinSize( 500, 200 );
   this.files_TreeBox.numberOfColumns = 1;
   this.files_TreeBox.headerVisible = false;

   for ( var i = 0; i < engine.inputFiles.length; ++i )
   {
      var node = new TreeBoxNode( this.files_TreeBox );
      node.setText( 0, engine.inputFiles[i] );
   }

   this.filesAdd_Button = new PushButton( this );
   this.filesAdd_Button.text = " Add ";
   this.filesAdd_Button.toolTip = "<p>Add image files to the input images list.</p>";

   this.filesAdd_Button.onClick = function()
   {
      var ofd = new OpenFileDialog;
      ofd.multipleSelections = true;
      ofd.caption = "Select Images";
      ofd.loadImageFilters();

      if ( ofd.execute() )
      {
         this.dialog.files_TreeBox.canUpdate = false;
         for ( var i = 0; i < ofd.fileNames.length; ++i )
         {
            var node = new TreeBoxNode( this.dialog.files_TreeBox );
            node.setText( 0, ofd.fileNames[i] );
            engine.inputFiles.push( ofd.fileNames[i] );
         }
         this.dialog.files_TreeBox.canUpdate = true;
      }
   };

   this.filesClear_Button = new PushButton( this );
   this.filesClear_Button.text = " Clear ";
   this.filesClear_Button.toolTip = "<p>Clear the list of input images.</p>";

   this.filesClear_Button.onClick = function()
   {
      this.dialog.files_TreeBox.clear();
      engine.inputFiles.length = 0;
   };

   this.filesInvert_Button = new PushButton( this );
   this.filesInvert_Button.text = " Invert Selection ";
   this.filesInvert_Button.toolTip = "<p>Invert the current selection of input images.</p>";

   this.filesInvert_Button.onClick = function()
   {
      for ( var i = 0; i < this.dialog.files_TreeBox.numberOfChildren; ++i )
         this.dialog.files_TreeBox.child( i ).selected =
               !this.dialog.files_TreeBox.child( i ).selected;
   };

   this.filesRemove_Button = new PushButton( this );
   this.filesRemove_Button.text = " Remove Selected ";
   this.filesRemove_Button.toolTip = "<p>Remove all selected images from the input images list.</p>";

   this.filesRemove_Button.onClick = function()
   {
      engine.inputFiles.length = 0;
      for ( var i = 0; i < this.dialog.files_TreeBox.numberOfChildren; ++i )
         if ( !this.dialog.files_TreeBox.child( i ).selected )
            engine.inputFiles.push( this.dialog.files_TreeBox.child( i ).text( 0 ) );
      for ( var i = this.dialog.files_TreeBox.numberOfChildren; --i >= 0; )
         if ( this.dialog.files_TreeBox.child( i ).selected )
            this.dialog.files_TreeBox.remove( i );
   };

   this.filesButtons_Sizer = new HorizontalSizer;
   this.filesButtons_Sizer.spacing = 4;
   this.filesButtons_Sizer.add( this.filesAdd_Button );
   this.filesButtons_Sizer.addStretch();
   this.filesButtons_Sizer.add( this.filesClear_Button );
   this.filesButtons_Sizer.addStretch();
   this.filesButtons_Sizer.add( this.filesInvert_Button );
   this.filesButtons_Sizer.add( this.filesRemove_Button );

   this.files_GroupBox = new GroupBox( this );
   this.files_GroupBox.title = "Input Images";
   this.files_GroupBox.sizer = new VerticalSizer;
   this.files_GroupBox.sizer.margin = 4;
   this.files_GroupBox.sizer.spacing = 4;
   this.files_GroupBox.sizer.add( this.files_TreeBox, 100 );
   this.files_GroupBox.sizer.add( this.filesButtons_Sizer );

   //

   this.outputDir_Edit = new Edit( this );
   this.outputDir_Edit.readOnly = true;
   this.outputDir_Edit.text = engine.outputDirectory;
   this.outputDir_Edit.toolTip =
      "<p>If specified, all converted images will be written to the output directory.</p>" +
      "<p>If not specified, converted images will be written to the same directories " +
      "of their corresponding input images.</p>";

   this.outputDirSelect_Button = new PushButton( this );
   this.outputDirSelect_Button.text = " Select ";
   this.outputDirSelect_Button.toolTip = "<p>Select the output directory.</p>";

   this.outputDirSelect_Button.onClick = function()
   {
      var gdd = new GetDirectoryDialog;
      gdd.initialPath = engine.outputDirectory;
      gdd.caption = "Select Output Directory";

      if ( gdd.execute() )
      {
         engine.outputDirectory = gdd.directory;
         this.dialog.outputDir_Edit.text = engine.outputDirectory;
      }
   };

   this.outputDir_GroupBox = new GroupBox( this );
   this.outputDir_GroupBox.title = "Output Directory";
   this.outputDir_GroupBox.sizer = new HorizontalSizer;
   this.outputDir_GroupBox.sizer.margin = 4;
   this.outputDir_GroupBox.sizer.spacing = 4;
   this.outputDir_GroupBox.sizer.add( this.outputDir_Edit, 100 );
   this.outputDir_GroupBox.sizer.add( this.outputDirSelect_Button );

   //

   this.outputExt_Label = new Label( this );
   this.outputExt_Label.text = "Output extension:";
   this.outputExt_Label.textAlignment = TextAlign_Right|TextAlign_VertCenter;

   this.outputExt_Edit = new Edit( this );
   this.outputExt_Edit.text = engine.outputExtension;
   this.outputExt_Edit.setFixedWidth( this.font.width( "MMMMMM" ) );
   this.outputExt_Edit.toolTip =
      "<p>Specify a file extension to identify the output file format.</p>";

   this.outputExt_Edit.onEditCompleted = function()
   {
      // Trim whitespace at both ends of the file extension string
      // TOTHINK: Add trim(), trimLeft() and trimRight() methods to String
      //          (which departs from ECMAScript specification) ?
      var ext = ""
      var i = 0;
      var j = this.text.length;
      for ( ; i < j && this.text.charAt( i ) == ' '; ++i ) {}
      for ( ; --j > i && this.text.charAt( j ) == ' '; ) {}
      if ( i <= j )
         ext = this.text.substring( i, j+1 ).toLowerCase();
            // Image extensions are always lowercase in PI/PCL.

      // Use the default extension if empty.
      // Ensure that ext begins with a dot character.
      if ( ext.length == 0 || ext == '.' )
         ext = DEFAULT_EXTENSION;
      else if ( ext.charAt( 0 ) != '.' )
         ext = '.' + ext;

      this.text = engine.outputExtension = ext;
   };

   this.overwriteExisting_CheckBox = new CheckBox( this );
   this.overwriteExisting_CheckBox.text = "Overwrite existing files";
   this.overwriteExisting_CheckBox.checked = engine.overwriteExisting;
   this.overwriteExisting_CheckBox.toolTip =
      "<p>Allow overwriting of existing image files.</p>" +
      "<p><b>* Warning *</b> Enabling this option may lead to irreversible data loss.</p>";

   this.overwriteExisting_CheckBox.onClick = function( checked )
   {
      engine.overwriteExisting = checked;
   }

   this.options_Sizer = new HorizontalSizer;
   this.options_Sizer.spacing = 4;
   this.options_Sizer.add( this.outputExt_Label );
   this.options_Sizer.add( this.outputExt_Edit );
   this.options_Sizer.addSpacing( 12 );
   this.options_Sizer.add( this.overwriteExisting_CheckBox );
   this.options_Sizer.addStretch();

   //

   this.ok_Button = new PushButton( this );
   this.ok_Button.text = " OK ";

   this.ok_Button.onClick = function()
   {
      this.dialog.ok();
   };

   this.cancel_Button = new PushButton( this );
   this.cancel_Button.text = " Cancel ";

   this.cancel_Button.onClick = function()
   {
      this.dialog.cancel();
   };

   this.buttons_Sizer = new HorizontalSizer;
   this.buttons_Sizer.spacing = 4;
   this.buttons_Sizer.addStretch();
   this.buttons_Sizer.add( this.ok_Button );
   this.buttons_Sizer.add( this.cancel_Button );

   //

   this.sizer = new VerticalSizer;
   this.sizer.margin = 6;
   this.sizer.spacing = 6;
   this.sizer.add( this.helpLabel );
   this.sizer.addSpacing( 4 );
   this.sizer.add( this.files_GroupBox, 100 );
   this.sizer.add( this.outputDir_GroupBox );
   this.sizer.add( this.options_Sizer );
   this.sizer.add( this.buttons_Sizer );

   this.windowTitle = TITLE + " Script";
   this.userResizable = true;
   this.adjustToContents();
}

// Our dialog inherits all properties and methods from the core Dialog object.
BatchFormatConversionDialog.prototype = new Dialog;

/*
 * Script entry point.
 */
function main()
{
   jsAutoGC = true;

   console.hide();

   // Show our dialog box, quit if cancelled.
   var dialog = new BatchFormatConversionDialog();
   for ( ;; )
   {
      if ( dialog.execute() )
      {
         if ( engine.inputFiles.length == 0 )
         {
            var msg = new MessageBox( "No input files have been specified!",
                                      TITLE, StdIcon_Error, StdButton_Ok );
            msg.execute()
            continue;
         }

#ifneq WARN_ON_NO_OUTPUT_DIRECTORY 0
         if ( engine.outputDirectory.length == 0 )
         {
            var msg = new MessageBox(
                  "<p>No output directory has been specified.</p>" +
                  "<p>Each converted image will be written to the directory of " +
                  "its corresponding input file.<br>" +
                  "<b>Are you sure ?</b></p>",
                  TITLE, StdIcon_Warning, StdButton_Yes, StdButton_No );
            if ( msg.execute() != StdButton_Yes )
               continue;
         }
#endif
         // Perform batch file format conversion and quit.
         console.show();
         console.abortEnabled = true;
         engine.convertFiles();

         var msg = new MessageBox(
            "Do you want to perform another format conversion ?",
            TITLE, StdIcon_Question, StdButton_Yes, StdButton_No );
         if ( msg.execute() == StdButton_Yes )
            continue;
      }

      break;
   }
}

main();
Just replace your current BatchFormatConversion.js file with the above code.

You can find all modifications by searching the "###" sequence (without quotes). The bugfix is a somewhat brute force workaround: just destroy the image transported by the image window at each iteration. Not the most elegant solution, but simple and certainly works

Note that a similar fix could be applied to solve the current problems with ImageContainer: a script that iterates a set of image files just like BatchFormatConversion, loads them, applies a hard-coded (but easy to write) ProcessContainer, and writes them back to disk. This would fix for example Harry's issue with Rescale applied to a set of images prior to register them.

Back to a nice Sunday of gardening and good food and drink 

[georg.viehoever]

Juan,

Thanks a lot! Harry, myself and others certainly appreciate the hard work you are doing!

Enjoy your vacation and come back with fresh ideas and new energy!

Georg

[georg.viehoever]

Hi Juan,

your patch works! Converted >500 images without getting a memory shortage.

Thanks,
Georg

[Juan Conejero]

Hi Georg,

Glad to know that. Of course this problem (and its ImageContainer counterpart) will be fixed in the next version, due in September. Naturally, I'll be in touch with this forum and all PI users during August, so don't hesitate to post any further problems or doubts here.