Need to perform file cleanup in JavaScript

[CCDer]

I've finished modifying the supplied PixInsight Batch Preprocessing script to do a single batch calibrate, register, and integrate of .xsif images from dual cameras from multiple sky positions into multiple "data sets" for input into the Tycho Tracker asteroid hunting software which uses .fit files. The only processing item remaining is to delete all those .xisf files left in the processing's wake. I found this on this forum for getting my list of .xisf files to be deleted from my processing directories but how do I delete them?

 

[CCDer]

I figured out a way to do this.
1. Create a text file "FilesToBeDeleted.txt" from the PI Java Script using the above pointed to "Array searchDirectory()" that will list all .xisf files to be deleted. One of these files will be created for each run of the script that processes all data sets for one night's imaging.
2. Create a VB Script that will look for the "FilesToBeDeleted.txt" files in the directory tree where every night's images are copied (disposable copies, not archived).
3. The VB Script runs in background on the computer, waking at a defined interval to scan the directory tree for the "FilesToBeDelete.txt" files. Once found, this script deletes all files in the "FilesToBeDeleted.txt" file as well as that file. Note that the .xisf files take up 9X the space of the final .fit files used with Tycho amounting to about 200GB of disk space each night for only 5 data sets. I've alreay written this VBScript and it does the job.

Part two of this problem is that I needed a way to start additional processing, the Tycho Tracker, after images are preprocessed. So the same technique will have to suffice. I'll create the .bat files for Tycho from the original PI Java Script and on completion of preprocessing also create a "Flag" flag to trigger another VBScript to execute the batch files for Tycho.

I think this is a bit crazy to resort to but given no other means . . .

 

[Juan Conejero]

Sorry for the late reply. I am extremely busy working on the next 1.8.8-6 version of PixInsight.

You don't need VB at all if you have PixInsight. Here is a routine for secure file removal, borrowed from our code maintenance scripts:

// ----------------------------------------------------------------------------
// PixInsight JavaScript Runtime API - PJSR Version 1.0
// ----------------------------------------------------------------------------
// Copyright (c) 2003-2024 Pleiades Astrophoto S.L.
//
// Redistribution and use in both source and binary forms, with or without
// modification, is permitted provided that the following conditions are met:
//
// 1. All redistributions of source code must retain the above copyright
//    notice, this list of conditions and the following disclaimer.
//
// 2. All redistributions in binary form must reproduce the above copyright
//    notice, this list of conditions and the following disclaimer in the
//    documentation and/or other materials provided with the distribution.
//
// 3. Neither the names "PixInsight" and "Pleiades Astrophoto", nor the names
//    of their contributors, may be used to endorse or promote products derived
//    from this software without specific prior written permission. For written
//    permission, please contact info@pixinsight.com.
//
// 4. All products derived from this software, in any form whatsoever, must
//    reproduce the following acknowledgment in the end-user documentation
//    and/or other materials provided with the product:
//
//    "This product is based on software from the PixInsight project, developed
//    by Pleiades Astrophoto and its contributors (http://pixinsight.com/)."
//
//    Alternatively, if that is where third-party acknowledgments normally
//    appear, this acknowledgment must be reproduced in the product itself.
//
// THIS SOFTWARE IS PROVIDED BY PLEIADES ASTROPHOTO AND ITS CONTRIBUTORS
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
// TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
// PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL PLEIADES ASTROPHOTO OR ITS
// CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
// EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, BUSINESS
// INTERRUPTION; PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; AND LOSS OF USE,
// DATA OR PROFITS) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
// POSSIBILITY OF SUCH DAMAGE.
// ----------------------------------------------------------------------------

/*!
* \brief Secure file removal routine.
*
* \param dirPath    The base directory where files will be removed. Must be an
*                   absolute directory path specification. Relative paths are
*                   not allowed, as well as attempts to climb up the
*                   filesystem by means of '..' directory items.
*
* \param pattern    A wildcard pattern for file selection, such as "*.txt" or
*                   "foo*.bar". This parameter is mandatory and cannot be an
*                   empty string.
*
* \param recursive  If true, existing selected files will be removed
*                   recursively on the entire directory tree rooted at the
*                   \a dirPath base directory. If false, selected files will
*                   only be deleted on the base directory. If not specified,
*                   the default value is \c false.
*
* For enforced security, this routine will not attempt to follow symbolic
* links on platforms that support them.
*
* Returns the total number of files removed.
*
* \warning Incorrect use of this routine is <strong>extremely
*          dangerous</strong>. If called with improper parameter values, this
*          function may lead to an irreversible data loss. Extreme care must
*          be taken in all cases, but very especially when files are deleted
*          recursively.
*/
function removeFiles( dirPath, pattern, recursive )
{
   function removeFiles_recursive( dirPath, pattern, recursive, baseDir )
   {
      if ( dirPath.indexOf( ".." ) >= 0 )
         throw new Error( "removeFiles(): Attempt to climb up the filesystem." );
      if ( !dirPath.startsWith( baseDir ) )
         throw new Error( "removeFiles(): Attempt to redirect outside the base directory." );
      if ( !File.directoryExists( dirPath ) )
         throw new Error( "removeFiles(): Attempt to remove a nonexistent directory." );

      let currentDir = dirPath;
      if ( !currentDir.endsWith( '/' ) )
         currentDir += '/';

      let count = 0;
      if ( recursive )
      {
         let f = new FileFind;
         if ( f.begin( currentDir + "*" ) )
            do
            {
               let itemPath = currentDir + f.name;
               if ( !f.isSymbolicLink )
                  if ( f.isDirectory )
                     if ( f.name != "." && f.name != ".." )
                        count += removeFiles_recursive( itemPath, pattern, true/*recursive*/, baseDir );
            }
            while ( f.next() );
      }

      let f = new FileFind;
      if ( f.begin( currentDir + pattern ) )
         do
         {
            let itemPath = currentDir + f.name;
            if ( !f.isSymbolicLink )
               if ( f.isFile )
               {
                  File.remove( itemPath );
                  ++count;
               }
         }
         while ( f.next() );

      return count;
   }

   // Mandatory and default parameters.
   if ( dirPath === undefined || dirPath.length == 0 )
      throw Error( "removeFiles(): Missing mandatory dirPath argument" );
   if ( pattern === undefined || pattern.length == 0 )
      throw Error( "removeFiles(): Missing mandatory pattern argument" );
   if ( recursive === undefined )
      recursive = false;

   // Use UNIX directory separators on all platforms.
   if ( CoreApplication.platform == "Windows" )
      dirPath = File.windowsPathToUnix( dirPath );

   // Validate base directory.
   if ( (CoreApplication.platform == "Windows") ?
            dirPath.indexOf( ':' ) != 1 || dirPath.indexOf( '/' ) != 2 :
            dirPath.indexOf( '/' ) != 0 )
      throw new Error( "removeFiles(): Relative directory." );
   if ( !File.directoryExists( dirPath ) )
      throw new Error( "removeFiles(): Nonexistent directory." );

   // Remove files.
   return removeFiles_recursive( dirPath, pattern, recursive, dirPath );
}

Any code fragment that can delete files is potentially dangerous and can lead to irreversible data losses. The code documentation already says this, but I have to insist: If misused, this routine can be extremely dangerous. Please be extremely careful and test this function extensively before using it for production purposes. This routine is already working in mission critical contexts and I post it here in the hope that it will be useful, but I DO NOT GUARANTEE ITS PERFORMANCE OR CORRECTNESS FOR ANY PURPOSE. I rarely publish licensed code directly on this forum, but considering potential issues, the routine above is licensed under our open-source PCL License.

Updated to fix the problem detected below as of 2024 July 11.

 

[CCDer]

Thank you for the info Juan, that's very useful!

It turns out that for "Part 2" above, using a separate VB Script to start Tycho has the added benefit of having several PC's monitor the shared PI drive for .bat files to appear and then "allocate" it for that PC to process. So I can have one PC complete the preprocessing in JavaScript, also creating the .bat file for each data set as that data set's preprocessing completes and also have 3 other PCs perform the Tycho processing in batch mode as those .bat files appear on the shared PI drive. Best use all around, I think. So no need for JavaScript to start anything else. The .bat files appearing do that best.

Thanks again
Mark

 

[ShawnGano]

Hello @Juan Conejero - by chance is there a newer version of this script that you would be willing to share? I noticed that when run on Windows that CoreApplication.platform == "Windows" and not "MSWindows" which is called out in the script above, which causes the script to throw errors when validating the path. Changing those instances would be easy but I wasn't sure if there were other things that also need changing. Thank you!

 

[Juan Conejero]

Hello. The script above is perfectly valid for current PixInsight versions. I have already edited it to replace "MSWindows" with "Windows". This change was introduced in version 1.8.9 to improve code readability.

 

[ShawnGano]

Thanks! I am having some difficulties using this script; specifically the recursive option. For example say I want to delete all files with "_n.jpg" at the end in all sub directories of some base directory. Something like:

var numFilesDeleted = removeFiles( baseDirectory , "*_n.jpg", true);

This seems to only delete the files matching the pattern at the top level directory. Looking at the script - in the recursive section where f is a FileFind object, it only returns directories that also match that pattern. Specifically the line is:

if ( f.begin( currentDir + pattern ) )

Since the '+ pattern is included in the search all sub directories aren't searched just the ones that also have the pattern (at least that is how it is working on my windows machine). In testing if I named a sub-directory "sub_test_n.jpg" it would then search this directory and remove the proper files from it but not another sub-directory called "sub_test".

Is this what was intended? I was expecting it to search all sub-directories no matter what those sub-directories were named.

 

[ShawnGano]

Sorry to bother you one more time @Juan Conejero - I have tested this script multiple times now on Windows and MacOS and with various patterns but it doesn't seem to recursively delete files from sub-directories (unless that sub directories name also matches the pattern). Meaning a call like:

var numFilesDeleted = removeFiles( baseDirectory , "*_n.jpg", true);

will only delete the files ending in '_n.jpg' in the base directory and not sub-directories (unless those sub directories have names that end in '_n.jpg' too).

Am I using this function incorrectly? Or is that the intended behavior and my assumption incorrect that it should search all sub-directories, no matter what they are named.

Thank you for the help.

 

[Juan Conejero]

True. I have updated the code in my original answer to fix the problem you have reported. Let me know if the routine works now as expected.

 

[ShawnGano]

True. I have updated the code in my original answer to fix the problem you have reported. Let me know if the routine works now as expected.

It now works as expected. Thank you! I tested it again on both MacOS and Windows along with a few different cases and found it didn't delete any unexpected files and it also traversed the entire sub-directory tree and deleted all the files matching the pattern.