Need to perform file cleanup in JavaScript

[CCDer]

I've finished modifying the supplied PixInsight Batch Preprocessing script to do a single batch calibrate, register, and integrate of .xsif images from dual cameras from multiple sky positions into multiple "data sets" for input into the Tycho Tracker asteroid hunting software which uses .fit files. The only processing item remaining is to delete all those .xisf files left in the processing's wake. I found this on this forum for getting my list of .xisf files to be deleted from my processing directories but how do I delete them?

 

[CCDer]

I figured out a way to do this.
1. Create a text file "FilesToBeDeleted.txt" from the PI Java Script using the above pointed to "Array searchDirectory()" that will list all .xisf files to be deleted. One of these files will be created for each run of the script that processes all data sets for one night's imaging.
2. Create a VB Script that will look for the "FilesToBeDeleted.txt" files in the directory tree where every night's images are copied (disposable copies, not archived).
3. The VB Script runs in background on the computer, waking at a defined interval to scan the directory tree for the "FilesToBeDelete.txt" files. Once found, this script deletes all files in the "FilesToBeDeleted.txt" file as well as that file. Note that the .xisf files take up 9X the space of the final .fit files used with Tycho amounting to about 200GB of disk space each night for only 5 data sets. I've alreay written this VBScript and it does the job.

Part two of this problem is that I needed a way to start additional processing, the Tycho Tracker, after images are preprocessed. So the same technique will have to suffice. I'll create the .bat files for Tycho from the original PI Java Script and on completion of preprocessing also create a "Flag" flag to trigger another VBScript to execute the batch files for Tycho.

I think this is a bit crazy to resort to but given no other means . . .

 

[Juan Conejero]

Sorry for the late reply. I am extremely busy working on the next 1.8.8-6 version of PixInsight.

You don't need VB at all if you have PixInsight. Here is a routine for secure file removal, borrowed from our code maintenance scripts:

// ----------------------------------------------------------------------------
// PixInsight JavaScript Runtime API - PJSR Version 1.0
// ----------------------------------------------------------------------------
// Copyright (c) 2003-2020 Pleiades Astrophoto S.L.
//
// Redistribution and use in both source and binary forms, with or without
// modification, is permitted provided that the following conditions are met:
//
// 1. All redistributions of source code must retain the above copyright
//    notice, this list of conditions and the following disclaimer.
//
// 2. All redistributions in binary form must reproduce the above copyright
//    notice, this list of conditions and the following disclaimer in the
//    documentation and/or other materials provided with the distribution.
//
// 3. Neither the names "PixInsight" and "Pleiades Astrophoto", nor the names
//    of their contributors, may be used to endorse or promote products derived
//    from this software without specific prior written permission. For written
//    permission, please contact info@pixinsight.com.
//
// 4. All products derived from this software, in any form whatsoever, must
//    reproduce the following acknowledgment in the end-user documentation
//    and/or other materials provided with the product:
//
//    "This product is based on software from the PixInsight project, developed
//    by Pleiades Astrophoto and its contributors (http://pixinsight.com/)."
//
//    Alternatively, if that is where third-party acknowledgments normally
//    appear, this acknowledgment must be reproduced in the product itself.
//
// THIS SOFTWARE IS PROVIDED BY PLEIADES ASTROPHOTO AND ITS CONTRIBUTORS
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
// TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
// PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL PLEIADES ASTROPHOTO OR ITS
// CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
// EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, BUSINESS
// INTERRUPTION; PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; AND LOSS OF USE,
// DATA OR PROFITS) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
// POSSIBILITY OF SUCH DAMAGE.
// ----------------------------------------------------------------------------

/*!
* \brief Secure file removal routine.
*
* \param dirPath    The base directory where files will be removed. Must be an
*                   absolute directory path specification. Relative paths are
*                   not allowed, as well as attempts to climb up the
*                   filesystem by means of '..' directory items.
*
* \param pattern    A wildcard pattern for file selection, such as "*.txt" or
*                   "foo*.bar". This parameter is mandatory and cannot be an
*                   empty string.
*
* \param recursive  If true, existing selected files will be removed
*                   recursively on the entire directory tree rooted at the
*                   \a dirPath base directory. If false, selected files will
*                   only be deleted on the base directory. If not specified,
*                   the default value is \c false.
*
* For enforced security, this routine will not attempt to follow symbolic
* links on platforms that support them.
*
* Returns the total number of files removed.
*
* \warning Incorrect use of this routine is <strong>extremely
*          dangerous</strong>. If called with improper parameter values, this
*          function may lead to an irreversible data loss. Extreme care must
*          be taken in all cases, but very especially when files are deleted
*          recursively.
*/
function removeFiles( dirPath, pattern, recursive )
{
   function removeFiles_recursive( dirPath, pattern, recursive, baseDir )
   {
      if ( dirPath.indexOf( ".." ) >= 0 )
         throw new Error( "removeFiles(): Attempt to climb up the filesystem." );
      if ( !dirPath.startsWith( baseDir ) )
         throw new Error( "removeFiles(): Attempt to redirect outside the base directory." );
      if ( !File.directoryExists( dirPath ) )
         throw new Error( "removeFiles(): Attempt to remove a nonexistent directory." );

      let currentDir = dirPath;
      if ( !currentDir.endsWith( '/' ) )
         currentDir += '/';

      let count = 0;
      let f = new FileFind;
      if ( f.begin( currentDir + pattern ) )
         do
         {
            let itemPath = currentDir + f.name;
            if ( !f.isSymbolicLink )
               if ( f.isFile )
               {
                  File.remove( itemPath );
                  ++count;
               }
               else if ( f.isDirectory )
               {
                  if ( recursive )
                     if ( f.name != "." && f.name != ".." )
                        count += removeFiles_recursive( itemPath, pattern, true, baseDir );
               }
         }
         while ( f.next() );

      return count;
   }

   // Mandatory and default parameters.
   if ( dirPath === undefined || dirPath.length == 0 )
      throw Error( "removeFiles(): Missing mandatory dirPath argument" );
   if ( pattern === undefined || pattern.length == 0 )
      throw Error( "removeFiles(): Missing mandatory pattern argument" );
   if ( recursive === undefined )
      recursive = false;

   // Use UNIX directory separators on all platforms.
   if ( CoreApplication.platform == "MSWindows" )
      dirPath = File.windowsPathToUnix( dirPath );

   // Validate base directory.
   if ( (CoreApplication.platform == "MSWindows") ?
            dirPath.indexOf( ':' ) != 1 || dirPath.indexOf( '/' ) != 2 :
            dirPath.indexOf( '/' ) != 0 )
      throw new Error( "removeFiles(): Relative directory." );
   if ( !File.directoryExists( dirPath ) )
      throw new Error( "removeFiles(): Nonexistent directory." );

   // Remove files.
   return removeFiles_recursive( dirPath, pattern, recursive, dirPath );
}

Any code fragment that can delete files is potentially dangerous and can lead to irreversible data losses. The code documentation already says this, but I have to insist: If misused, this routine can be extremely dangerous. Please be extremely careful and test this function extensively before using it for production purposes. This routine is already working in mission critical contexts and I post it here in the hope that it will be useful, but I DO NOT GUARANTEE ITS PERFORMANCE OR CORRECTNESS FOR ANY PURPOSE. I rarely publish licensed code directly on this forum, but considering potential issues, the routine above is licensed under our open-source PCL License.

 

[CCDer]

Thank you for the info Juan, that's very useful!

It turns out that for "Part 2" above, using a separate VB Script to start Tycho has the added benefit of having several PC's monitor the shared PI drive for .bat files to appear and then "allocate" it for that PC to process. So I can have one PC complete the preprocessing in JavaScript, also creating the .bat file for each data set as that data set's preprocessing completes and also have 3 other PCs perform the Tycho processing in batch mode as those .bat files appear on the shared PI drive. Best use all around, I think. So no need for JavaScript to start anything else. The .bat files appearing do that best.

Thanks again
Mark