Saving in CSV

[bianry]

I am working on a script for fixing dark columns/rows. It saves the bad columns to a file. The file is binary.

Code: [Select]

f.createForWriting(fileName);
            f.write(engine.columns.length,DataType_UInt16);
            console.writeln("Number of columns saved: " + engine.columns.length);
            if(engine.columns.length != 0)
               f.write(engine.columns,DataType_Int32);

            f.write(engine.rows.length,DataType_UInt16);
            console.writeln("Number of rows saved: " + engine.rows.length);
            if(engine.rows.length != 0)
               f.write(engine.rows,DataType_Int32);

            f.write(engine.iterations,DataType_UInt16);
            console.writeln("Number of iterations saved: " + engine.iterations);
            f.close();How can I save the file in text format.CSV, json or XML. Would prefer to be able to edit the files outside my script.

regards

Mats

[bitli]

Look at the object JSON (Object Browser, I thnik), it is quite convenient and generally available in Javascript.  I use it to save some data in the string on one of my scripts.
I think it is in VaryParams, just search in the sources (directory src/scripts I think in the PixInsight folder) for an example.

[EDITED In fact it is in FitsFileManager/FITSFileManager-parameters.jsh
https://bitbucket.org/bitli/fitsfilemanager/src/1dadc8434f48798f8b329918667adfd8e11c1a34/main/js/FITSFileManager-parameters.jsh?at=default

but the usage is trivial.

-- bitli

[Juan Conejero]

Hi Mats,

Assuming a simple CSV encoding:

columns,<ncols>
<col1>, ..., <coln>
rows,<nrows>
<row1>, ..., <rown>

the following object can be useful to implement CSV file read/write operations (warning: code not tested):

Code: [Select]

#include <pjsr/DataType.jsh>

function LineRepairCSVFile( filePath, columns, rows )
{
   this.filePath = filePath;

   if ( columns == undefined )
      this.columns = new Array;
   else
      this.columns = columns;

   if ( rows == undefined )
      this.rows = new Array;
   else
      this.rows = rows;

   /*
    * Writes column and row coordinates to this CSV file.
    */
   this.write = function()
   {
      function outputCSVSequence( file, items, title )
      {
         file.outTextLn( format( "%s,%d", title, items.length ) );
         if ( items.length > 0 )
         {
            for ( var i = 0; ; )
            {
               file.outText( format( "%d", items[i] ) );
               if ( ++i == items.length )
                  break;
               file.outText( ',' );
            }
            file.outTextLn( '' );
         }
      }

      var file = new File;
      file.createForWriting( this.filePath );
      outputCSVSequence( file, this.columns, "columns" );
      outputCSVSequence( file, this.rows, "rows" );
      file.close();
   };

   /*
    * Reads column and row coordinates from this CSV file.
    */
   this.read = function()
   {
      function inputCSVSequence( lines, title )
      {
         var items = new Array;
         for ( var i = 0; i < lines.length; ++i )
            if ( lines[i].indexOf( title ) == 0 )
            {
               var tokens = lines[i].split( ',' );
               if ( tokens.length > 0 )
               {
                  var n = parseInt( tokens[1] );
                  if ( n > 0 )
                  {
                     var tokens = lines[i+1].split( ',' );
                     if ( tokens.length >= n )
                        for ( var i = 0; i < n; ++i )
                           items.push( parseInt( tokens[i] ) );
                  }
               }
            }
         return items;
      }

      function getTextLines( buffer )
      {
         var lines = new Array;
         for ( var bol = 0; ; )
         {
            var eol = buffer.linearSearch( 0x0A, bol ); // LF
            if ( eol < 0 )
               eol = buffer.length;
            lines.push( buffer.utf8ToString( bol, eol-bol ).trim() );
            bol = eol + 1;
            if ( bol > buffer.upperBound )
               break;
         }
         return lines;
      }

      var file = new File;
      file.openForReading( this.filePath );
      var buffer = file.read( DataType_ByteArray, file.size );
      file.close();
      var lines = getTextLines( buffer );
      this.columns = inputCSVSequence( lines, "columns" );
      this.rows = inputCSVSequence( lines, "rows" );
   };
}

Examples:

Load row and column coordinates from an already generated CSV file:

Code: [Select]

var csv = new LineRepairCSVFile( "/path/to/file.csv" );
csv.read();
var columns = csv.columns;
var rows = csv.rows;
Create a new CSV file with existing line repair coordinates:

Code: [Select]

var csv = new LineRepairCSVFile( "/path/to/file.csv", columns, rows );
csv.write();
Hope this helps.

[bianry]

Thanks Juan.
I will look in to it.

regards

Mats

[bianry]

It works quite OK.
The read function is excellent but the write function suffers from some kind of encoding trouble.
The text looks like

Code: [Select]

clms°åid°å,2
1,4
rw,2
4,8
Clearly a decoding issue. I tried to read the SpiderMonkey API but could not find any function like 'outTextLn'. Have you built your own API on top of SpiderMonkey?

regards

mats

[Juan Conejero]

Hi Mats,

The script works without problems on my machine. The generated CSV file is encoded as UTF-8. Note that this is actually identical to plain 8-bit ASCII in the case of this script. What application are you using to check the CSV file? Try loading it with PixInsight's script editor.

Have you built your own API on top of SpiderMonkey?

Of course. It is called PixInsight JavaScript Runtime (PJSR). PixInsight embeds a JavaScript engine (SpiderMonkey) just as any web browser does. The main difference is that PJSR does not have any DOM-related objects because they have nothing to do with image processing. Instead of a DOM, PJSR defines a POM (PixInsight Object Model) that includes lots of image processing resources and objects that reflect the internals of the PixInsight platform.

[bianry]

I use a windows 7 (64bit)
Even when I open it in PI editor it looks like attached picture.

If I save with code

Code: [Select]


var csv = new LineRepairCSVFile( "c:/temp/file.csv", engine.columns, engine.rows );
      csv.write();
and then try to read with

Code: [Select]


        var csv = new LineRepairCSVFile( "c:/temp/file.csv");
        csv.read();
        var columns = csv.columns;
        console.writeln("CSV columns: " + columns);
it does not read anything.

If I open the file in Notepad++ and corrects the the text to

Code: [Select]

columns,2
1,4
rows,2
4,8
then it also reads it correctly.

If I read a corrected csv-file and writes it back it can not be read again.

I could try to hunt down the problem if I get a link to the API that we are using.....

regards

Mats

[Juan Conejero]

This is a bug in the format() method on Windows. I'll fix it in the next version of PixInsight.

To work around this bug, replace line #24 of the script with:

         file.outTextLn( title + format( ",%d", items.length ) );

Now the script works well on all platforms. Sorry for the inconvenience, and thank you for discovering this bug.

[Andres.Pozo]

Could this be related to this other problem?
http://pixinsight.com/forum/index.php?topic=3925

[bianry]

The lack of a debugger is having me stomped now.
I need to save the # of iterations so I added

Code: [Select]

if ( iterations == undefined )
   {
      this.iterations = new Array;
      this.iterations.push(1);
   }
   else
   {
      this.iterations = new Array;
      this.iterations.push(iterations);
   }iterations is an int

then

Code: [Select]

outputCSVSequence( file, this.iterations, "iterations" );
Everything OK so far. But when I add

Code: [Select]

this.iterations = inputCSVSequence( lines, "iterations" );the script hangs on execution and I have to kill PI

I tried debugging with console.writeln but it hangs before anything is written. If I change  this.iterations = inputCSVSequence( lines, "iterations" ) to this.iterations = inputCSVSequence( lines, "rows" ) it does not hang.

I can not see what could cause this.
Whole function below if someone cares to take a look.

Code: [Select]

function LineRepairCSVFile( filePath, columns, rows, iterations )
{
   this.filePath = filePath;

   if ( columns == undefined )
      this.columns = new Array;
   else
      this.columns = columns;

   if ( rows == undefined )
      this.rows = new Array;
   else
      this.rows = rows;

   if ( iterations == undefined )
   {
      this.iterations = new Array;
      this.iterations.push(1);
   }
   else
   {
      this.iterations = new Array;
      this.iterations.push(iterations);
   }

//console.writeln("Iter: " + this.iterations);
   /*
    * Writes column and row coordinates to this CSV file.
    */
   this.write = function()
   {
      function outputCSVSequence( file, items, title )
      {
        // file.outTextLn( format( "%s,%d", title, items.length ) );
         file.outTextLn( title + format( ",%d", items.length ) );
         if ( items.length > 0 )
         {
            for ( var i = 0; ; )
            {
               file.outText( format( "%d", items[i] ) );
               if ( ++i == items.length )
                  break;
               file.outText( ',' );
            }
            file.outTextLn( '' );
         }
      }

      var file = new File;
      file.createForWriting( this.filePath );
      outputCSVSequence( file, this.columns, "columns" );
      outputCSVSequence( file, this.rows, "rows" );
      outputCSVSequence( file, this.iterations, "iterations" );
      file.close();
   };

   /*
    * Reads column and row coordinates from this CSV file.
    */
   this.read = function()
   {
      function inputCSVSequence( lines, title )
      {
         var items = new Array
         for ( var i = 0; i < lines.length; ++i )
            if ( lines[i].indexOf( title ) == 0 )
            {
               var tokens = lines[i].split( ',' );
               console.writeln(tokens);
               if ( tokens.length > 0 )
               {
                  var n = parseInt( tokens[1] );
                  if ( n > 0 )
                  {
                     var tokens = lines[i+1].split( ',' );
                     if ( tokens.length >= n )
                        for ( var i = 0; i < n; ++i )
                           items.push( parseInt( tokens[i] ) );
                  }
               }
            }
         return items;
      }

      function getTextLines( buffer )
      {
         var lines = new Array;
         for ( var bol = 0; ; )
         {
            var eol = buffer.linearSearch( 0x0A, bol ); // LF
            if ( eol < 0 )
               eol = buffer.length;
            lines.push( buffer.utf8ToString( bol, eol-bol ).trim() );
            bol = eol + 1;
            if ( bol > buffer.upperBound )
               break;
         }
         return lines;
      }

      var file = new File;
      file.openForReading( this.filePath );
      var buffer = file.read( DataType_ByteArray, file.size );
      file.close();
      var lines = getTextLines( buffer );
      console.writeln(lines);
      this.columns = inputCSVSequence( lines, "columns" );
      this.rows = inputCSVSequence( lines, "rows" );
      this.iterations = inputCSVSequence( lines, "iterations" );
   };
}
regards

Mats

[bitli]

Hi,
There is plenty of bizarre stuff in your code, although I am not able to understand it fully. For example the section:

Code: [Select]

if ( iterations == undefined )
   {
      this.iterations = new Array;
      this.iterations.push(1);
   }
   else
   {
      this.iterations = new Array;
      this.iterations.push(iterations);
   }
will create the array in both branch, and may either push '1' or a single value.  Then why is it an array ? Did you not intend to push something on it ?

You also use 'this' in a few places where there is no object in sight.  This tends to create unexpected global variables.

IMHO you should first try to simplify your code, make it work with global functions easier to debug (use a prefix to avoid name clash), test the logic independently of any PI functionality.
PixInsight is VERY sensitive to any error in parameters passed to an intenral objects. The internal objects are implemented in C++ and are not much protected when accessed from Javascript.  You better make sure you call them with the proper objects.

You may find it easier to prepare code independently of PI in a standalone version of Javascript, and port it to PI when it works well. I think I used the one from http://code.google.com/p/jslibs/wiki/JSLibs.

-- bitli

[bianry]

Hi bitli.
I agree that that part is suboptimal. I got the function from Juan in response to asking about saving values in human readable format. His code solved the problem with saving col/row values (which are arrays). I needed to also save the # of iterations and as a first approach I saved it as a array with one member.

The problem for me is that I can not understand why the code hangs if saving one more array.

regards

Mats