Author Topic: Annotation script  (Read 151950 times)

Offline Josh Lake

  • PixInsight Old Hand
  • ****
  • Posts: 424
Re: Annotation script
« Reply #105 on: 2012 April 04 16:01:45 »
Once again, fantastic work! This new version works great with my original HIP archive data and it took care of the duplication issue nicely.

Attached is an example of output with the custom catalog of HIP stars. Also, here is the HIP catalog file for those interested: dropbox.com/sh/v11byktj1var569/Dp-BkBbX_0/Hipparcos.txt

One thing to adjust for the future: on the Mac (or perhaps just at my resolution?), the original script window is squeezed. It is no problem to resize it, but it'd be best if it fully fit from the start. I'm not sure if that would make it look to big on the PC, though...

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Annotation script
« Reply #106 on: 2012 April 04 18:10:15 »
I've made a few improvements - version 0.65 attached:

* Fixed a couple problems with dialog dimensions (but this is a provisional workaround - we need a better method).

* Fixed a few message strings.

* A more robust method to compute the local path to the NGC/IC catalog (the script can now be stored anywhere in the local filesystem)..

To find the changes I've made in the source code, search for the string: /*###*/

The script needs a robust routine to split a text file into text lines in a platform-independent way. Tomorrow I can adapt the routine I have used in the PIDoc compiler script (can read UTF8 text files with any combination of end-of-line sequences).

Andrés, let me know if you agree with these changes.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline slang

  • Member
  • *
  • Posts: 60
Re: Annotation script
« Reply #107 on: 2012 April 04 18:25:39 »
I've made a few improvements - version 0.65 attached:

* Fixed a couple problems with dialog dimensions (but this is a provisional workaround - we need a better method).

* Fixed a few message strings.

* A more robust method to compute the local path to the NGC/IC catalog (the script can now be stored anywhere in the local filesystem)..

To find the changes I've made in the source code, search for the string: /*###*/

The script needs a robust routine to split a text file into text lines in a platform-independent way. Tomorrow I can adapt the routine I have used in the PIDoc compiler script (can read UTF8 text files with any combination of end-of-line sequences).

Andrés, let me know if you agree with these changes.

Hi Juan.

That's awesome - I'm really loving this new functionality.

I'm trying (at present) to integrate session info from APT which I use to control my DSLR. I've got access to DEC, RA, focal length, pixel size etc., and have a working method to add this metainfo to the .cr2 files produced by the Canon cameras (and APT).

At present, I note that when you import raw images, you set DATE_OBS based on the camera date setting. Given that I can create almost any other tag with the required information from the imaging software, what tag names should I set to get the information automatically into Pixinsight? Will the default tag names (FOCALLEN, XPIXSZ, YPIXSZ etc.) be imported, or are there other changes required in Pixinsight required to read/import this information?
--
Mounts: Orion Atlas 10 eq-g, Explore Scientific G11-PMC8
Scopes: GSO RC8, Astrophysics CCDT67, ES FCD100-80, TSFLAT2
Guiding: ST80/QHY OAG/QHY5L-II-M
Cameras: Canon EOS 450D (IR Mod), QHY8L, QHY163m/QHYFW2-US/Astronomik LRGBHaSiiOii

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: Annotation script
« Reply #108 on: 2012 April 04 19:35:14 »
okay, i'm not crazy then. i have been hesitant to apply the updates since someone was having a big problem with the new DSLR_RAW module and that's kind of important to me...

maybe i'll make a copy of the app and upgrade that one, just to see. thanks for the feedback.
Keep in mind that both AnnotateImage and ImageSolver use functionality introduced by Juan very recently. For using this scripts you MUST have the last version of PI with all the updates.

i'm fairly certain that i was up to date a few weeks ago when i reported the problem initially... i think the updates i'm referring to have appeared in the last couple of weeks.

Offline Andres.Pozo

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 927
Re: Annotation script
« Reply #109 on: 2012 April 05 01:30:27 »
I've made a few improvements - version 0.65 attached:

* Fixed a couple problems with dialog dimensions (but this is a provisional workaround - we need a better method).

* Fixed a few message strings.

* A more robust method to compute the local path to the NGC/IC catalog (the script can now be stored anywhere in the local filesystem)..

To find the changes I've made in the source code, search for the string: /*###*/

The script needs a robust routine to split a text file into text lines in a platform-independent way. Tomorrow I can adapt the routine I have used in the PIDoc compiler script (can read UTF8 text files with any combination of end-of-line sequences).

Andrés, let me know if you agree with these changes.

Juan, I agree with the changes.

There is a debug message that I forgot to delete in the line 887 in AstronomicalCatalogs.jsh.
Also, I am not very fluent in Javascript (as in english) and I don't know if the mechanism for generalizing the layers and catalogs (GetConstructor functions and __catalogRegister__) is the best one.


Offline Andres.Pozo

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 927
Re: Annotation script
« Reply #110 on: 2012 April 05 01:54:59 »
I have found a serious error in the layer management. The buttons delete, move up and move down modify the layers on the dialog but these changes were not applied to the engine and not used in the annotation.

The version 0.7 fixes this problem.

Offline Andres.Pozo

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 927
Re: Annotation script
« Reply #111 on: 2012 April 05 02:09:13 »
Once again, fantastic work! This new version works great with my original HIP archive data and it took care of the duplication issue nicely.

Attached is an example of output with the custom catalog of HIP stars. Also, here is the HIP catalog file for those interested: dropbox.com/sh/v11byktj1var569/Dp-BkBbX_0/Hipparcos.txt

Nice work!! This proves that the custom catalogs are useful.

However, using the last version of the script you can get the same annotation using only predefined catalogs. The trick is using the Tycho-2 catalog with a max magnitude of 8 and selecting the HIP code for the label.

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Annotation script
« Reply #112 on: 2012 April 05 02:13:04 »
Buenos días Andrés

Believe me that nobody could say that you are not very fluent with JavaScript---I know professional web programmers that would be unable to write something with half the complexity of your Annotate.js.

Quote
I don't know if the mechanism for generalizing the layers and catalogs (GetConstructor functions and __catalogRegister__) is the best one.

There is something important to note in this regard. The eval function is extremely dangerous in JavaScript, since it opens a door to inject malicious code in any script. For this reason all script parameter values stored with the Parameters PJSR object are simple strings, and type conversions (for example, from String to Number) are performed by internal C++ code to which no script has direct access. This guarantees that a script parameter cannot be used to execute any unwanted code with the Script process in PixInsight. However, if your code calls eval() for parameter strings, this is a potential security breach. I don't want to be more explicit --- suffice this to show that eval() is a very serious potential security risk.

So I'd rewrite your code to avoid eval() completely. Other than this, your script is absolutely wonderful :)
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Andres.Pozo

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 927
Re: Annotation script
« Reply #113 on: 2012 April 05 02:35:30 »
Buenos días Andrés

Believe me that nobody could say that you are not very fluent with JavaScript---I know professional web programmers that would be unable to write something with half the complexity of your Annotate.js.


I am a professional programmer with lots (too many?) years of experience, although not in Javascript.

Quote
[...]suffice this to show that eval() is a very serious potential security risk
I was aware of this, but I couldn't find a better way. Although Javascript has some nice things, it has too many rough edges. There are a lot of things that were learned in the 70's and 80's and that have been forgotten in Javascript.

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Annotation script
« Reply #114 on: 2012 April 05 03:28:36 »

Currently you are storing something like this in your engine_layers script parameter:

"new GridLayer()|" +
"new CatalogLayer(new NamedStarsCatalog())|" +
"new CatalogLayer(new NGCICCatalog())|" +
"new CatalogLayer(new TychoCatalog())|" +
"new CatalogLayer(new PGCCatalog())"


Then your script runs the following code to recreate a set of annotation layers:

Code: [Select]
      var layerConstructors = layersStr.split( "|" );
      this.layers = new Array;
      for ( var l = 0; l < layerConstructors.length; l++ )
      {
         this.layers[l] = eval( layerConstructors[l] );
         this.layers[l].SetId( l );
         this.layers[l].LoadParameters();
      }

Although this solution is indeed elegant and powerful (for example, it is not restricted to a finite and fixed set of layer and catalog objects), it in inherently insecure. As a somewhat 'dramatic' example, somebody could change one of these parameters to remove all files in the user's home directory, and your script would execute the corresponding code efficiently in the first call to eval() :)

You can replace this system with a simple parser. Many people would use XML to implement this functionality, but XML tends to generate a lot of unnecessary overhead in these cases, just to be 'fancy'. Another cleaner option would be JSON. Personally I tend to favor even simpler adhoc solutions. Suppose that instead of the above parameter we had the following stored in engine_layers:

"GRID|CATALOG:NAMED_STARS|CATALOG:NGCIC|CATALOG:TYCHO|CATALOG:PGC"

Now instead of your eval()-based code we can implement the following:

Code: [Select]
      var layerConstructors = layersStr.split( "|" );
      this.layers = new Array;
      for ( var l = 0; l < layerConstructors.length; l++ )
      {
         var constructorParameters = layerConstructors[l].split( ":" );
         if ( constructorParameters.length < 1 )
            throw new Error( "Missing layer constructor in parameter." );
         switch ( constructorParameters[0] )
         {
         case "GRID":
            if ( constructorParameters.length > 1 )
               throw new Error( "Invalid layer constructor: The GRID constructor takes no parameters." );
            this.layers[l] = new GridLayer;
            break;

         case "CATALOG":
            if ( constructorParameters.length != 2 )
               throw new Error( "Invalid layer constructor: The CATALOG constructor takes one parameter." );
            var catalog;
            switch ( constructorParameters[1] )
            {
            case "NAMED_STARS": catalog = new NamedStarsCatalog(); break;
            case "NGCIC":       catalog = new NGCICCatalog(); break;
            case "TYCHO":       catalog = new TychoCatalog(); break;
            case "PGC":         catalog = new PGCCatalog(); break;
            default:
               throw new Error( "Invalid layer constructor: Unknown catalog type." );
            }
            this.layers[l] = new CatalogLayer( catalog );
            break;

         default:
            throw new Error( "Invalid layer constructor: Unknown layer type." );
         }
     
         this.layers[l].SetId( l );
         this.layers[l].LoadParameters();
      }

with the corresponding changes to generate the parameter contents. The above code is secure, since there is no way to force it to do anything unwanted. It just generates your annotation layers with basic error checking, and cannot execute anything else. It is considerably uglier and not too elegant, but hey, enforcing security has always had a price :)
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Andres.Pozo

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 927
Re: Annotation script
« Reply #115 on: 2012 April 05 03:58:29 »
[...] The above code is secure, since there is no way to force it to do anything unwanted. It just generates your annotation layers with basic error checking, and cannot execute anything else. It is considerably uglier and not too elegant, but hey, enforcing security has always had a price :)
I was just trying to avoid the giant switch and implement a generic factory of objects, but if it is not possible we have to bear the limitations of javascript.

One possible idea would be validating the constructor text before calling eval. If the constructor is not found in the layer or catalog registry it would not be executed.

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Annotation script
« Reply #116 on: 2012 April 05 04:06:30 »
Quote
One possible idea would be validating the constructor text before calling eval. If the constructor is not found in the layer or catalog registry it would not be executed.

Yes. If you can validate the semantics of the code stored in script parameters, then calling eval() would be secure. However, I don't know if the code necessary to implement this would be even more complex than the giant switch.

Another possibility would be validating the parameter's code with a cryptographic digest, such as MD5. This would be a lot faster and very clean. But, how can we store the digest in a way that a malicious parameter modification could not simulate?
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Josh Lake

  • PixInsight Old Hand
  • ****
  • Posts: 424
Re: Annotation script
« Reply #117 on: 2012 April 05 04:23:37 »
Once again, fantastic work! This new version works great with my original HIP archive data and it took care of the duplication issue nicely.

Attached is an example of output with the custom catalog of HIP stars. Also, here is the HIP catalog file for those interested: dropbox.com/sh/v11byktj1var569/Dp-BkBbX_0/Hipparcos.txt

Nice work!! This proves that the custom catalogs are useful.

However, using the last version of the script you can get the same annotation using only predefined catalogs. The trick is using the Tycho-2 catalog with a max magnitude of 8 and selecting the HIP code for the label.

Ah ha! I hadn't explored that new keyword functionality in the newest version.

I experimented with it and am very happy with the results, but I noticed one thing: if I filter to 9th magnitude, I get stars that are marked but not labeled, presumably because they are too small. This is not the end of the world for my frames, but it might lead to clutter in bigger  or richer fields. Take a look.


Offline Andres.Pozo

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 927
Re: Annotation script
« Reply #118 on: 2012 April 05 04:31:59 »
I experimented with it and am very happy with the results, but I noticed one thing: if I filter to 9th magnitude, I get stars that are marked but not labeled, presumably because they are too small. This is not the end of the world for my frames, but it might lead to clutter in bigger  or richer fields. Take a look.
This happens because some stars have not HIP code in the TYCHO-2 catalog.

Offline Andres.Pozo

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 927
Re: Annotation script
« Reply #119 on: 2012 April 05 04:45:53 »
Yes. If you can validate the semantics of the code stored in script parameters, then calling eval() would be secure. However, I don't know if the code necessary to implement this would be even more complex than the giant switch.
I was thinking in something easier. Simply check if there is a layer in the registry which constructor is the same as the parameter value.

Another idea would be that the layer registry associates the layer name with the constructor. Instead of saving the constructor, we'll save the name. For example, the layer register could be constructed this way:
Code: [Select]
for( var c=0; c<__catalogRegister__.length; c++ )
{
   var catalog = eval(__catalogRegister__[c]);
   var layer = new CatalogLayer(catalog);
   __layerRegister__[layer.layerName] = layer.GetConstructor() );
}