Author Topic: Projects and updated scripts/modules  (Read 4212 times)

Offline cs_pixinsight

  • PixInsight Addict
  • ***
  • Posts: 156
Projects and updated scripts/modules
« on: 2012 May 14 11:49:43 »
I've noticed several of my saved projects no longer work when I attempt to bring up a saved script or process via their icons.  They fail with a checksum error, which I assume means the source code checksum no longer matches so the script/process doesn't attempt to run. 

This presents a problem as I'd like to revisit projects from time to time as I usually don't work on a single project from beginning to end.  Even at the end of a project, I see value is keeping it around as my post processing skills improve  ;).  So is there a way to transfer the settings to a new process/script icon or get the existing icons to work again?

Thanks,
Craig

Offline mschuster

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1087
Re: Projects and updated scripts/modules
« Reply #1 on: 2012 May 16 16:34:26 »
Craig, for script icons try this: Right click the icon and choose launch. Select the MD5 checksum and delete it, leave the box empty. Drag the icon button to create a new script icon and try it. This will not work if the script has changed significantly, the listed parameters may not match the new version of the script. In this case you can use the parameter listing as a reference to create a new icon from scratch.
Mike

Offline cs_pixinsight

  • PixInsight Addict
  • ***
  • Posts: 156
Re: Projects and updated scripts/modules
« Reply #2 on: 2012 May 17 12:56:22 »
Mike, thanks for the suggestion.  I'll give this a try, but as you said in your reply, it may or may not work depending on the changes made to the script/module.  This takes the luster off the great project functionality  :sad: 

Perhaps, saving all the modules and scripts with the project data would be possible?  Then when the project is opened, it would use the saved versions instead of those stored in Program Files.  Just not sure how difficult this would be...

Craig


Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Projects and updated scripts/modules
« Reply #3 on: 2012 May 17 13:05:59 »
I think the problem is the MD5 script verification functionality. It just does not make sense to prohibit execution of a new script version just because a single ASCII character changed. After all, PCL modules also are not prevented from executing after a new module version was released.

I believe Juan should think about some way to enable a script writer to indicate backward compatibility, or at least allow the user to indicate that he still wants to execute the changed script (at his own risk).

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Projects and updated scripts/modules
« Reply #4 on: 2012 May 18 01:07:46 »
Quote
it may or may not work depending on the changes made to the script/module.

It must work, if the developer of the script or module knows what he or she is doing. In fact,  AFAIK this has not happened with any standard script or module so far.

We have many resources implemented to support backward compatibility. For example, there are a few standard tools that have changed significantly since the first versions we published back in 2008. In the rare cases where this happens, we always provide a compatibility version and automatic replacement of existing instances. An example is ATrousWaveletTransform. Right now you can open a .psm file created in 2008, and all ATrousWaveletTransform icons will be converted automatically into ATrousWaveletTransformV1 icons. The ATrousWaveletTransformV1 process is available under the Compatibility category. We take backward compatibility very seriously.

Quote
Perhaps, saving all the modules and scripts with the project data would be possible?  Then when the project is opened, it would use the saved versions instead of those stored in Program Files.

This would introduce serious problems. For example, the automatic update system would basically not work. Bugs would never get fixed, improved versions would not be distributed consistently, and mutual interactions between different tools and versions would be unpredictable. Imagine that each time you save a document created with your favorite text editor, the whole text editing application would be saved with the document.

Quote
I think the problem is the MD5 script verification functionality. It just does not make sense to prohibit execution of a new script version just because a single ASCII character changed.

Scripts are extremely vulnerable by nature, since the entire source code is always exposed. For example, somebody or 'something' could add a little code snippet at the beginning of a script to remove all files on the user's home directory, just to name a relatively 'innocent' possibility.

As the principal developer of the PixInsight platform, I am responsible for keeping it as secure as possible. Script checksum validation is a basic security feature to give the user an opportunity to stop a potential security risk. For example, if you know that a script has not been updated, and checksum validation fails for an existing icon, then something strange has happened and you should not trust the script. Checksum validation confers the responsibility for script execution to the user.

On the other hand, nothing prohibits or prevents execution of an existing Script instance when the script's source code has been modified. See the attached screen shot. You just have to remove the MD5 checksum on the Script tool, and no checksum validation will be performed. You can also save an icon without a checksum from the Script tool, although doing that is strongly discouraged for security reasons.

Of course, the current script security system can be improved. It will be improved in future PI versions. For example, scripts could be encrypted/decrypted automatically, or PJSR could provide resources for script authentication. However, I think the current checksum validation system is reasonably secure and does not cause too much trouble to the user.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Projects and updated scripts/modules
« Reply #5 on: 2012 May 18 02:23:32 »
....
Quote
I think the problem is the MD5 script verification functionality. It just does not make sense to prohibit execution of a new script version just because a single ASCII character changed.

Scripts are extremely vulnerable by nature, since the entire source code is always exposed. For example, somebody or 'something' could add a little code snippet at the beginning of a script to remove all files on the user's home directory, just to name a relatively 'innocent' possibility.

As the principal developer of the PixInsight platform, I am responsible for keeping it as secure as possible. Script checksum validation is a basic security feature to give the user an opportunity to stop a potential security risk. For example, if you know that a script has not been updated, and checksum validation fails for an existing icon, then something strange has happened and you should not trust the script. Checksum validation confers the responsibility for script execution to the user.
...
Juan,

I really appreciate that you are taking responsibility for security. Far too many developers do not... But I neverhtheless have two remarks:
- scripts are not special in any way. While it is probably true that scripts are easily manipulated due to their ASCII nature, it is not much more difficult to create a PCL module that does vicious things. If you know the right tools, it is just one button away...

- The problem with the current MD5 functionality is that it even prevents execution if the script has been upgraded through the update system. I think it is reasonable to assume that such scripts do not do bad things and have a certain backward compatibility. Maybe you want to add some kind of certificate to such "official" scripts, making an MD5 check for these redundant.

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Projects and updated scripts/modules
« Reply #6 on: 2012 May 18 04:22:01 »
Hi Georg,

Thank you.

Quote
- scripts are not special in any way. While it is probably true that scripts are easily manipulated due to their ASCII nature, it is not much more difficult to create a PCL module that does vicious things. If you know the right tools, it is just one button away...

The main difference is that a script can be modified very easily, and no system security mechanism can protect you in such case. However, modifying a binary (compiled) module to inject malicious code is much more difficult, and there are standard system security features to prevent this to happen (SELinux, virus analysis applications, etc.).

Quote
Maybe you want to add some kind of certificate to such "official" scripts, making an MD5 check for these redundant.

In fact, the whole PI scripting security system is a work in progress. For example, currently you can execute a script from the Script menu without checksum validation, which is insecure. For now, source code checksums only work for Script process icons.

The current system has to be improved to make it both more secure and more usable. There are several possibilities, but the most appealing one is a script validation mechanism implemented in PJSR. Basically, when an official script is released it should include a manifest file, so source code checksums would occur in a completely transparent way. This requires also new core functionality to regenerate all script manifests automatically, in case a standard .jsh file is updated. This also involves changes to the update system. All of this should be implemented during the 1.8 cycle. All ideas are welcome, as always.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline cs_pixinsight

  • PixInsight Addict
  • ***
  • Posts: 156
Re: Projects and updated scripts/modules
« Reply #7 on: 2012 May 18 12:28:10 »
Quote
it may or may not work depending on the changes made to the script/module.

It must work, if the developer of the script or module knows what he or she is doing. In fact,  AFAIK this has not happened with any standard script or module so far.

We have many resources implemented to support backward compatibility. For example, there are a few standard tools that have changed significantly since the first versions we published back in 2008. In the rare cases where this happens, we always provide a compatibility version and automatic replacement of existing instances. An example is ATrousWaveletTransform. Right now you can open a .psm file created in 2008, and all ATrousWaveletTransform icons will be converted automatically into ATrousWaveletTransformV1 icons. The ATrousWaveletTransformV1 process is available under the Compatibility category. We take backward compatibility very seriously.

Quote
Perhaps, saving all the modules and scripts with the project data would be possible?  Then when the project is opened, it would use the saved versions instead of those stored in Program Files.

This would introduce serious problems. For example, the automatic update system would basically not work. Bugs would never get fixed, improved versions would not be distributed consistently, and mutual interactions between different tools and versions would be unpredictable. Imagine that each time you save a document created with your favorite text editor, the whole text editing application would be saved with the document.



Juan, thank you for taking your time responding to my request.  I feel the project functionality sets PI far above all the other software out there.  Being able to save your entire session and come back a month later just as you left it is amazing!  I also understand your concerns out security and I'm glad you are not taking the normal cursory glance at it and saying it's good enough.

However, this great feature does have this little catch.  If any of your saved scripts are updated since the project was saved you run the risk that it can no longer be used.  The MD5 workaround helps, but as we have all stated - it may or may not work depending on the changes to the script.  My issue came about with the BatchPreprocessing (which is a standard script) and CosmeticCalibration scripts being updated, but it could have been something else. 

Based on your response it seems that modules may be a better option for backward compatibility due to checksum validation only being performed on scripts, but the author must take it upon themselves to ensure the compatibility.  My suggestion to save the old versions was really just a wild idea for backward compatibility - perhaps versioning the scripts/modules in the installation directory but only exposing the latest version in the PI menus is a better option to ensure everything still works when returning to a project.

In any case, I always enjoy reading your posts due to their sincere and information rich content.

Thanks again,
Craig