Author Topic: PCL: Python Module  (Read 19782 times)

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
PCL: Python Module
« on: 2012 December 02 09:09:08 »
This is an alpha release of a Python http://www.python.org/download/releases/2.7.3/
language extension for PixInsight. Python is a scripting language that is hugely popular
in scientific computing and many other areas. It is well defined, modern and powerful. As such,
I hope it is a powerful extension for PixInsight.

I am releasing this module in alpha state in the hope to find collaborators that help to
advance development of this module. See section "Development" below.

Features:
---------

- Allows to run Python 2.7 scripts within PixInsight

- all provided and used software has licenses compatible with use in PixInsight.

- Python scripts can access PCL classes for creating and manipulating images

- Python with PixInsight can use many popular Python extensions such an numpy http://numpy.scipy.org/,
  scipy http://www.scipy.org/, matplotlib http://matplotlib.org/, pyephem http://rhodesmill.org/pyephem/,
  and other modules such as those from PiPy http://pypi.python.org/pypi.
  Those module can do many useful tasks such as solving equation
  systems, image processing, plotting graphs, and computing the ephemerides of celestial objects.
  They save a lot of work/lines of code. JavaScript and other scripting languages
  are not nearly as strong in this domain. See the examples for proof.
 
- PixInsight can pass parameters as sys.argv to Python

- Python output (print, error messages) goes to the PixInsight console

- Depending on local (blue rectangle/drag blue triangle) or global execution (blue circle), different methods in Python
  (execute_on()/execute_global()) are called.

- New Instance (blue triangle) also works as with normal processes.

- A Python IDE (IDLE, with context help etc) can be used to edit scripts

- The following PCL objects are bound to corresponding Python objects:
  - pcl::String/IsoString <-> str
  - pcl::Array<type> -> list where elements are the correspondingly translated Python types
  - pcl::Exception <-> Exception. A Python Exception raises a pcl::SourceCodeError
  - pcl::ImageWindow <-> ImageWindow
  - pcl::View <-> View
  - pcl::ImageVariant <-> Image
  - method names have been translated to Python naming convention, e.g. GetLuminance()->get_luminance().
    The methods that have been translated currently are only a small subset of those available.
 
 - some basic help is available for instance via help(View) in Python
 
 - Execution of a script happens as follows:
   - read in script, with sys.argv set as defined by user. This will execute code in the __Main__ module.
   - For Apply() (drag blue triangle on Image, blue rectangle), execute_on(view) is called.
   - For global apply (blue circle), execute() is called.
   - Both return True for success, or False if something goes wrong. PI responds to this as usual.
   - Errors and Exceptions are captured and forwarded to PCL.
 
 - The execution model should enable use with ImageContainer (not tested yet)

 - The execution model should (at some time) allow writing proper PixInsight processes in Python
 
This is an Alpha Release!
-------------------------

This is a very early release of this module. Problems are to be expected. At this stage, the
module should only be used by those that are bold and stress resistant (such as developers).

Known problems:

- so far only developed on Fedora 17-x64 Linux. No attempt porting it to other platforms,
  in particular Windows or MacOS, has been made. However, it should be possible to port the
  software to any PI platform: The supporting modules are available for them as well.

- The language bindings cover only a small part of the PCL interface. I implemented only what was
  needed for the example scripts, or what was trivial.

- There are known problems. In particular:

  - Image.get_luminance() does not work for unknown reasons

  - After the use of the IDLE gui, we sometimes get SIGSEGV (11). Scripts outside IDL work without problems
    though. This needs further investigation.

  - There are incompatibilities between library versions. For instance, Python must not use any library
    that uses Qt (such as PyQt), because PixInsight is using an incompatible version

- only rudimentary tests have been performed.

Use:
----

Here some explanations on how to use the current module:

- The module appears as "Python" in the module list. Selecting it opens the GUI shown in gui.JPG.

- GUI Elements:

  - IDLE button: If pressed, opens the IDLE development environment. NOTE: This still frequently crashed. Be careful!
    -If Use File is selected, it opens an editor with syntax coloring for the selected file (idleEditor.JPG)
    -If Use File is not selected, it opens in interactive mode with context help (idleInteractive.JPG)

  - Use File checkbox: If selected, the selected file contains the Python script to be executed. Otherwise, the script
    text contained in the TextBox at the bottom of the dialog is executed. File selection happens with a standard
    file selection dialog.

  - Arguments: blank separated arguments that are passed to the script sys.argv, just like command line arguments would
    be passed to a shell script.

  - Text box: Enter your script text here if Use File is not selected.

  - Apply Global (blue circle), Apply (blue rectangle) and New Instance (Blue Triangle) can be used as usual.
    See under "Features" on what happens on Apply/Apply Global.

Next posting: Some examples.

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: PCL: Python Module Examples
« Reply #1 on: 2012 December 02 09:11:10 »
Examples:
---------

The module comes with 3 examples.

1. Code in the text box. This code is written to demonstrate how Python is executed and how PCL objects can be access.
   Output is printed to the Console Window. See applyGlobal.JPG for the script and the results of an Apply Global.

2. python3DPlot.py: is a re-implementation of the 3DPlot.js script that comes with PI. Instead of 722 lines of code
   it needs just 51 lines (!), and allows to rotate and zoom the plot interactively. It uses the Python matplotlib
   to do this. See python3DPlot.JPG for a screenshot showing the result of the .js script on the right, and the
   interactive Python display on the left.
   
3. pythonEphemPlot.py: This a module that computes sunset/sunrise times plus moon/planet visibilities for a year of your choice.
   The expected darkness is plotted as a background, based on the sun position (twilight) and moon position/phase.
   The rising/setting times for each celestial object are plotted as colored lines. A legend explains which line
   has which meaning.
   
   The heavy lifting is done by the libraries pyEphem (celestial calculations) and matplotlib (graphics). The whole
   program is 422 lines of code (including help texts, command line evaluation, ...).
   
   Select the year for your plot by specifying -y <year> as an argument. Additional options allow to specify your
   location. Use option -h to see those parameters. The screenshot pythonEphemPlot.JPG shows
   the graph for 2013 in Munich.
   
   BTW: this script not only works within PI. You can also use it directly from the shell command line. Make your
   own visibilities plot for next year even if you do not want to use the PI module (yet)!

Next: Information for developers

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: PCL: Python Module Developer Information
« Reply #2 on: 2012 December 02 09:13:31 »
As said, I am releasing this alpha release to find collaborators for this module.
This section explains how to build and extend the module.

- The module sources are available on BitBucket https://bitbucket.org/georgviehoever/pixinsight,
  version controlled using Mercurial. If you want to join development of this module, you are encouraged
  to use Mercurial as well. If you do not want to do that: Feel free to send changes. (Thanks to Bitli for
  introducing me to BitBucket/Mercurial)
 
- After setting up your PCL development environment, clone the sources in src/modules/processes/Python/

- Regenerate the Makefiles/projects using the PixInsight Makefile generator settings shown in makefileGenerator.JPG,
  or edit the provided Makefiles/projects to have suitable paths. If you find an entry like "-lboost_python.lib",
  change it into "-lboost_python" (this is a bug in the current version of the MakefileGenerator)

- Try to build. You may need to install additional modules from your Linux distributions repository. For instance, on
  Fedora 17 I install "python-devel", "boost-devel" (development headers for those tools) using "Add/Remove Software"

- On Fedora 17, you also may need to install an older version of GCC (version 4.7.2 appears to be incompatible with
  the latest PCL libary). I am using gcc 4.6.3.

- For running the examples, you may also need to install pyephem, python-matplotlib, numpy and scipy.

The source is organized as follows:

- PythonInterface.cpp, PythonParameters.cpp, PythonInterface.h, PythonParameters.h, PythonModule.cpp, PythonProcess.cpp,
  PythonModule.h, PythonProcess.h contain the usual boilerplate code to define the module, gui, parameters. Nothing
  very exciting there.

- PythonInstance.cpp/PythonInstance.h contain the engine that drives the Python interpreter. It is using Boost::Python
  to control the interpreter (which in turn is a thin interface on what is usually done using the Python-CAPI defined
  in Python.h)

- The language mapping is defined in PythonBindings.cpp/PythonBindings.h. This is very the really complicated stuff happens.
  Boost::Python does a lot of work behind the scenes, such as most of the conversions and refCounting.
  In many cases, mapping methods and objects is pretty straight forward, but things can also get fairly involved.
  PythonBindings.cpp contains some pointers to literature. I would recommend the tutorial in
  http://www.boost.org/doc/libs/1_5w_0/libs/python/doc/ as a starting point.
 
The Language Mapping Philosophy
-------------------------------

Here the general ideas that I followed when implementing the language bindings. I do not claim that they are the final
wisdom, and you are welcome to comment:

- map to native Python types where possible, e.g. pcl::Array->list and pcl::String->str

- using consistent Python naming conventions, i.e. lower case for modules (pixinsight),
  CamelWrite with upper case first letter for classes, lower case letters with _ separators
  as method names.

- Use Pythons weak typing. We dont have several GenericImage<> classes in Python, just one Image.

- Do not map everything you find in PCL to Python. Many PCL methods are redundant, or are unlikely to be used in scripts

- still, stay close to PCL (not like the PJSR solution that introduces many new/simplified objects). The idea is to
  stay close to PCL so it may be possible in future to implement PI-Processes in PI without the restrictions of
  PJSR

- do not map GUI elements (yet) - use Python GUI instead. This may change in future...

 
Development: What needs to be done?
-----------------------------------

Obviously, there is still a lot to be done. Here is a list with a preliminary estimate of the difficulty
(H=high, M=medium, L=low). I am afraid most tasks need a lot of stamina...

- M: Examine and solve problem with Image.get_luminance()

- M: Investigate crashes when using IDLE

- complete Language mapping for ImageWindow, View, Image/ImageVariant.
  - M: It should be possible to create/delete instance of these from Python
  - L: Image processing/manipulation Ops on Image. Should be simple with PCL2.0, where much of this is available on ImageVariant
  - L: Transfer of whole images from Python to PCL (set_image_as_list())

- M: Implement option to call other PCL processes (PCL2.0)

- H: Design way to define PI processes in Python. I currently dont see fundamental obstacles to this.

- M: Make PCL GUI available. I am currently not sure if we should to this.

- H: Find out how to properly reset Python with Boost::Python. Boost::Python currently does not support Py_Finalize().

- H: Run Python out-of-process, sending the PCL calls as RPCs. This should resolve the issues with Py_Finalize() and
     especially avoid possible library incompatibilities between PixInsight and Python (as already seen with Qt), and
     to avoid difficulties with unloading libraries. For instance IDLE already does this (if not started with -n),
     probably for similar reasons.
     
-M: Go through the FIXMEs in the source code.

- L: Test, Tests, Tests

- ...
 
Next: Your responses  :)

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Thorsten Lockert

  • Newcomer
  • Posts: 36
    • Personal Home Page
Re: PCL: Python Module Developer Information
« Reply #3 on: 2012 December 02 09:52:19 »
Next: Your responses  :)

Uhm. Wow!

I am definitely going to try to play with this (on Mac OS X, so that could be interesting -- and maybe result in at least patches to make that work?).

Thorsten

Offline Nocturnal

  • PixInsight Jedi Council Member
  • *******
  • Posts: 2727
    • http://www.carpephoton.com
Re: PCL: Python Module
« Reply #4 on: 2012 December 02 10:26:45 »

Terrific work!
Best,

    Sander
---
Edge HD 1100
QHY-8 for imaging, IMG0H mono for guiding, video cameras for occulations
ASI224, QHY5L-IIc
HyperStar3
WO-M110ED+FR-III/TRF-2008
Takahashi EM-400
PIxInsight, DeepSkyStacker, PHD, Nebulosity

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2172
  • Join the dark side... we have cookies
    • http://www.astrophoto.cl
Re: PCL: Python Module
« Reply #5 on: 2012 December 03 06:52:56 »
Sounds excellent! Great job! I'm really excited about this. I hope that this will open the window to many other developers, to join the PI community! :)
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline bitli

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 513
Re: PCL: Python Module
« Reply #6 on: 2012 December 03 08:07:44 »
Very interesting, thanks. Unfortunately I wont have the time to look at it before Christmas.

-- bitli

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: PCL: Python Module
« Reply #7 on: 2012 December 03 11:13:47 »
Well, what can I say? I am strongly against Spanish stereotypes, but I think that "ole, ole y ole" could be quite appropriate in this case  O0

Despite the fact that I don't like Python, I love the work you're doing. Let me know where I can help.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: PCL: Python Module
« Reply #8 on: 2012 December 03 11:20:42 »
By the way, I'm going to add automatic Python syntax highlighting to Script Editor. In PCL 2.0 you have the new CodeEditor PCL component, which allows you to integrate the whole functionality of Script Editor inside a PCL control (ProcessInterface, Dialog, etc.). Basically, you just have to replace your current TextBox with a CodeEditor and make a few minor changes to your code.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: PCL: Python Module
« Reply #9 on: 2012 December 03 13:09:42 »
By the way, I'm going to add automatic Python syntax highlighting to Script Editor....
Great! I am looking forward to PCL2.0!
Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: PCL: Python Module
« Reply #10 on: 2012 December 03 15:44:31 »
very nice! i have only done a little bit of programming in python and found the type conversion stuff to be pretty cool. however, the whole "indentation as control flow" thing kind of freaks me out.

anyway the more languages the better!

Offline bitli

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 513
Re: PCL: Python Module
« Reply #11 on: 2012 December 03 22:28:53 »
Now if somebody would like to do a Ruby interface....
-- bitli

Offline Nocturnal

  • PixInsight Jedi Council Member
  • *******
  • Posts: 2727
    • http://www.carpephoton.com
Re: PCL: Python Module
« Reply #12 on: 2012 December 04 10:34:14 »
I would prefer ruby too although I've been forced into python dev at work for the last year and a half or so.
Best,

    Sander
---
Edge HD 1100
QHY-8 for imaging, IMG0H mono for guiding, video cameras for occulations
ASI224, QHY5L-IIc
HyperStar3
WO-M110ED+FR-III/TRF-2008
Takahashi EM-400
PIxInsight, DeepSkyStacker, PHD, Nebulosity

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: PCL: Python Module
« Reply #13 on: 2012 December 05 02:16:32 »
very nice! i have only done a little bit of programming in python and found the type conversion stuff to be pretty cool. however, the whole "indentation as control flow" thing kind of freaks me out.
....

I know that many people do not like the indentation stuff in Python. From my point of view, this is a matter of getting used to it (simple rule: After ":" indent equally until your construct ends - not really difficult). Some people dont like {} either, and others discuss if {} should be on the same line as the "if" statement, and how many spaces the following code should be indented. Anyway, the Python syntax really helps to keep code readable, and it also usually supports the Python interpreter to exactly identify the line of a (syntax) error. Now compare this to C++ templates or Tcl.

Others say that Python is slow. IMHO, it depends on how you look at it. It is true that Python is not particularly fast (see for example http://shootout.alioth.debian.org/), but  it is also not particularly slow for a weakly typed scripting language. The nice thing about Python is that you get performance optimized tools & libraries for many interesting topics, for instance numpy/scipy. See http://www.scipy.org/PerformancePython for a case study.

The reason why I like Python really is not its speed, or its syntax: To me, it is just a reasoably modern language that allows to use a huge library of pre-built solutions (see for instance http://pypi.python.org/pypi). Try to do that with JavaScript or Lua.

Python basically opens a new world for PixInsight, and sooner or later it will also make PixInsight more accessible for the world.

Now, enough about the language wars from my side. Feel free to implement language bindings for any language you like. We may even be able to share some infrastructure code...

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Cheyenne

  • PixInsight Addict
  • ***
  • Posts: 146
    • Link to Picasa gallery of my astronomy photos
Re: PCL: Python Module
« Reply #14 on: 2012 December 05 07:54:01 »
This is an alpha release of a Python http://www.python.org/download/releases/2.7.3/
language extension for PixInsight. Python is a scripting language that is hugely popular
in scientific computing and many other areas. It is well defined, modern and powerful. As such,
I hope it is a powerful extension for PixInsight.

I am releasing this module in alpha state in the hope to find collaborators that help to
advance development of this module. See section "Development" below.


You Sir are my hero..

I've been lacking time to do exactly what you have done

Thanks for the effort :)
Cheyenne Wills
Takahashi 130 TOA
Losmandy G11
SBIG STF8300M
Canon 20Da
SBIG ST-i + openPHD for autoguiding