PixInsight 1.6.1 - Some notes on current PixInsight development

[Juan Conejero]

Hi there!

I've been somewhat absent from the forum for a few weeks. However, this doesn't mean that I've stopped anything —it is just the opposite.

Along with the usual bug fixes and updated modules (ImageRegistration and ImageCalibration will see important improvements), these are some important new features that will be included in PixInsight 1.6.1. For each item, I have written in parentheses the current approximate percentage of completion:


New automatic updates system (40%)

Starting from version 1.6.1, you normally won't need to download and install a whole new version of Pixinsight anymore, unless you want to install it on a different machine or hard disk. The new updates system will allow you to download and install updated packages for the different components of Pixinsight on your platform, in an automatic and transparent way.

This is an important milestone in PixInsight development. It means that from now on, any bug fix, updated or new module, etc., will be available immediately, as soon as we write it, to all users on all platforms. Right now we are completely stuck because I need to generate and pack a complete new version in order to fix all bugs that have been discovered, and to include all new modules, scripts, etc. In some cases I have published updated modules and patches, but this obviously isn't the right solution to the problem. The new updates system is the perfect solution, and will make PixInsight a much more dynamic and quickly evolving platform, improving the user's experience considerably.


New module installation system (100%)

The entire installation system has been redesigned and rewritten from scratch in the PCL and the core application. The new module installation model leads to a huge benefit for users and developers: Starting from version 1.6.1, existing PixInsight modules are guaranteed to continue working with all future releases, without requiring a recompilation against new PCL versions. As added benefits, module installation is now much faster, about a 30% faster than in 1.6.0, and much more secure (much more difficult to hack).

Extended information for developers
(normal users don't need to read this)

My initial intention was to turn the PCL library into a shared object (.so, .dylib or .dll). In this way new versions of PixInsight would just have to change the PCL object, and all existing modules would continue working. This would work because PCL performs all the low-level communication with Core, so as long as PCL's high-level ABI doesn't change, significant changes in the low-level layer are invisible to modules, so everything works across successive PCL versions.

Unfortunately, PCL cannot be a shared object in a portable way. Notably on Linux, FreeBSD and Mac OS X, several shared objects loaded dynamically by the same application cannot be clients of the same shared object X. When this happens, there exists a unique copy of X's static data in memory, which is shared among all clients, and hence can be modified asynchronously and produce all kinds of inconsistencies. This is a severe restriction because PCL uses static data internally. There are ways to avoid this problem when shared objects are loaded, but at the cost of limiting the number of installable PixInsight modules (less than 10 modules). There are other possible solutions, such as dividing PCL into several sub-modules to isolate all the static data, but this is very complicated and leads to an "ugly" solution.

My solution has been clean and relatively simple: keep PCL as a static library, and write a dynamic symbol resolver for PixInsight. Basically, PixInsight replaces the default object loader from the operating system (e.g. ld on Linux) with its own, custom loader. In this way the interface between a module and the PixInsight Core application does no longer depend on a complex low-level ABI, but just on a unique C function call that resolves the addresses of all low-level routines required by the module. It turns out that this works, and works remarkably well.


First Cocoa-based, 64-bit version of PixInsight for Mac OS X (100%)

See this thread. The new version is still experimental and has some glitches and issues, but is fully stable.


Improved PCL Module Development (100%)
(Information for developers only)

Starting from version 1.6.1, PixInsight modules don't depend on any external objects. So far most modules have been dependent on custom builds of the Little CMS, CFITSIO, LibTIFF and JPEGLib libraries. This no longer happens: all PixInsight modules can now be compiled and linked just against the PCL static library.

For this to be possible, all dependencies on LCMS (color management engine) have been removed from PCL and implemented statically in the Core application. In this way the entire color management functionality is now available to modules through the low-level C API, which does not require linkage against additional libraries. Note that the high-level C++ API (ICCProfile and ICCProfileTransformation classes) has not changed.

The built-in support for FITS, TIFF and JPEG has also been removed from the PCL. This means that the old FITS, TIFF and JPEG PCL classes no longer exist, and their functionality (and more) is now available exclusively through the intermodule communication mechanism provided by the FileFormatInstance class. For example, to read or write a FITS file, a module has to instantiate FileFormatInstance to access the standard FITS module (assuming it is installed) in a secure way. Many standard modules already do that, such as ImageCalibration, ImageRegistration and ImageIntegration.


Integration of the TraceMonkey JavaScript engine version 1.9 (30%)

My intention is to upgrade the current JavaScript engine in PixInsight (SpiderMonkey version 1.8.0 RC1) to the latest stable version (TraceMonkey 1.9). This is basically the same version of JavaScript that is being included in Mozilla Firefox 3.6. The best part of it is that the new engine includes a just-in-time compiler (JIT) that converts JavaScript code into machine code on the fly. This should lead to an important performance improvement.

Several building and portability issues are preventing me from completing this task. Mozilla's build system is awkward and overly complex (IMO), and very difficult to be adapted to the current PixInsight building scheme on all platforms. I'm working on it, but the fact that Mozilla does not release a new version of its JS engine officially as a standalone product makes everything really painful for embedders.

This task, however, will lead just to a transitional state because my real intention is to drop Mozilla's JavaScript engine in favor of QtScript. For more information about the current state and performance of QtScript, see this entry on Qt development blogs.

With QtScript we'll have Apple's SquirrelFish Extreme as the underlying JavaScript engine in PixInsight, with the added benefit of a total integration with Qt, and hence with PixInsight's GUI. Among other nice things this means that we'll have, along with a large performance improvement, a JavaScript debugger integrated with the current Script Editor window.


New PJSR functionality (75%)

The new Workspace and Icon core JavaScript objects will provide access to PixInsight workspaces from scripts. A script will be able to organize, move, resize and change the stacking order of image windows and icons, create, rename and delete workspaces, etc.

The new XMLReader and XMLWriter objects will add XML parsing and generation capabilities to PJSR.


Things being developed for version 1.6.x

More active development is taking place on several key features, which will be integrated in subsequent versions, during the 1.6 cycle:

A new, XML-based open image file format (10%)

The new format is the next step toward completion of a project functionality in PixInsight. This format will be able to store multiple images with their complete processing histories (as embedded XPSM sections), history states (undo/redo images), image metadata, ICC color profiles, color space definitions, arbitrary extensions, etc. Images will be stored in all the data types supported by PixInsight (8, 16, 32 and 64-bit signed and unsigned integers, 32 and 64-bit real and complex floating point) and there will be no ambiguous elements. For example, the black/white points will always be unambiguously defined.

The new format will be completely open. There will be a publicly available formal description and open-source software tools and reference implementation libraries that will not depend on PixInsight.

A new integrated documentation system (20%)

This new system will allow you to browse documentation for modules and the core application from a dedicated graphical interface (an adaptation of the Qt Assistant application).

New port of PixInsight for FreeBSD (70%)

I really love FreeBSD. The first version will only be available for x86_64 (64-bit). It will be a Tier-1 implementation, fully stable and with all the features that PixInsight has currently on Linux/X11 (the reference platform).


That's all for now. Thanks for reading, and stay tuned!

[Niall Saunders]

And here was me thinking that you were down in South Africa, blowing your "pixizelu", and generally having a good time

So, judging by the 'percentage complete' figures, you should be all done and ready to release v1.6.1 in a couple of weeks -a couple of non-Einsteinian, Conejero-weeks, that is

Keep up the good work Juan.

Cheers,

[Carlos Milovic]

I was going to say the same thing I though he was in South Africa cheering Spain (btw, we will kick your buts next week jajajajajaja).

I'm very happy with the new changes, specially the upgrade and the new module installation systems. As you said, is a big step forward more dynamism in the development field. I'm convinced that this is the milestone for the future growth of PixInsight as more people begin to write modules and use the platform for other purposes or image processing investigation.

[RBA]

Very cool improvements, Juan. The automatic updates system will particularly be very very useful, both for users and for developers as well!

Do I dare to ask for a deadline for the release?  The "missing process dialog boxes" is still happening to me and having to reset the configuration every time is becoming a bit of a nuisance.

I was going to say the same thing I though he was in South Africa cheering Spain (btw, we will kick your buts next week jajajajajaja).

While your team is busy "kicking butts", ours will be busy scoring goals 

[Harry page]

Hi

I understood some of that 

I am english and the english don't do football , every watched us play 

Harry

[Nocturnal]

good stuff Juan! Glad to see work is being done in the documentation and projects area. Prospects of a javascript debugger also look very promising. Why bother with TraceMonkey at all when you're going to drop that JS engine anyway?

[georg.viehoever]

Juan,

...
My solution has been clean and relatively simple: keep PCL as a static library, and write a dynamic symbol resolver for PixInsight. Basically, PixInsight replaces the default object loader from the operating system (e.g. ld on Linux) with its own, custom loader. In this way the interface between a module and the PixInsight Core application does no longer depend on a complex low-level ABI, but just on a unique C function call that resolves the addresses of all low-level routines required by the module. It turns out that this works, and works remarkably well.
...

are you sure that you want to do something like this? Quite a lot can go wrong here (especially with C++), certain tools like debuggers or profilers will not work as usual, libraries will fail to link with PCL modules, ... . I can imagine a lot of rather ugly side effects. You need to do quite a lot of tricky stuff to avoid such effects...

Keeping fingers crossed you get this right.
Georg

[Juan Conejero]

Hi Sander,

Why bother with TraceMonkey at all when you're going to drop that JS engine anyway?

Because the switch to QtScript means a huge amount of work. Basically, the entire JavaScript embedding has to be rewritten in a completely different way. This is a delicate task that will have to wait until 1.7, perhaps later.

The current JavaScript engine in PixInsight has some performance issues, compared to what is mainstream today. The necessary work to integrate a faster version of the Mozilla engine is relatively small and hence is worth the effort, given the expected performance improvements. Sometimes we have to make one step back to make two steps forward in the future

[Juan Conejero]

Hi Georg,

are you sure that you want to do something like this?

Absolutely

No, I am not changing the way PixInsight modules are loaded and dynamically linked by the operating system —I am not so fool.

What I have implemented is a mechanism that acts like a dynamic loader, but is part of the private communication protocol between modules and the core application; the operating system knows nothing about it, and the private loader does not interfere with the OS loader in any way.

Instead of linking dynamically against a shared PCL that opacifies low-level communication with the core application, the PCL is linked statically to each module and the private loader requests the addresses of all low-level subroutines directly to the core application, just when the module is loaded and initialized. We get exactly the same functionality: low-level communication between modules and the core application no longer depends on a fixed ABI that forces us to recompile all modules when the ABI changes. I am completely free to add new low-level functionality in successive versions, without breaking compatibility with existing modules.