Hi there!
I've been somewhat absent from the forum for a few weeks. However, this doesn't mean that I've stopped anything ?it is just the opposite.
Along with the usual bug fixes and updated modules (ImageRegistration and ImageCalibration will see important improvements), these are some important new features that will be included in PixInsight 1.6.1. For each item, I have written in parentheses the current approximate percentage of completion:
New automatic updates system (40%)
Starting from version 1.6.1, you normally won't need to download and install a whole new version of Pixinsight anymore, unless you want to install it on a different machine or hard disk. The new updates system will allow you to download and install updated packages for the different components of Pixinsight on your platform, in an automatic and transparent way.
This is an important milestone in PixInsight development. It means that from now on, any bug fix, updated or new module, etc., will be available immediately, as soon as we write it, to all users on all platforms. Right now we are completely stuck because I need to generate and pack a complete new version in order to fix all bugs that have been discovered, and to include all new modules, scripts, etc. In some cases I have published updated modules and patches, but this obviously isn't the right solution to the problem. The new updates system is the perfect solution, and will make PixInsight a much more dynamic and quickly evolving platform, improving the user's experience considerably.
New module installation system (100%)
The entire installation system has been redesigned and rewritten from scratch in the PCL and the core application. The new module installation model leads to a huge benefit for users and developers: Starting from version 1.6.1, existing PixInsight modules are guaranteed to continue working with all future releases, without requiring a recompilation against new PCL versions. As added benefits, module installation is now much faster, about a 30% faster than in 1.6.0, and much more secure (much more difficult to hack).
Extended information for developers
(normal users don't need to read this)
My initial intention was to turn the PCL library into a shared object (.so, .dylib or .dll). In this way new versions of PixInsight would just have to change the PCL object, and all existing modules would continue working. This would work because PCL performs all the low-level communication with Core, so as long as PCL's high-level ABI doesn't change, significant changes in the low-level layer are invisible to modules, so everything works across successive PCL versions.
Unfortunately, PCL cannot be a shared object in a portable way. Notably on Linux, FreeBSD and Mac OS X, several shared objects loaded dynamically by the same application cannot be clients of the same shared object X. When this happens, there exists a unique copy of X's static data in memory, which is shared among all clients, and hence can be modified asynchronously and produce all kinds of inconsistencies. This is a severe restriction because PCL uses static data internally. There are ways to avoid this problem when shared objects are loaded, but at the cost of limiting the number of installable PixInsight modules (less than 10 modules). There are other possible solutions, such as dividing PCL into several sub-modules to isolate all the static data, but this is very complicated and leads to an "ugly" solution.
My solution has been clean and relatively simple: keep PCL as a static library, and write a dynamic symbol resolver for PixInsight. Basically, PixInsight replaces the default object loader from the operating system (e.g. ld on Linux) with its own, custom loader. In this way the interface between a module and the PixInsight Core application does no longer depend on a complex low-level ABI, but just on a unique C function call that resolves the addresses of all low-level routines required by the module. It turns out that this works, and works remarkably well.
First Cocoa-based, 64-bit version of PixInsight for Mac OS X (100%)
See this thread. The new version is still experimental and has some glitches and issues, but is fully stable.
Improved PCL Module Development (100%)
(Information for developers only)
Starting from version 1.6.1, PixInsight modules don't depend on any external objects. So far most modules have been dependent on custom builds of the Little CMS, CFITSIO, LibTIFF and JPEGLib libraries. This no longer happens: all PixInsight modules can now be compiled and linked just against the PCL static library.
For this to be possible, all dependencies on LCMS (color management engine) have been removed from PCL and implemented statically in the Core application. In this way the entire color management functionality is now available to modules through the low-level C API, which does not require linkage against additional libraries. Note that the high-level C++ API (ICCProfile and ICCProfileTransformation classes) has not changed.
The built-in support for FITS, TIFF and JPEG has also been removed from the PCL. This means that the old FITS, TIFF and JPEG PCL classes no longer exist, and their functionality (and more) is now available exclusively through the intermodule communication mechanism provided by the FileFormatInstance class. For example, to read or write a FITS file, a module has to instantiate FileFormatInstance to access the standard FITS module (assuming it is installed) in a secure way. Many standard modules already do that, such as ImageCalibration, ImageRegistration and ImageIntegration.
Integration of the TraceMonkey JavaScript engine version 1.9 (30%)
My intention is to upgrade the current JavaScript engine in PixInsight (SpiderMonkey version 1.8.0 RC1) to the latest stable version (TraceMonkey 1.9). This is basically the same version of JavaScript that is being included in Mozilla Firefox 3.6. The best part of it is that the new engine includes a just-in-time compiler (JIT) that converts JavaScript code into machine code on the fly. This should lead to an important performance improvement.
Several building and portability issues are preventing me from completing this task. Mozilla's build system is awkward and overly complex (IMO), and very difficult to be adapted to the current PixInsight building scheme on all platforms. I'm working on it, but the fact that Mozilla does not release a new version of its JS engine officially as a standalone product makes everything really painful for embedders.
This task, however, will lead just to a transitional state because my real intention is to drop Mozilla's JavaScript engine in favor of QtScript. For more information about the current state and performance of QtScript, see this entry on Qt development blogs.
With QtScript we'll have Apple's SquirrelFish Extreme as the underlying JavaScript engine in PixInsight, with the added benefit of a total integration with Qt, and hence with PixInsight's GUI. Among other nice things this means that we'll have, along with a large performance improvement, a JavaScript debugger integrated with the current Script Editor window.
New PJSR functionality (75%)
The new Workspace and Icon core JavaScript objects will provide access to PixInsight workspaces from scripts. A script will be able to organize, move, resize and change the stacking order of image windows and icons, create, rename and delete workspaces, etc.
The new XMLReader and XMLWriter objects will add XML parsing and generation capabilities to PJSR.
Things being developed for version 1.6.x
More active development is taking place on several key features, which will be integrated in subsequent versions, during the 1.6 cycle:
A new, XML-based open image file format (10%)
The new format is the next step toward completion of a project functionality in PixInsight. This format will be able to store multiple images with their complete processing histories (as embedded XPSM sections), history states (undo/redo images), image metadata, ICC color profiles, color space definitions, arbitrary extensions, etc. Images will be stored in all the data types supported by PixInsight (8, 16, 32 and 64-bit signed and unsigned integers, 32 and 64-bit real and complex floating point) and there will be no ambiguous elements. For example, the black/white points will always be unambiguously defined.
The new format will be completely open. There will be a publicly available formal description and open-source software tools and reference implementation libraries that will not depend on PixInsight.
A new integrated documentation system (20%)
This new system will allow you to browse documentation for modules and the core application from a dedicated graphical interface (an adaptation of the Qt Assistant application).
New port of PixInsight for FreeBSD (70%)
I really love FreeBSD. The first version will only be available for x86_64 (64-bit). It will be a Tier-1 implementation, fully stable and with all the features that PixInsight has currently on Linux/X11 (the reference platform).
That's all for now. Thanks for reading, and stay tuned!
I've been somewhat absent from the forum for a few weeks. However, this doesn't mean that I've stopped anything ?it is just the opposite.
Along with the usual bug fixes and updated modules (ImageRegistration and ImageCalibration will see important improvements), these are some important new features that will be included in PixInsight 1.6.1. For each item, I have written in parentheses the current approximate percentage of completion:
New automatic updates system (40%)
Starting from version 1.6.1, you normally won't need to download and install a whole new version of Pixinsight anymore, unless you want to install it on a different machine or hard disk. The new updates system will allow you to download and install updated packages for the different components of Pixinsight on your platform, in an automatic and transparent way.
This is an important milestone in PixInsight development. It means that from now on, any bug fix, updated or new module, etc., will be available immediately, as soon as we write it, to all users on all platforms. Right now we are completely stuck because I need to generate and pack a complete new version in order to fix all bugs that have been discovered, and to include all new modules, scripts, etc. In some cases I have published updated modules and patches, but this obviously isn't the right solution to the problem. The new updates system is the perfect solution, and will make PixInsight a much more dynamic and quickly evolving platform, improving the user's experience considerably.
New module installation system (100%)
The entire installation system has been redesigned and rewritten from scratch in the PCL and the core application. The new module installation model leads to a huge benefit for users and developers: Starting from version 1.6.1, existing PixInsight modules are guaranteed to continue working with all future releases, without requiring a recompilation against new PCL versions. As added benefits, module installation is now much faster, about a 30% faster than in 1.6.0, and much more secure (much more difficult to hack).
Extended information for developers
(normal users don't need to read this)
My initial intention was to turn the PCL library into a shared object (.so, .dylib or .dll). In this way new versions of PixInsight would just have to change the PCL object, and all existing modules would continue working. This would work because PCL performs all the low-level communication with Core, so as long as PCL's high-level ABI doesn't change, significant changes in the low-level layer are invisible to modules, so everything works across successive PCL versions.
Unfortunately, PCL cannot be a shared object in a portable way. Notably on Linux, FreeBSD and Mac OS X, several shared objects loaded dynamically by the same application cannot be clients of the same shared object X. When this happens, there exists a unique copy of X's static data in memory, which is shared among all clients, and hence can be modified asynchronously and produce all kinds of inconsistencies. This is a severe restriction because PCL uses static data internally. There are ways to avoid this problem when shared objects are loaded, but at the cost of limiting the number of installable PixInsight modules (less than 10 modules). There are other possible solutions, such as dividing PCL into several sub-modules to isolate all the static data, but this is very complicated and leads to an "ugly" solution.
My solution has been clean and relatively simple: keep PCL as a static library, and write a dynamic symbol resolver for PixInsight. Basically, PixInsight replaces the default object loader from the operating system (e.g. ld on Linux) with its own, custom loader. In this way the interface between a module and the PixInsight Core application does no longer depend on a complex low-level ABI, but just on a unique C function call that resolves the addresses of all low-level routines required by the module. It turns out that this works, and works remarkably well.
First Cocoa-based, 64-bit version of PixInsight for Mac OS X (100%)
See this thread. The new version is still experimental and has some glitches and issues, but is fully stable.
Improved PCL Module Development (100%)
(Information for developers only)
Starting from version 1.6.1, PixInsight modules don't depend on any external objects. So far most modules have been dependent on custom builds of the Little CMS, CFITSIO, LibTIFF and JPEGLib libraries. This no longer happens: all PixInsight modules can now be compiled and linked just against the PCL static library.
For this to be possible, all dependencies on LCMS (color management engine) have been removed from PCL and implemented statically in the Core application. In this way the entire color management functionality is now available to modules through the low-level C API, which does not require linkage against additional libraries. Note that the high-level C++ API (ICCProfile and ICCProfileTransformation classes) has not changed.
The built-in support for FITS, TIFF and JPEG has also been removed from the PCL. This means that the old FITS, TIFF and JPEG PCL classes no longer exist, and their functionality (and more) is now available exclusively through the intermodule communication mechanism provided by the FileFormatInstance class. For example, to read or write a FITS file, a module has to instantiate FileFormatInstance to access the standard FITS module (assuming it is installed) in a secure way. Many standard modules already do that, such as ImageCalibration, ImageRegistration and ImageIntegration.
Integration of the TraceMonkey JavaScript engine version 1.9 (30%)
My intention is to upgrade the current JavaScript engine in PixInsight (SpiderMonkey version 1.8.0 RC1) to the latest stable version (TraceMonkey 1.9). This is basically the same version of JavaScript that is being included in Mozilla Firefox 3.6. The best part of it is that the new engine includes a just-in-time compiler (JIT) that converts JavaScript code into machine code on the fly. This should lead to an important performance improvement.
Several building and portability issues are preventing me from completing this task. Mozilla's build system is awkward and overly complex (IMO), and very difficult to be adapted to the current PixInsight building scheme on all platforms. I'm working on it, but the fact that Mozilla does not release a new version of its JS engine officially as a standalone product makes everything really painful for embedders.
This task, however, will lead just to a transitional state because my real intention is to drop Mozilla's JavaScript engine in favor of QtScript. For more information about the current state and performance of QtScript, see this entry on Qt development blogs.
With QtScript we'll have Apple's SquirrelFish Extreme as the underlying JavaScript engine in PixInsight, with the added benefit of a total integration with Qt, and hence with PixInsight's GUI. Among other nice things this means that we'll have, along with a large performance improvement, a JavaScript debugger integrated with the current Script Editor window.
New PJSR functionality (75%)
The new Workspace and Icon core JavaScript objects will provide access to PixInsight workspaces from scripts. A script will be able to organize, move, resize and change the stacking order of image windows and icons, create, rename and delete workspaces, etc.
The new XMLReader and XMLWriter objects will add XML parsing and generation capabilities to PJSR.
Things being developed for version 1.6.x
More active development is taking place on several key features, which will be integrated in subsequent versions, during the 1.6 cycle:
A new, XML-based open image file format (10%)
The new format is the next step toward completion of a project functionality in PixInsight. This format will be able to store multiple images with their complete processing histories (as embedded XPSM sections), history states (undo/redo images), image metadata, ICC color profiles, color space definitions, arbitrary extensions, etc. Images will be stored in all the data types supported by PixInsight (8, 16, 32 and 64-bit signed and unsigned integers, 32 and 64-bit real and complex floating point) and there will be no ambiguous elements. For example, the black/white points will always be unambiguously defined.
The new format will be completely open. There will be a publicly available formal description and open-source software tools and reference implementation libraries that will not depend on PixInsight.
A new integrated documentation system (20%)
This new system will allow you to browse documentation for modules and the core application from a dedicated graphical interface (an adaptation of the Qt Assistant application).
New port of PixInsight for FreeBSD (70%)
I really love FreeBSD. The first version will only be available for x86_64 (64-bit). It will be a Tier-1 implementation, fully stable and with all the features that PixInsight has currently on Linux/X11 (the reference platform).
That's all for now. Thanks for reading, and stay tuned!
