Hi all,
After three months of intense research, development and testing work, I am glad to announce the release of a new version of PixInsight: 1.8.9. Given the amount of important changes, new features and improvements included in this version, we have decided to end the previous 1.8.8 cycle to start a new release series.
Installation packages for PixInsight 1.8.9 are now available to all licensed users for Linux, macOS and Windows:
https://pixinsight.com/dist/
As happens with all updates to the PixInsight core application, this version cannot be downloaded as an application update. You have to install version 1.8.9 manually:
New Features
The following is a partial list with the most important new features in PixInsight 1.8.9.
New Image Weighting Algorithms
In this version we have redesigned and reimplemented the image weighting algorithms that we have integrated with our standard image preprocessing pipeline. We now have two robust weighting methods based on a hybrid PSF/aperture photometry and multiscale analysis methodology, namely PSF Signal Weight (PSFSW) and PSF SNR. PSFSW is a comprehensive estimator able to capture several critical quality parameters, and PSF SNR tends to generate weights based exclusively on relative signal-to-noise ratio. Note that PSFSW is very different from its homonym method in previous versions. It also uses completely different metadata.
We have released a technical document where these algorithms are described in detail. The document also includes practical examples, a description of the implementation of these algorithms in our standard preprocessing tools, and a special section on regression analysis contributed by John Pane:
pixinsight.com
New LocalNormalization Process
Our standard LocalNormalization tool (LN) has also been completely redesigned and reimplemented in version 1.8.9 of PixInsight. LN implements a multiscale local normalization algorithm based on the multiscale median transform (MMT) and two methods for relative scale evaluation: one based on our hybrid PSF/aperture photometry approach (which is the default one), and another based on multiscale analysis techniques. LN is now a rock-solid, high-accuracy normalization process that can be fully automated and executed without supervision. It is now fully integrated with the WeightedBatchPreprocessing script (WBPP).
Automatic Calibration Pedestal Generation
The ImageCalibration tool, as well as the WBPP script, provide a new automatic mode for generation of output pedestals. When a prescribed minimum fraction of negative or insignificant pixels are detected after calibration, the process gathers the whole set of negative pixels and computes a robust and significant mean value, and the absolute value of this mean is the automatic pedestal. This feature has two advantages: pedestals are added when necessary in a completely automated way, and pedestals are optimized to minimize the amount of truncated pixels in a statistically significant way.
WeightedBatchPreprocessing Script Version 2.4.0
This version fully integrates the new image weighting and local normalization in our standard image preprocessing pipeline. It also adds a large amount of new features and usability improvements, including automatic pedestals, new pixel rejection algorithm selection criteria, automatic generation of high-quality local normalization reference images, and so many changes that the list is too big to include it here. Stay tuned because we are going to post a dedicated thread where all WBPP new features and improvements will be described in detail.
New WeightsOptimizer Script
The WeightsOptimizer script (WO), written by Roberto Sartori, finds the subframe weights that maximize or minimize a given set of properties in the integrated image. Mathematically this is addressed as an optimization problem. With this script you can specify an optimization expression and the subframes that need to be integrated, then WO runs its optimization solver to find the optimal or sub-optimal weights. WeightsOptimizer is fully documented:
pixinsight.com
New N* Robust Noise Estimator
N* (N-star) is as an alternative noise estimator available in ImageCalibration, Debayer, SubframeSelector and ImageIntegration (also in PJSR with specific new methods of the Image object). N* does not assume or require any particular statistical distribution in the evaluated data, contrarily to MRS, which assumes a normal distribution. However, the MRS algorithm continues being the default option for continuity and standardization reasons. The description of N* is included in the reference documentation for the new image weighting algorithms.
New Robust Chauvenet Rejection (RCR) Pixel Rejection Method
Robust Chauvenet Rejection is now available in the ImageIntegration process. This is an implementation of the algorithms described in the following paper:
Maples, M. et al. (2018). Robust Chauvenet Outlier Rejection. The Astrophysical Journal Supplement Series. 238. 2.
https://doi.org/10.3847/1538-4365/aad23d
RCR is an adaptive rejection algorithm which requires no parameters. However, in our implementation we have included a single parameter to control the degree of rejection with a simple modification to the algorithm. While we consider Extreme Studentized Deviate (ESD) the best rejection algorithm currently available, RCR is more controllable and its use is more intuitive when we need a precise control over pixel rejection by manual trial/error work. RCR is also at the core of the new LocalNormalization process and PSF flux based image weighting algorithms.
PixelMath: New
The pragma directive allows you to write PixelMath expressions that can be largely independent on process parameters. Each pragma directive argument modifies a PixelMath process parameter directly from the executed expression, overriding its current value. For example, you can force rescaling of the output image and using a single RGB/K expression as follows:
The complete list of available pragma arguments is described on the Expression Editor dialog.
The symbols directive defines a set of constants and variables directly from the executed expression. When the symbols directive is present it overrides the current contents of the symbols process parameter, that is, the contents of the Symbols section of the PixelMath window will be ignored if this directive is present in the expression(s). For example:
SubframeSelector: Support for .xnml and .xdrz Files
SubframeSelector accepts now local normalization files (.xnml) and drizzle integration files (.xdrz) in its list of input subframes, just as ImageIntegration does. If selected, these auxiliary files are written (with the appropriate modifications to point to the new target images) along with the accepted frames to the output directory by the Output subframes task. This allows us to use SS with registered frames, which is great to avoid having to register and normalize accepted files again when making tests with integrations of different subsets of images. Since all the noise and signal evaluation metadata have been generated for the original calibrated raw data, using registered frames makes no difference.
SubframeSelector: Ability to Sort Input Subframes
SubframeSelector has a new "Sort..." button, which allows you to sort the list of input subframes by acquisition times. This is very useful when file names don't provide coherent date/time information.
New DynamicPSF Features
The DynamicPSF tool has three new features:
Image Navigation: New Zoom At Cursor Feature
With this option enabled, zooming in/out images happens at the current cursor position. With this option disabled, zooming is applied at the current center coordinates (old behavior). This feature, which is enabled by default, can be controlled with Preferences, and can be toggled by pressing the Shift key when using the mouse wheel, or when clicking in Zoom In/Out mode.
New Copy STF Feature
Click on a view selector, then press Alt and hold it while you drop it on another view selector. This will copy the STFs of the starting view to the target view. The mouse cursor changes to show that the copy STF action is about to be applied.
Extended MultiscaleMedianTransform Scale Range
The MultiscaleMedianTransform process supports now up to 16 layers and linear scaling increments up to 128 pixels. This allows working with very-large-scale transforms and accelerated transforms based on linear scaling sequences.
Documentation Compiler Script: New Features
The Documentation Compiler script (version 1.6.8) accepts now
pixinsight.com
New Secure Script Code Signing System
Scripting resources are extremely powerful and versatile in PixInsight, and the degree of sophistication of the scripts available on our platform is now really amazing. All of this sophistication, along with the open nature of scripting code, involves important risks to both our users and our platform. For example, a script can be altered very easily to perform all kinds of malicious or even destructive actions, including exploits that can compromise data and identities. With the improvements that we are going to implement in the PixInsight JavaScript Runtime in future versions (new JavaScript engines, hybrid JavaScript/C++ runtime, etc), these risks will be even more relevant.
Obviously this situation requires an action on our part to make PixInsight a secure development platform. Since version 1.8.9, we start a program of Certified PixInsight Developers (CPDs) and a new Secure Script Code Signing System (SSCSS). CPDs are recognized software developers with the right to generate code signatures for their scripts. Code signatures enforce security of executed scripts, provide reputation to developers, and guarantee that the code being executed is exactly the same that the original developer has released. Signature entitlements allow execution of special actions that are considered potential security risks, both to the user and to PixInsight.
In version 1.8.9 of PixInsight we have implemented a first version of SSCSS, which is fully functional but does not introduce breaking changes by requiring only a limited set of entitlements to perform some potentially dangerous actions. We want to introduce code security as an important added value and a clear sign of evolution of our platform, but without causing problems to existing scripts and dependent implementations.
In this initial version of the new code security system, some of the most important scripts have been digitally signed with our corporate PTeam CPD identity. This includes, among others, WeightedBatchPreprocessing, the new WeightsOptimizer script, all development scripts, all platform initialization scripts, and all standard .jsh header files. We expect that most developers on the PixInsight platform will be interested in becoming CPDs in the immediate future.
Breaking Changes
The new image weighting methods require new metadata, which is not compatible with metadata generated by previous versions. This means that all data sets should be recalibrated (that is, preprocessed again) with version 1.8.9 in order to get optimal results.
ImageIntegration and DrizzleIntegration require local normalization data generated by the new LocalNormalization process. This means that existing .xnml files generated with previous versions are no longer valid. All .xnml files should be regenerated with the new LocalNormalization tool, or with the new WBPP script version 2.4.0.
Updated Third-Party Libraries
Recommended Platforms and Supported Operating Systems
The reference implementation of PixInsight, where you may expect the maximum performance and best user experience, is the Linux version. Currently our primary development platforms are:
We strongly recommend running PixInsight on Kubuntu Linux 20.04 LTS with the KDE Plasma desktop environment.
PixInsight 1.8.9 for Linux requires GLIBC 2.27 or a newer version.
PixInsight 1.8.9 for macOS is only compatible with macOS 10.15 Catalina, 11.x Big Sur and 12.x Monterey. macOS 10.14 Mojave and older versions are no longer supported. All of our macOS installation packages are digitally signed with our corporate Apple developer certificates and have been notarized by Apple.
PixInsight 1.8.9 for Windows is compatible with Windows 10 and Windows 11 exclusively. Windows 8.1, Windows 7 and older versions are not supported. The application might run on Windows 7, but there is no guarantee and please do not report problems on Windows 7 and Windows 8. All of our Windows executables have been digitally signed with our corporate extended validation (EV) code signing certificate.
Known Issues
Linux
Windows
macOS and Windows
All Platforms
——————————————————————————
Thank you for your attention,
The PixInsight Team at Pleiades Astrophoto S.L.
After three months of intense research, development and testing work, I am glad to announce the release of a new version of PixInsight: 1.8.9. Given the amount of important changes, new features and improvements included in this version, we have decided to end the previous 1.8.8 cycle to start a new release series.
Installation packages for PixInsight 1.8.9 are now available to all licensed users for Linux, macOS and Windows:
https://pixinsight.com/dist/
As happens with all updates to the PixInsight core application, this version cannot be downloaded as an application update. You have to install version 1.8.9 manually:
- Linux: The installation packages are xz-compressed tar archives (.tar.xz). To install one of these archives, just extract its full contents on any directory, preferably on one located under your home directory. Then become root and run the installer program from a terminal. Normally you should just type "yes" and press Enter to accept the default settings. Say './installer --help' for detailed information on command-line arguments.
- macOS: If you have a previous version of PixInsight already installed, you must uninstall it by moving the /Applications/PixInsight folder to trash (also known as bin on macOS 10.15 and later). Run the .pkg installation package and follow the instructions. All our macOS installation packages are digitally signed with our corporate Apple developer certificate and have been notarized by Apple.
- Windows: Run the installation package for version 1.8.9 and follow the instructions. The installer will remove a previous version automatically, so there is no need to uninstall it first. All executable files in the PixInsight distribution for Windows, as well as our Windows installation packages, have been signed with our corporate Extended Validation (EV) code signing certificate.
New Features
The following is a partial list with the most important new features in PixInsight 1.8.9.
New Image Weighting Algorithms
In this version we have redesigned and reimplemented the image weighting algorithms that we have integrated with our standard image preprocessing pipeline. We now have two robust weighting methods based on a hybrid PSF/aperture photometry and multiscale analysis methodology, namely PSF Signal Weight (PSFSW) and PSF SNR. PSFSW is a comprehensive estimator able to capture several critical quality parameters, and PSF SNR tends to generate weights based exclusively on relative signal-to-noise ratio. Note that PSFSW is very different from its homonym method in previous versions. It also uses completely different metadata.
We have released a technical document where these algorithms are described in detail. The document also includes practical examples, a description of the implementation of these algorithms in our standard preprocessing tools, and a special section on regression analysis contributed by John Pane:
PixInsight Reference Documentation | New Image Weighting Algorithms in PixInsight
A description of the image weighting algorithms currently implemented on the PixInsight platform.
pixinsight.com
New LocalNormalization Process
Our standard LocalNormalization tool (LN) has also been completely redesigned and reimplemented in version 1.8.9 of PixInsight. LN implements a multiscale local normalization algorithm based on the multiscale median transform (MMT) and two methods for relative scale evaluation: one based on our hybrid PSF/aperture photometry approach (which is the default one), and another based on multiscale analysis techniques. LN is now a rock-solid, high-accuracy normalization process that can be fully automated and executed without supervision. It is now fully integrated with the WeightedBatchPreprocessing script (WBPP).
Automatic Calibration Pedestal Generation
The ImageCalibration tool, as well as the WBPP script, provide a new automatic mode for generation of output pedestals. When a prescribed minimum fraction of negative or insignificant pixels are detected after calibration, the process gathers the whole set of negative pixels and computes a robust and significant mean value, and the absolute value of this mean is the automatic pedestal. This feature has two advantages: pedestals are added when necessary in a completely automated way, and pedestals are optimized to minimize the amount of truncated pixels in a statistically significant way.
WeightedBatchPreprocessing Script Version 2.4.0
This version fully integrates the new image weighting and local normalization in our standard image preprocessing pipeline. It also adds a large amount of new features and usability improvements, including automatic pedestals, new pixel rejection algorithm selection criteria, automatic generation of high-quality local normalization reference images, and so many changes that the list is too big to include it here. Stay tuned because we are going to post a dedicated thread where all WBPP new features and improvements will be described in detail.
New WeightsOptimizer Script
The WeightsOptimizer script (WO), written by Roberto Sartori, finds the subframe weights that maximize or minimize a given set of properties in the integrated image. Mathematically this is addressed as an optimization problem. With this script you can specify an optimization expression and the subframes that need to be integrated, then WO runs its optimization solver to find the optimal or sub-optimal weights. WeightsOptimizer is fully documented:
PixInsight Reference Documentation | WeightsOptimizer
Finds the sub-optimal subframe weights to optimize the measurement of the master integrated image.
pixinsight.com
New N* Robust Noise Estimator
N* (N-star) is as an alternative noise estimator available in ImageCalibration, Debayer, SubframeSelector and ImageIntegration (also in PJSR with specific new methods of the Image object). N* does not assume or require any particular statistical distribution in the evaluated data, contrarily to MRS, which assumes a normal distribution. However, the MRS algorithm continues being the default option for continuity and standardization reasons. The description of N* is included in the reference documentation for the new image weighting algorithms.
New Robust Chauvenet Rejection (RCR) Pixel Rejection Method
Robust Chauvenet Rejection is now available in the ImageIntegration process. This is an implementation of the algorithms described in the following paper:
Maples, M. et al. (2018). Robust Chauvenet Outlier Rejection. The Astrophysical Journal Supplement Series. 238. 2.
https://doi.org/10.3847/1538-4365/aad23d
RCR is an adaptive rejection algorithm which requires no parameters. However, in our implementation we have included a single parameter to control the degree of rejection with a simple modification to the algorithm. While we consider Extreme Studentized Deviate (ESD) the best rejection algorithm currently available, RCR is more controllable and its use is more intuitive when we need a precise control over pixel rejection by manual trial/error work. RCR is also at the core of the new LocalNormalization process and PSF flux based image weighting algorithms.
PixelMath: New
.pragma and .symbols DirectivesThe pragma directive allows you to write PixelMath expressions that can be largely independent on process parameters. Each pragma directive argument modifies a PixelMath process parameter directly from the executed expression, overriding its current value. For example, you can force rescaling of the output image and using a single RGB/K expression as follows:
.pragma rescale, single_expr;
3*sqrt( *$T )The complete list of available pragma arguments is described on the Expression Editor dialog.
The symbols directive defines a set of constants and variables directly from the executed expression. When the symbols directive is present it overrides the current contents of the symbols process parameter, that is, the contents of the Symbols section of the PixelMath window will be ignored if this directive is present in the expression(s). For example:
.pragma rescale, new_image;
.symbols m = median($T), b = 0.25;
b*($T < m)SubframeSelector: Support for .xnml and .xdrz Files
SubframeSelector accepts now local normalization files (.xnml) and drizzle integration files (.xdrz) in its list of input subframes, just as ImageIntegration does. If selected, these auxiliary files are written (with the appropriate modifications to point to the new target images) along with the accepted frames to the output directory by the Output subframes task. This allows us to use SS with registered frames, which is great to avoid having to register and normalize accepted files again when making tests with integrations of different subsets of images. Since all the noise and signal evaluation metadata have been generated for the original calibrated raw data, using registered frames makes no difference.
SubframeSelector: Ability to Sort Input Subframes
SubframeSelector has a new "Sort..." button, which allows you to sort the list of input subframes by acquisition times. This is very useful when file names don't provide coherent date/time information.
New DynamicPSF Features
The DynamicPSF tool has three new features:
- Draw FWHM. Draws an ellipse for each star corresponding to the fitted PSF at the full width at half maximum level. These ellipses are drawn with a solid line style.
- Draw FWTM. Draws an ellipse for each star corresponding to the fitted PSF at the full width at tenth maximum level. These ellipses are drawn with a dashed line style.
- Show best fits. When multiple PSF models are selected and fitted, draw only the fits with the smallest robust mean absolute deviation (MAD) residual for each star.
Image Navigation: New Zoom At Cursor Feature
With this option enabled, zooming in/out images happens at the current cursor position. With this option disabled, zooming is applied at the current center coordinates (old behavior). This feature, which is enabled by default, can be controlled with Preferences, and can be toggled by pressing the Shift key when using the mouse wheel, or when clicking in Zoom In/Out mode.
New Copy STF Feature
Click on a view selector, then press Alt and hold it while you drop it on another view selector. This will copy the STFs of the starting view to the target view. The mouse cursor changes to show that the copy STF action is about to be applied.
Extended MultiscaleMedianTransform Scale Range
The MultiscaleMedianTransform process supports now up to 16 layers and linear scaling increments up to 128 pixels. This allows working with very-large-scale transforms and accelerated transforms based on linear scaling sequences.
Documentation Compiler Script: New Features
The Documentation Compiler script (version 1.6.8) accepts now
imageWidth, imageHeight and imageBorder parameters for the \imageselect and \imageswap commands. These parameters provide control on the images used internally by these commands to generate mouseover comparisons. There is also a new include required special command that facilitates organization of large PIDoc documents into several units with the \include command. The complete reference documentation for the PIDoc language has been updated and is now available here:PixInsight Reference Documentation | The PixInsight Reference Documentation System
A description of the PixInsight Reference Documentation System and the PIDoc markup language.
pixinsight.com
New Secure Script Code Signing System
Scripting resources are extremely powerful and versatile in PixInsight, and the degree of sophistication of the scripts available on our platform is now really amazing. All of this sophistication, along with the open nature of scripting code, involves important risks to both our users and our platform. For example, a script can be altered very easily to perform all kinds of malicious or even destructive actions, including exploits that can compromise data and identities. With the improvements that we are going to implement in the PixInsight JavaScript Runtime in future versions (new JavaScript engines, hybrid JavaScript/C++ runtime, etc), these risks will be even more relevant.
Obviously this situation requires an action on our part to make PixInsight a secure development platform. Since version 1.8.9, we start a program of Certified PixInsight Developers (CPDs) and a new Secure Script Code Signing System (SSCSS). CPDs are recognized software developers with the right to generate code signatures for their scripts. Code signatures enforce security of executed scripts, provide reputation to developers, and guarantee that the code being executed is exactly the same that the original developer has released. Signature entitlements allow execution of special actions that are considered potential security risks, both to the user and to PixInsight.
In version 1.8.9 of PixInsight we have implemented a first version of SSCSS, which is fully functional but does not introduce breaking changes by requiring only a limited set of entitlements to perform some potentially dangerous actions. We want to introduce code security as an important added value and a clear sign of evolution of our platform, but without causing problems to existing scripts and dependent implementations.
In this initial version of the new code security system, some of the most important scripts have been digitally signed with our corporate PTeam CPD identity. This includes, among others, WeightedBatchPreprocessing, the new WeightsOptimizer script, all development scripts, all platform initialization scripts, and all standard .jsh header files. We expect that most developers on the PixInsight platform will be interested in becoming CPDs in the immediate future.
Breaking Changes
The new image weighting methods require new metadata, which is not compatible with metadata generated by previous versions. This means that all data sets should be recalibrated (that is, preprocessed again) with version 1.8.9 in order to get optimal results.
ImageIntegration and DrizzleIntegration require local normalization data generated by the new LocalNormalization process. This means that existing .xnml files generated with previous versions are no longer valid. All .xnml files should be regenerated with the new LocalNormalization tool, or with the new WBPP script version 2.4.0.
Updated Third-Party Libraries
- Qt updated to version 5.15.8 LTS on all supported platforms.
Recommended Platforms and Supported Operating Systems
The reference implementation of PixInsight, where you may expect the maximum performance and best user experience, is the Linux version. Currently our primary development platforms are:
We strongly recommend running PixInsight on Kubuntu Linux 20.04 LTS with the KDE Plasma desktop environment.
PixInsight 1.8.9 for Linux requires GLIBC 2.27 or a newer version.
PixInsight 1.8.9 for macOS is only compatible with macOS 10.15 Catalina, 11.x Big Sur and 12.x Monterey. macOS 10.14 Mojave and older versions are no longer supported. All of our macOS installation packages are digitally signed with our corporate Apple developer certificates and have been notarized by Apple.
PixInsight 1.8.9 for Windows is compatible with Windows 10 and Windows 11 exclusively. Windows 8.1, Windows 7 and older versions are not supported. The application might run on Windows 7, but there is no guarantee and please do not report problems on Windows 7 and Windows 8. All of our Windows executables have been digitally signed with our corporate extended validation (EV) code signing certificate.
Known Issues
Linux
- Recent versions of the KDE Plasma desktop environment include a morphing popups desktop effect that causes problems with recent PixInsight versions. If this effect is enabled, it slows down some dragging operations in the PixInsight core application, such as dragging icons and view selectors. You should disable this effect, which is very easy from KDE's System Settings panel.
- On recent versions of the KDE Plasma desktop environment, a display tearing prevention option (v-sync) is enabled by default. This option can cause input lag problems, such as slow cursor movement over image windows, with certain hardware configurations. If you experience these problems, this option should be disabled for optimal PixInsight performance:
- System Settings > Display and Monitor > Tearing prevention: set to 'Never'
- Click Apply
- PixInsight is not compatible with the open-source Nouveau graphics driver. If your distribution is using Nouveau, you must remove it to use the proprietary NVIDIA driver in order to use PixInsight.
Windows
- On some laptops with nonstandard screen resolutions, such as the Microsoft Surface, PixInsight cannot find valid screen scaling factors automatically. On these machines, you may have to run the PixInsight core application with the --ui-scaling command-line argument. For example, to apply a UI scaling factor of 2:
C:\Program Files\PixInsight\bin\PixInsight.exe --ui-scaling=2
You only have to do this once, since the scaling factor will be stored in application preferences automatically. Of course, if you already are using a version 1.8.6 or later and the interface looks correct, then you don't need to do this; this is only required for some new installations.
- The Windows installer package reports an invalid space requirement value (for example, 12 MB) sometimes. This is just an aesthetic issue without any practical consequence; the installation works perfectly despite this error.
macOS and Windows
- The INDIClient module is only available in the Linux version of PixInsight. We hope to solve this problem to release macOS and Windows versions of this module as soon as possible. Sorry for the inconvenience.
All Platforms
- The StarNet module uses the TensorFlow library, which requires a processor with support of AVX2 and FMA instructions. This excludes very old machines with Intel processors released before 2013 and Apple machines with M1 processors (because the Rosetta emulator on macOS Big Sur does not support AVX2 instructions). The macOS installation package for version 1.8.9 includes a post-install script that will delete the StarNet module automatically when the host machine runs an ARM processor, so there should be no problems on Apple Silicon computers.
If you are using a vintage pre-2013 Intel processor, or if the StarNet module has not been deleted automatically on your M1 machine for some odd reason, PixInsight 1.8.9 won't work as it is installed by default. However, this problem is very easy to fix: just remove the StarNet module. Do the following:
Linux
Enter the following command from a terminal:
sudo rm /opt/PixInsight/bin/StarNet-pxm.so
macOS
Remove the following file:
/Applications/PixInsight/bin/StarNet-pxm.dylib
Windows
Remove the following file:
C:\Program Files\PixInsight\bin\StarNet-pxm.dll
In all cases you need administrative privileges to remove the required file, which you also need to install PixInsight. After this simple action, PixInsight will run without problems on your old machine.
——————————————————————————
Thank you for your attention,
The PixInsight Team at Pleiades Astrophoto S.L.