Hi all,
I am glad to announce a new version of PixInsight: 1.8.8-10. This release provides important new features and improvements on all platforms.
Installation packages for PixInsight 1.8.8-10 are now available to all licensed users for Linux, macOS and Windows:
https://pixinsight.com/dist/
As happens with all updates to the PixInsight core application, this version cannot be downloaded as an application update. You have to install version 1.8.8-10 manually:
Main Improvements
This version is an important release focused on performance and stability improvements, paving the way towards PixInsight 1.8.9. Version 1.8.8-10 fixes all confirmed bugs since the previous 1.8.8-9 release and provides some relevant new features and improvements. A partial list follows with the most important changes.
New Image Weighting Algorithms
PixInsight 1.8.8-10 comes with two new image weighting methods:
Here is an example with the new version of the SubframeSelector tool, which now includes automatic altitude and azimuth measurements:
This is a quality test data set kindly provided by Adam Block. Note the strong correlation between PSF signal weight (blue line graph) and altitude above the horizon (dotted gray line graph) on the screenshot above.
Despite being about 7 degrees lower over the horizon, frame #56 (void circle) has a much higher PSF signal weight than #55 (circle with red cross). Why? Because the latter was exposed through considerable haze, besides having large tracking errors, as can be seen on the screenshot above (click on the image to enlarge). This is just an example of the efficiency and robustness of the new PSF photometry weighting methods.
Technical Description
PSF Signal Weight is a conceptually simple image weighting method based on PSF photometry. We begin by detecting stars in the image. This first step is critical because we want to detect stars, not extended objects or potential bright artifacts such as uncorrected hot pixels, cosmics, etc. We already have this task well implemented in our code base. The second step is fitting a point spread function to each detected star. We have a state-of-the-art implementation of PSF fitting based on the Levenberg-Marquardt algorithm. Now for each detected and successfully fitted star compute a weighted mean signal estimate:
}{B}}\textup{PSF}_i}{\sum_{i\in&space;r}\textup{PSF}_i},\quad&space;v_i&space;>&space;B,)
where
is the star detection region,
represents a pair of coordinates within
,
is a pixel sample value,
is the local background estimate, and
is the fitted point spread function normalized to unit amplitude and evaluated at
. The resulting image weight is:
\sigma_n},\quad&space;s_i&space;\le&space;s_{i+1},\quad&space;t_0&space;<&space;t_1,)
where
and
are clipping indexes calculated with a special iterative sigma clipping procedure,
is a PSF mean signal estimate, and
is an estimate of the standard deviation of the noise, computed for the entire image with the MRS algorithm using 4 wavelet layers. The correct balance between the statistical properties of robustness and efficiency is crucially important here, hence we are using a carefully implemented trimmed mean instead of the mean (efficient but non-robust) or the median (robust but inefficient).
The PSF Power Weight variant is as follows. For each successfully fitted PSF compute a mean weighted squared signal estimate:
^2}{B^2}}\textup{PSF}_i}{\sum_{i\in&space;r}\textup{PSF}_i},\quad&space;v_i&space;>&space;B,)
then the PSF power weight is given by:
\sigma_n^2},\quad&space;s_{2i}&space;\le&space;s_{2i+1},\quad&space;t_0&space;<&space;t_1.)
The results achieved with these weighting methods are remarkably reliable and consistent in all of our tests. PSF Signal is now the default weighting algorithm in ImageIntegration. PSF Signal and PSF Power are directly available in SubframeSelector as new PSFSignalWeight and PSFPowerWeight variables.
These methods implement image weights as direct estimates of the signal-to-noise ratio. As is well known, there are two main interpretations of SNR:
Option 1 is the concept of SNR normally used in most engineering applications, and the only one that we have had until now. Option 2 is equally valid in my opinion, although of course both are mutually incompatible in terms of the interpretation of the data being measured. We now have both available with the same robust and efficient implementation.
Three very important properties of the new weighting methods must be pointed out:
The improved SNR Weight method shows a nice example of the robustness versus efficiency problem in statistics. This algorithm is the classical formulation of SNR as a ratio of powers (option 1 described above):
where
is an estimate of statistical variability for the image and
is an estimate of the standard deviation of the noise. This formulation actually represents a scaled version of the inverse noise variance weighting method. Since the inverse variance weighted average is a maximum likelihood estimator of deterministic signal, this method can theoretically yield the highest possible SNR improvement in the integrated result. The standard deviation of the noise can be evaluated without problems with the MRS algorithm, which is remarkably robust. However, a meaningful estimate of dispersion for the image is problematic in this context because we need a highly efficient estimator, which excludes robust scale estimators. This poses a very hard problem. Basically, we have tested all known estimators of statistical scale and none works for this task except—of course, what else—the old and so often disdained standard deviation. The reason is that any attempt to reject outliers beyond a delicate limit damages the efficiency of the estimator, which leads to a completely meaningless result. This is precisely what was happening with the—now deprecated and completely removed—old noise evaluation weighting method in previous versions of PixInsight. In the new implementation I have designed a special method to evaluate the standard deviation of the image in three successive iterations, with decreasing tolerances based on the previous estimates. This is a sort of self-stabilizing estimator that performs a slight but sufficient rejection of outliers without damaging efficiency. In addition, the estimator is bilateral, which means that dispersion is evaluated separately for pixels below and above the median. This makes it even more efficient for asymmetrical distributions (graphically: asymmetrical histogram peaks). Despite these efforts, this method is sensitive to large outliers, such as plane trails and similar artifacts, which inevitably degrade its performance to extents proportional to the amount of outlier pixels.
This technical description, extended to include some important algorithmic and implementation details, along with interesting examples showing the performance of the new weighting methods with real data, will be included in an announcement post on the Release Information forum board, as well as in a dedicated software tutorial that I am preparing.
Acknowledgments
Thanks to John Murphy and Adam Block for exposing the problems and deficiencies of the old noise evaluation weighting method, and for pushing me to solve them with a completely new implementation.
Embedded Processing Histories in XISF Files
As the title says, now you can save the processes applied to an image in an XISF file. When the XISF file is opened, the stored processing history is loaded automatically and is available as the image's initial state on History Explorer. This provides basically the same functionality as if you'd save a process icons file with the complete processing applied to the image, but the process instances are embedded in the same XISF file instead of a separate .xpsm file. In other words, images know how they have been processed and can remember it.
There are new items on the XISF Options dialog and the Preferences tool to control this feature:
Here is an example of initial state for an image saved with embedded processing history:
New NetworkService Module
NetworkService implements an asynchronous, network-based, distributed client-server task processing system that can be controlled and monitored through interprocess communication (IPC) messages. This module is at the heart of our AstroBin+PixInsight collaboration and other similar projects that are being developed now, where PixInsight runs in a completely automated way as an on-demand service. Now we have released the generic NetworkService module as an open-source product, and have included it in the official PixInsight distribution, available to developers on the PixInsight/PCL platform.
NetworkService's general workflow is as follows:
PCL Development Files Available on the Standard PixInsight Distribution
In preparation for important new features that will be implemented in future versions (such as an embedded C++ compiler and a hybrid JavaScript/C++ runtime), critical parts of our PixInsight Class Library (PCL) development framework are now included in the standard distribution of PixInsight on all supported platforms.
This will simplify the development of new PixInsight modules because all required header and source files are now available within the local PixInsight installation, subject to automatic updates as new features and bug fixes are being released.
Other Important Improvements and Bug Fixes
Updated Support Libraries
Recommended Platforms and Supported Operating Systems
The reference implementation of PixInsight, where you may expect the maximum performance and best user experience, is the Linux version. Currently our primary development platforms are:
We strongly recommend running PixInsight on Kubuntu Linux 20.04 LTS with the KDE Plasma desktop environment.
PixInsight 1.8.8-10 for Linux requires GLIBC 2.27 or a newer version.
PixInsight 1.8.8-10 for macOS is only compatible with macOS 10.15 Catalina, 11.x Big Sur, and 12.x Monterey. macOS 10.14 Mojave and older versions are no longer supported. All of our macOS installation packages are digitally signed with our corporate Apple developer certificates and have been notarized by Apple.
PixInsight 1.8.8-10 for Windows is compatible with Windows 10 and Windows 11 exclusively. Windows 8.1, Windows 7 and older versions are not supported. The application might run on Windows 7, but there is no guarantee and please do not report problems on Windows 7 and Windows 8. All of our Windows installation executables have been digitally signed with our corporate extended validation (EV) code signing certificate.
Known Issues
FreeBSD and Linux
macOS
You only have to do this once, since the scaling factor will be stored in application preferences automatically. Of course, if you already are using a version 1.8.6 or later and the interface looks correct, then you don't need to do this; this is only required for new installations.
All Platforms
——————————————————————————
Thank you for your attention and support.
I am glad to announce a new version of PixInsight: 1.8.8-10. This release provides important new features and improvements on all platforms.
Installation packages for PixInsight 1.8.8-10 are now available to all licensed users for Linux, macOS and Windows:
https://pixinsight.com/dist/
As happens with all updates to the PixInsight core application, this version cannot be downloaded as an application update. You have to install version 1.8.8-10 manually:
- Linux: The installation packages are xz-compressed tar archives (.tar.xz). To install one of these archives, just extract its full contents on any directory, preferably on one located under your home directory. Then become root and run the installer program from a terminal. Normally you should just type "yes" and press Enter to accept the default settings. Say './installer --help' for detailed information on command-line arguments.
- macOS: If you have a previous version of PixInsight already installed, you must uninstall it by moving the /Applications/PixInsight folder to trash (also known as bin on macOS 10.15 and later). Run the .pkg installation package and follow the instructions. All our macOS installation packages are digitally signed with our corporate Apple developer certificate and have been notarized by Apple.
- Windows: If you have a previous version of PixInsight already installed, please uninstall it using Add or Remove Programs. Then run the installation package for version 1.8.8-10 and follow the instructions. All executable files in the PixInsight distribution for Windows, as well as our installation packages, have been signed with our corporate Extended Validation (EV) code signing certificate.
Main Improvements
This version is an important release focused on performance and stability improvements, paving the way towards PixInsight 1.8.9. Version 1.8.8-10 fixes all confirmed bugs since the previous 1.8.8-9 release and provides some relevant new features and improvements. A partial list follows with the most important changes.
New Image Weighting Algorithms
PixInsight 1.8.8-10 comes with two new image weighting methods:
- A new weighting method based on signal estimation with PSF photometry, available in two variants: PSF Signal and PSF Power. This method is robust and accurate, and is the new default option in the ImageIntegration process. The old noise evaluation weighting method has been deprecated and removed.
- An improved version of the SNRWeight method.
Here is an example with the new version of the SubframeSelector tool, which now includes automatic altitude and azimuth measurements:
This is a quality test data set kindly provided by Adam Block. Note the strong correlation between PSF signal weight (blue line graph) and altitude above the horizon (dotted gray line graph) on the screenshot above.
Despite being about 7 degrees lower over the horizon, frame #56 (void circle) has a much higher PSF signal weight than #55 (circle with red cross). Why? Because the latter was exposed through considerable haze, besides having large tracking errors, as can be seen on the screenshot above (click on the image to enlarge). This is just an example of the efficiency and robustness of the new PSF photometry weighting methods.
Technical Description
PSF Signal Weight is a conceptually simple image weighting method based on PSF photometry. We begin by detecting stars in the image. This first step is critical because we want to detect stars, not extended objects or potential bright artifacts such as uncorrected hot pixels, cosmics, etc. We already have this task well implemented in our code base. The second step is fitting a point spread function to each detected star. We have a state-of-the-art implementation of PSF fitting based on the Levenberg-Marquardt algorithm. Now for each detected and successfully fitted star compute a weighted mean signal estimate:
where
where
The PSF Power Weight variant is as follows. For each successfully fitted PSF compute a mean weighted squared signal estimate:
then the PSF power weight is given by:
The results achieved with these weighting methods are remarkably reliable and consistent in all of our tests. PSF Signal is now the default weighting algorithm in ImageIntegration. PSF Signal and PSF Power are directly available in SubframeSelector as new PSFSignalWeight and PSFPowerWeight variables.
These methods implement image weights as direct estimates of the signal-to-noise ratio. As is well known, there are two main interpretations of SNR:
- As the ratio of the expected signal power and the variance of the noise, or mean-squared signal-to-noise ratio. This is the classical concept, compatible with e.g. Gonzalez/Woods, Digital Image Processing. A robust approach to this interpretation is now implemented as the PSF Power weighting method.
- As the ratio of the mean signal and the standard deviation of the noise. A robust approach to this interpretation is now implemented as the PSF Signal weighting method.
Three very important properties of the new weighting methods must be pointed out:
- Both methods work directly on calibrated raw data before any interpolation. This is true for monochrome CCD/CMOS data and CFA mosaiced data.
- The new methods don't depend on a reference image. Actually, the concept of reference image no longer exists in the ImageIntegration process for image weighting.
- The new methods generate universally comparable weights. This means that, in principle and without the influence of other external factors, one can compare weights calculated for different data sets.
The improved SNR Weight method shows a nice example of the robustness versus efficiency problem in statistics. This algorithm is the classical formulation of SNR as a ratio of powers (option 1 described above):
where
This technical description, extended to include some important algorithmic and implementation details, along with interesting examples showing the performance of the new weighting methods with real data, will be included in an announcement post on the Release Information forum board, as well as in a dedicated software tutorial that I am preparing.
Acknowledgments
Thanks to John Murphy and Adam Block for exposing the problems and deficiencies of the old noise evaluation weighting method, and for pushing me to solve them with a completely new implementation.
Embedded Processing Histories in XISF Files
As the title says, now you can save the processes applied to an image in an XISF file. When the XISF file is opened, the stored processing history is loaded automatically and is available as the image's initial state on History Explorer. This provides basically the same functionality as if you'd save a process icons file with the complete processing applied to the image, but the process instances are embedded in the same XISF file instead of a separate .xpsm file. In other words, images know how they have been processed and can remember it.
There are new items on the XISF Options dialog and the Preferences tool to control this feature:
Here is an example of initial state for an image saved with embedded processing history:
New NetworkService Module
NetworkService implements an asynchronous, network-based, distributed client-server task processing system that can be controlled and monitored through interprocess communication (IPC) messages. This module is at the heart of our AstroBin+PixInsight collaboration and other similar projects that are being developed now, where PixInsight runs in a completely automated way as an on-demand service. Now we have released the generic NetworkService module as an open-source product, and have included it in the official PixInsight distribution, available to developers on the PixInsight/PCL platform.
NetworkService's general workflow is as follows:
- A new task request is generated somewhere and stored in a task requests database, which is publicly accessible on the Internet through a task provider URL.
- NetworkService is running somewhere on PixInsight and checking the task provider URL periodically.
- If a new task request is available, NetworkService retrieves it (with secure authentication) and executes the specified task.
- When NetworkService completes the task, it sends the corresponding data and notification to a callback URL (which was specified in the task request).
PCL Development Files Available on the Standard PixInsight Distribution
In preparation for important new features that will be implemented in future versions (such as an embedded C++ compiler and a hybrid JavaScript/C++ runtime), critical parts of our PixInsight Class Library (PCL) development framework are now included in the standard distribution of PixInsight on all supported platforms.
This will simplify the development of new PixInsight modules because all required header and source files are now available within the local PixInsight installation, subject to automatic updates as new features and bug fixes are being released.
Other Important Improvements and Bug Fixes
- The Windows version of PixInsight is now a long path aware application. This means that file and directory paths are no longer limited to 260 characters.
- The number of file reading and writing threads is now 8 by default. We are no longer optimized for obsolete rotational disks. We now offer a reasonable default number of I/O threads for SSD devices. However, please note that the optimal number of I/O threads depends on a variety of hardware and software related factors, and must be found through benchmarks. For fast NVMe devices on systems with 32 or more logical processors, 16 or 24 I/O threads can be a good starting point.
- StarAlignment: Improved local distortion correction thanks to a redesigned implementation of the surface spline interpolation C++ classes, including new radial basis functions and improved quadtree-based recursive surface splines.
- WeightedBatchPreprocessing script version 2.3.0, with support of the new PSF photometry weighting method and other improvements and new features — the list is too long to include it here, so stay tuned for a dedicated announcement post.
- PhotometricMosaic script version 3.4.0, with important improvements and new features. See the complete description here.
- NormalizeScaleGradient script version 1.4.3.
- AnnotateImage script version 2.2.2: Many small bug fixes, improved coordinate labels with superscript 'hms' units, new Milliquas catalog (Million Quasars Catalog). All of these improvements have been developed in the context of our AstroBin+PixInsight collaboration.
- ImageSolver script version 5.6.2: Fixed a rare loop stalling condition in the surface simplifier algorithm. Also from our AstroBin+PixInsight collaboration.
- CatalogStarGenerator script version 2.1.6 with improved graphical interface and automation capabilities. Again, an improvement implemented to support our AstroBin+PixInsight collaboration.
- Fixed a serious performance regression caused by the suppression of the dark frame optimization window parameter of ImageCalibration. The dark frame optimization algorithm is now based on variance minimization instead of noise evaluation. It is now much faster and more accurate.
- Text on message boxes is no longer entirely written with a bold font on macOS.
- Official Windows 11 support.
- Official macOS 12 Monterey support.
Updated Support Libraries
- Little CMS Engine (LCMS) version 2.12.
- cURL version 7.79.1.
- LibTIFF version 4.3.0, released April 2021.
- CFITSTIO version 4.0.0, released May 2021.
- Dygraphs version 2.1.0.
- Qt version 5.15.7 LTS on Linux and Windows. Unfortunately, the macOS version uses Qt 5.15.6 LTS because of unexpected/undocumented changes introduced in macOS 12 Monterey.
- Apple Clang version 13.0.0 with the macOS 12 SDK used for all macOS development.
- Microsoft Visual Studio 2019 version 16.11.6 used for all Windows development.
Recommended Platforms and Supported Operating Systems
The reference implementation of PixInsight, where you may expect the maximum performance and best user experience, is the Linux version. Currently our primary development platforms are:
We strongly recommend running PixInsight on Kubuntu Linux 20.04 LTS with the KDE Plasma desktop environment.
PixInsight 1.8.8-10 for Linux requires GLIBC 2.27 or a newer version.
PixInsight 1.8.8-10 for macOS is only compatible with macOS 10.15 Catalina, 11.x Big Sur, and 12.x Monterey. macOS 10.14 Mojave and older versions are no longer supported. All of our macOS installation packages are digitally signed with our corporate Apple developer certificates and have been notarized by Apple.
PixInsight 1.8.8-10 for Windows is compatible with Windows 10 and Windows 11 exclusively. Windows 8.1, Windows 7 and older versions are not supported. The application might run on Windows 7, but there is no guarantee and please do not report problems on Windows 7 and Windows 8. All of our Windows installation executables have been digitally signed with our corporate extended validation (EV) code signing certificate.
Known Issues
FreeBSD and Linux
- Recent versions of the KDE Plasma desktop environment include a morphing popups desktop effect that causes problems with recent PixInsight versions. If this effect is enabled, it slows down some dragging operations in the PixInsight core application, such as dragging icons and view selectors. You should disable this effect, which is very easy from KDE's System Settings panel.
- On recent versions of the KDE Plasma desktop environment, a display tearing prevention option (v-sync) is enabled by default. This option can cause input lag problems, such as slow cursor movement over image windows, with certain hardware configurations. If you experience these problems, this option should be disabled for optimal PixInsight performance:
- System Settings > Display and Monitor > Tearing prevention: set to 'Never'
- Click Apply
- PixInsight is not compatible with the open-source Nouveau graphics driver. If your distribution is using Nouveau, you must remove it to use the proprietary NVIDIA driver in order to use PixInsight.
macOS
- On Apple MacBook laptops with dual graphics cards,we recommend disabling the integrated Intel HD graphics adapter before running PixInsight for maximum performance and stability. This can be done very easily:
- System Preferences > Energy Saver > Automatic graphics switching: disable.
- On some laptops with nonstandard screen resolutions, such as the Microsoft Surface, PixInsight cannot find valid screen scaling factors automatically. On these machines, you may have to run the PixInsight core application with the --ui-scaling command-line argument. For example, to apply a UI scaling factor of 2:
C:\Program Files\PixInsight\bin\PixInsight.exe --ui-scaling=2You only have to do this once, since the scaling factor will be stored in application preferences automatically. Of course, if you already are using a version 1.8.6 or later and the interface looks correct, then you don't need to do this; this is only required for new installations.
All Platforms
- The StarNet module uses the TensorFlow library, which requires a processor with support of AVX2 and FMA instructions. This excludes very old machines with Intel processors released before 2013 and Apple machines with M1 processors (because the Rosetta emulator on macOS Big Sur does not support AVX2 instructions). The macOS installation package for version 1.8.8-10 includes a post-install script that will delete the StarNet module automatically when the host machine runs an ARM processor, so there should be no problems on Apple Silicon computers.
If you are using a vintage pre-2013 Intel processor, or if the StarNet module has not been deleted automatically on your M1 machine for some odd reason, PixInsight 1.8.8-10 won't work as it is installed by default. However, this problem is very easy to fix: just remove the StarNet module. Do the following:
Linux
Enter the following command from a terminal:
sudo rm /opt/PixInsight/bin/StarNet-pxm.so
macOS
Remove the following file:
/Applications/PixInsight/bin/StarNet-pxm.dylib
Windows
Remove the following file:
C:\Program Files\PixInsight\bin\StarNet-pxm.dll
In all cases you need administrative privileges to remove the required file, which you also need to install PixInsight. After this simple action, PixInsight will run without problems on your old machine.
——————————————————————————
Thank you for your attention and support.
Last edited:
