PixInsight 1.8.8 Released

Juan Conejero

PTeam Member
Sep 2, 2004
7,364
58
56
Valencia, Spain
pixinsight.com
I am glad to announce that PixInsight 1.8.8 has been released today. This version is a new milestone in the history of PixInsight, with important new features and a strong focus on stability and performance. It is the result of an intense research and development work carried out during the last months. Expect much more during the months ahead.

Installation packages for PixInsight 1.8.8 are now available to all licensed users for FreeBSD, Linux, macOS and Windows:

https://pixinsight.com/dist/

This version is a major update that you have to install manually. It is not available as an application update.

If you have version 1.8.6 or 1.8.7 already installed, this new version does not require a license reactivation. If you are still using a 1.8.5 version or older, see the official 1.8.6 version announcement for a detailed description of our new licensing system. See also FAQ 2.10 for additional information on license reactivations.


New Features

Here is a brief list with the most important changes and new features implemented in PixInsight 1.8.8.



Improved Real-Time Preview

The Real-Time Preview interface has been largely redesigned and reimplemented in version 1.8.8 of PixInsight. You can now zoom the real-time preview image by clicking and dragging with the mouse to define a rectangular selection. This can be done in Select Preview mode (click the concentric squares button), or directly by clicking the middle mouse button and dragging.


You can navigate through a chain of successive zoomed previews with the Next Preview, Previous Preview and Full Preview buttons. Finally, you can adjust the size of the real-time preview window to fit the current image automatically by clicking the Fit Preview button. The rest of real-time preview features and functions are basically the same as in previous versions.

Besides new user interface features, the Real-Time Preview interface is now faster, more accurate and reliable with all tools currently implementing real-time preview functionality.



New Pixel Rejection Algorithm: Generalized Extreme Studentized Deviate (ESD)

The ImageIntegration tool implements a new pixel rejection algorithm: Generalized Extreme Studentized Deviate (ESD) Test, or ESD rejection for short. This is an implementation of the method described by Bernard Rosner in his 1983 paper Percentage Points for a Generalized ESD Many-Outlier procedure, adapted to the image integration task. The ESD algorithm assumes that each pixel stack, in absence of outliers, follows an approximately normal (Gaussian) distribution. It aims at avoiding masking, a serious issue that occurs when an outlier goes undetected because its value is similar to another outlier. The performance of this algorithm can be excellent for large data sets of 25 or more images, and especially for very large sets of 50 or more frames. I have seen very good results in some of our tests for more modest sets of 12-15 images.

ESD rejection takes just two parameters:

Expected maximum fraction of outliers. For example, a value of 0.2 applied to a stack of 10 pixels means that the ESD algorithm will be limited to detect a maximum of two outlier pixels, or in other words, only 0, 1 or 2 outliers will be detectable in such case. The default value is 0.3, which allows the algorithm to detect up to a 30% of outlier pixels in each pixel stack.

Significance level. This is the probability of making a Type I error (false positive), or the significance level of the outlier detection hypothesis test. For example, a significance level of 0.01 means that a 1% chance of being wrong when rejecting the null hypothesis (that there are no outliers in a given pixel stack) is acceptable. The default value is 0.05 (5% significance level).

The documentation of the R implementation of this algorithm provides interesting information with the results of 10,000 simulations for different data sets and parameters. I'll describe this algorithm in detail, including our implmentation and its advantages for the image integration task, in the release information board of this forum during the coming days.



PixelMath: New Functions and Features

The following PixelMath functions have been implemented in version 1.8.8 of PixInsight:

Code:
[b]range( x, a, b )[/b]
x constrained to the [a,b] range:
Returns x if x >= a and x <= b
Returns a if x < a
Returns b if x > b

Code:
[b]rescale( x, a, b )[/b]
x rescaled linearly from the [a,b] range to [0,1]:
Returns 0 if x < a or b - a is zero or insignificant.
Returns 1 if x > b
Returns (x - a)/(b - a) if x >= a and x <= b
a and b are swapped internally (and transparently) if b < a.

Code:
[b]gauss()[/b]
Normal (Gaussian) random deviate with zero mean and unit standard deviation. A typical application of this function to add normal distributed noise to the target image is:

Code:
$T + k*mdev( $T )*gauss()

where the median absolute deviation from the median (MAD), multiplied by an optional amplitude parameter k, defines the standard deviation of additive noise adaptively.

Code:
[b]poisson( x )[/b]
Random Poisson deviate for the specified pixel value x. A typical application of this function is:

Code:
poisson( 65535*$T )/65535

which will generate Poisson noise for the current pixel value in the [0,1] range, simulating a 16-bit detector of unit gain.

Code:
[b]studentt( degreesOfFreedom )[/b]
Random deviate from a Student's t distribution with the specified degrees of freedom, which must be > 0.

Code:
[b]chisq( degreesOfFreedom )[/b]
Random deviate from a chi squared distribution with the specified degrees of freedom, which must be > 0.

Code:
[b]gamma( shape[, scale=1] )[/b]
Random deviate from a Gamma distribution with the specified shape and (optional) scale parameters. If not specified, the default scale is 1.0.

PixelMath comments

The PixelMath language supports now C-style comments with the '/*' and '*/' delimiters. This is very useful when one wants to document briefly the PixelMath expressions, or to try out several function parameters or subexpressions without having to create temporary process icons. For example, the following expression can be used to combine two images selectively:

Code:
iif( $T < k, $T + (1 - k)*A, $T )

If one wants to check which areas of the target image will be combined with pixels from A, a comment can be used as follows:

Code:
iif( $T < k, 1/*$T + (1 - k)*A*/, $T )

which will replace all target pixels less than k with white temporarily. After finding a good value for k in this way, the comment can be removed to apply the actual PixelMath transformation.



Improved Blink Tool


Blink is now faster and more reliable. Besides important improvements in its graphical interface, Blink implements now a fully parallelized statistics calculation procedure. On machines with a reasonable number of CPU cores, Blink's automatic histogram transformation is orders of magnitude faster than before. The same applies to the series analysis report.



PixInsight Platform: Improved Multithreading Performance

In this version we have reimplemented a significant part of our code base to improve performance of multithreaded code. The improvement is especially relevant for machines with many cores running CPU-intensive tasks. See for example these benchmarks performed on the same workstation:

http://pixinsight.com/benchmark/benchmark-report.php?sn=4ON47FRJB0NF090S421V1LHNH2B6O9EW
http://pixinsight.com/benchmark/benchmark-report.php?sn=2BEVIS11SHGMU7F0774YJ3HTN4YH1MDV

Besides the difference in the amount of installed RAM, which is not relevant in this case, the machine is the same in both benchmarks: AMD Ryzen Threadripper 2990WX with 64 or 128 GiB of RAM (DDR4 UDIMM ECC 2666 MHz). With 32 physical cores and 64 threads, this machine shows a performance improvement close to a 10% in version 1.8.8 with respect to versions 1.8.6 and 1.8.7 (CPU performance indexes of 25087 and 22844, respectively).



PixInsight Platform: Improved Project Management

Bundled PixInsight projects (.pxiproj file suffix) can now be freely renamed without breaking their functionality. Of course, existing projects will work flawlessly and will be converted to the new format transparently when rewritten with the File > Save Project command.

Besides this, loading projects is now faster and the image reading tasks are now more responsive and provide more feedback on the process console.



PixInsight Platform: New Hack Font

The Hack font has replaced DejaVu Sans Mono as the standard core manospaced font on all platforms. Hack provides better screen legibility for source code and pixel readouts.



PixInsight Platform: New Pseudo-Random Number Generators

Since version 1.8.8 of PixInsight we use exclusively xoshiro256** and xoroshiro1024** PRNGs (XOR/shift/rotate and XOR/rotate/shift/rotate, respectively) on the entire PixInsight/PCL/PJSR platform. Previously used Marsenne Twister and xorshift generators have been deprecated, although they are still available for compatibility.



PixInsight Platform: Support for Bitmaps Larger than 2 GiB


Bitmap objects (available on the PCL and PJSR development environments) can now be larger than 2 GiB. This includes pixel data allocation and direct pixel access, along with the entire drawing capabilities of our Graphics and VectorGraphics objects. This overcomes an important limitation present in core versions 1.8.7 and older.



Updated Third-Party Support Libraries

PixInsight 1.8.8 includes the following updates to critical support libraries:

* The RAW format support module integrates the latest LibRaw 201910 snapshot version. This means that we now support Canon CR3 raw files, among many other new camera raw formats and other improvements.

* Core application built with the latest Qt version 5.13.2. The core PixInsight 1.8.8 application uses the latest version of the Qt library (as of writing this) on Linux, macOS and Windows. We are still using Qt 5.12.2 on FreeBSD, but this will change as soon as possible.



Bug Fixes

PixInsight 1.8.8 fixes all confirmed bugs detected since the last 1.8.7 release, with a unique exception: a complex bug involving the PixelMath and Statistics tools, which still is present in this version. I'll try to fix this problem as soon as possible with an update to the core application.


Known Problems

FreeBSD

* Recent versions of the KDE Plasma desktop environment include a morphing popups desktop effect that causes problems with recent PixInsight versions. If this effect is enabled, it slows down some dragging operations in the PixInsight core application, such as dragging icons and view selectors. You should disable this effect, which is very easy from KDE's System Settings panel. This problem is also present on Linux.

Linux

* PixInsight is not compatible with the open-source Nouveau graphics driver. If your distribution is using Nouveau, you must remove and replace it with the proprietary Nvidia driver in order to use PixInsight.

* The same problem with KDE's morphing popups desktop effect exists on FreeBSD and Linux (see above).

macOS

* On Apple laptops with dual graphics cards, the integrated Intel graphics driver causes problems with recent versions of PixInsight. If you use one of these laptops, we recommend you disable the Automatic graphics switching option on System Preferences (Energy Saver section) before running the PixInsight core application. Otherwise you may experience some interface usability problems.

Windows

* There are problems with Intel graphics drivers on machines where the only graphics card available is an integrated Intel HD graphics GPU. This mainly affects laptops. This does not mean that PixInsight cannot work on these machines, just that you can expect screen rendering and usability issues. Unfortunately, there is nothing I can do to solve these problems, for now.

* On some laptops with nonstandard screen resolutions, such as the Microsoft Surface, PixInsight cannot find valid screen scaling factors automatically. On these machines, you may have to run the PixInsight core application with the --ui-scaling command line argument. For example, to apply a UI scaling factor of 2:

C:\Program Files\PixInsight\bin\PixInsight.exe --ui-scaling=2

You only have to do this once, since the scaling factor will be stored in application preferences automatically. Of course, if you already are using version 1.8.6 and the interface looks correct, then you don't need to do this; this is only required for new installations.

* The Export as PDF feature of the integrated web browser component does not work on Windows. I'll try to fix this in a future release.


Recommended Platforms and Supported Operating Systems

The reference implementations of PixInsight 1.8.8, where you may expect the maximum performance and best user experience, are the FreeBSD and Linux versions. Currently our primary development platforms are:

- Kubuntu Linux 18.04 LTS
- FreeBSD 12.0

On FreeBSD and Linux we strongly recommend running PixInsight on the KDE Plasma desktop environment. PixInsight 1.8.8 has been tested on the following Linux distributions:

Kubuntu 18.04 LTS
Kubuntu 19.04
Fedora 29
Fedora 30
Linux Mint 19.1

PixInsight 1.8.8 for Linux requires GLIBC 2.27 or a newer version.

PixInsight 1.8.8 for macOS is only compatible with macOS 10.12, 10.13, 10.14 and 10.15. Older versions are not supported. Our installation packages have been notarized by Apple.

PixInsight 1.8.8 for Windows is compatible with Windows 10 exclusively. Windows 8.1, Windows 7 and older versions are not supported. The application may run well on Windows 7, but there is no guarantee.


Hardware Requirements: Breaking Changes

The FreeBSD, Linux and macOS versions of PixInsight 1.8.7 and 1.8.8 require a processor with SSE4.2 instruction support.


Short-Term Development Lines

During the next months we plan on releasing the following new products on the current PixInsight 1.8 platform:

* MosaicGenerator. An astrometry based automatic mosaic construction tool.

* DynamicBackground. An improved background modeling/correction tool, which will be the successor to the current DBE tool.

* A new tool (name still unknown) for astrometry based measurements, integrated with the core application.

* A new high-performance format for storage of large star databases. The Gaia DR2 catalog will be available for download as one of these databases. This will greatly improve all of our astrometry based tools, such as the ImageSolver script and the new MosaicGenerator tool, which will be able to work with local files without the limitations imposed by online services.


___________________

Thank you for your attention and continued support. Enjoy!
 

wvanreeven

Well-known member
May 30, 2017
83
0
Thanks very much for this update Juan. One minor comment. The file startup.scp still states as the bottom

// ----------------------------------------------------------------------------
// /PixInsight/etc/startup/startup.scp
// Released 2018 OCT 14

Perhaps this could be updated with each new version?


Wouter
 

Juan Conejero

PTeam Member
Sep 2, 2004
7,364
58
56
Valencia, Spain
pixinsight.com
Hi Wouter,

Thank you!

wvanreeven said:
// Released 2018 OCT 14

Perhaps this could be updated with each new version?
Generally, files pertaining to the standard PixInsight distribution are not modified unless their contents change in a new version. The startup.scp file was last updated with version 1.8.5 of PixInsight, and has not been changed in newer versions.
 

KenM

Well-known member
Dec 25, 2016
47
0
47
Belgium
Nice to see improved multi core support, my next PC before the end of the year will be 16 core or higher. :) (of course it's Zen2)

edit:  this in mine top score in windows ????: https://pixinsight.com/benchmark/benchmark-report.php?sn=8W4EV00Y4IXSPQSFOY1OL3HCV3K3CK61
Linux version of dual boot system coming later. (because Windows is slower)

Juan Conejero said:
* A new tool (name still unknown) for astrometry based measurements, integrated with the core application.
Does this include measuring magnitude of variable stars?
 

pbkoden

New member
Jun 1, 2016
3
0
* PixInsight is not compatible with the open-source Nouveau graphics driver. If your distribution is using Nouveau, you must remove and replace it with the proprietary Nvidia driver in order to use PixInsight.
Does Pixinsight on Linux support AMD graphics cards under Linux?
 

Juan Conejero

PTeam Member
Sep 2, 2004
7,364
58
56
Valencia, Spain
pixinsight.com
pbkoden said:
* PixInsight is not compatible with the open-source Nouveau graphics driver. If your distribution is using Nouveau, you must remove and replace it with the proprietary Nvidia driver in order to use PixInsight.
Does Pixinsight on Linux support AMD graphics cards under Linux?
Yes, absolutely. You should have no problems with AMD graphics cards on any supported platform.
 

KenR

New member
Aug 25, 2018
4
0
Nice features in this release! Thanks!

I get an error using the new real time preview window. To reproduce this, open an image, open MLT, click the real time preview button, then after the preview appears close the RTP window, then close MLT, then close the image and I get the error. No need to do anything with MLT.
 

Attachments

Hi Juan, congratulations for the new release!!!

KenR said:
I get an error using the new real time preview window. To reproduce this, open an image, open MLT, click the real time preview button, then after the preview appears close the RTP window, then close MLT, then close the image and I get the error. No need to do anything with MLT.
I was able to reproduce it in Linux

Saludos, Alejandro.
 

Ghsmith45

Well-known member
Mar 3, 2012
922
4
Sydney, Australia
KenR said:
Nice features in this release! Thanks!

I get an error using the new real time preview window. To reproduce this, open an image, open MLT, click the real time preview button, then after the preview appears close the RTP window, then close MLT, then close the image and I get the error. No need to do anything with MLT.
I recently got this same error in the same way using windows10
 

LarryC

Well-known member
Sep 29, 2015
60
0
The new features are great, but just as a data point, my benchmark scores dropped ~5% under Windows with my Ryzen 3900X and 1Tb NvMe drive with 10 x RAM drive (16Gb of 32Gb total) swap files (tested to be the fastest for me about 2 months ago).   
 

boelaars

Member
Sep 15, 2018
6
0
Real-time preview zooming! Yesssssssss!

I thank you form the bottom of my heart, and also from my wife's bottom.  :p ;)
 

Juan Conejero

PTeam Member
Sep 2, 2004
7,364
58
56
Valencia, Spain
pixinsight.com
KenR said:
Nice features in this release! Thanks!

I get an error using the new real time preview window. To reproduce this, open an image, open MLT, click the real time preview button, then after the preview appears close the RTP window, then close MLT, then close the image and I get the error. No need to do anything with MLT.
This is a small bug that I am fixing right now. The warning message is exaggerated in this case; just ignore it. I'll release an update with this bug fixed ASAP. Sorry for the trouble.
 

mschuster

PTeam Member
Nov 27, 2011
1,126
3
San Francisco
Juan, thank you for ESD. I tried it on a couple of my projects (darks, flats, and lights). It is working really well. I am thinking now that I no longer have to spend time tweaking rejection thresholds, as ESD appears to get things right automatically.

Here is an example on a 33 frame, 40 minute Ha exposure set. ESD gets very close to my trial and error optimized thresholds, a bit less aggressive which IMO is a good thing actually (ie SNR++). A couple star core differences stand out, but on investigation this is a non-issue for me.

My optimized Sigma Clipping:



ESD with default parameters:


 

Juan Conejero

PTeam Member
Sep 2, 2004
7,364
58
56
Valencia, Spain
pixinsight.com
KenR said:
Nice features in this release! Thanks!

I get an error using the new real time preview window. To reproduce this, open an image, open MLT, click the real time preview button, then after the preview appears close the RTP window, then close MLT, then close the image and I get the error. No need to do anything with MLT.
Just a note to say that I have fixed this bug today (not an easy one). It was a small memory deallocation error with a tiny memory leak of 32 bytes per image. I'll release an update to the core application tomorrow for all platforms. Thank you for your patience.
 

Juan Conejero

PTeam Member
Sep 2, 2004
7,364
58
56
Valencia, Spain
pixinsight.com
Hi Mike,

mschuster said:
Juan, thank you for ESD. I tried it on a couple of my projects (darks, flats, and lights). It is working really well. I am thinking now that I no longer have to spend time tweaking rejection thresholds, as ESD appears to get things right automatically.
Thank you! That's a very nice example, and I am glad to know it is working so well for you.

ESD rejection is quite different from the rest of rejection algorithms in several important aspects. One of them is the fact that it does not apply fixed rejection thresholds, as sigma clipping and its variants do. ESD applies different thresholds adaptively (known as critical values) as a function of the sample size, as new outliers are being detected and removed from the sample. Another important difference is the use of a Student's t distribution, instead of a normal distribution, which is also adapted to the sample size through the distribution's degrees of freedom parameter. I plan on implementing a Studentized variant of the linear fit clipping algorithm, which I hope will work better than the current one. We have a lot of experimentation work ahead here.
 

mijoschille

Active member
Mar 9, 2017
40
0
Northwest IL
Juan,
Thank you and congratulations on the release of 1.8.8. The updates and new features are what make PI the absolute best astro-processing software available - period. Of course I know I'm preaching to the choir here...

Short term, I'm looking forward to using ESD and the new R-T P functionality.

Mike