I am glad to announce an important new feature now available for PixInsight 1.8.8-6 on all supported platforms: The local Gaia DR2 database and its associated GaiaDR2 module and process. The Gaia DR2 database consists of a set of files providing positions, proper motions, parallaxes and three-band magnitudes, as well as data availability and quality flags, for the entire set of 1,692,919,135 sources in the Gaia DR2 catalog. The new GaiaDR2 process acts as a server to provide access to the Gaia DR2 local database. Basically, the GaiaDR2 process allows scripts and modules to perform extremely fast search operations on the entire Gaia DR2 catalog, among other tasks.
High-performance local star databases will play a crucial role in PixInsight from now on, making possible a new generation of astrometry and photometry based tools and scripts, such as astrometric image registration and mosaic construction, photometry-based image normalization and background modeling, and positional astronomy applications, along with the current integration of solar system ephemerides. As you'll see below, the new version 5.4.8 of our ImageSolver script can right now work with the GaiaDR2 process to compute astrometric solutions with unprecedented accuracy and flexibility.
As you probably know, the Gaia Early Data Release 3 catalog (Gaia EDR3) will be publicly available in December. Now that we have built the necessary infrastructure to implement these large catalogs as local databases efficiently, we'll release a new Gaia EDR3 local database and its associated GaiaEDR3 process very soon after the official Gaia release.
The Gaia DR2 database files are now freely available for download:
*** NOTE - The Gaia DR2 database files released for version 1.8.8-6 of PixInsight are no longer available. The latest version 1.8.8-7 of PixInsight has been released with new databases for the APASS DR10 and DR9 catalogs, plus Gaia EDR3 and DR2, which are now available for download on our software distribution system. See also the official 1.8.8-7 release announcement.
These files have been generated in a new database format that we have developed for fast and efficient access to large astrometric and photometric star catalogs: the XPSD (eXtensible Point Source Database) format. The architecture and organization of XPSD are similar to XISF, but adapted to implement efficient storage and fast search operations on spherical surfaces for huge sets of point sources. As is customary with our recent developments, the definition and implementation of XPSD are open-source, although a formal definition document is not yet available. A technical description of this format and its usage for script and module development in PixInsight is beyond the scope of this announcement. I'll cover all of the associated development topics in a forthcoming document.
Downloading Local Gaia DR2 Database Files
See the table above for direct links to the 16 Gaia DR2 database files. Note that each file except the last one provides about 110 million stars. You normally won't need to download all of the 16 files. You'll just need the files necessary to cover the appropriate range of magnitudes for the images that you acquire with your equipment, or the images for which you want to compute astrometric solutions. For most practical purposes, the first 10 database files should suffice. They allow you to plate solve images up to the 20th magnitude, which is reasonable for focal lengths up to 3000 or 4000 mm. If you only do wide-field work, the first two files provide stars up to 17.5 magnitude, which can be more than sufficient in these cases. At any rate, you should download a set of consecutive database files without gaps.
For good database search performance, you should store these files on SSD storage units, or much better on fast NVMe SSD disks. The XPSD format implementation that we are releasing here has been optimized for multithreaded, indexed random access to database files stored on solid-state drives. Avoid rotational hard disk drives for this task, as they can lead to very poor performance, especially when accessed from parallel execution threads.
The new GaiaDR2 process provides access to the local Gaia DR2 database. It works as a server for scripts and modules, allowing them to perform arbitrary, thread-safe search operations on the entire Gaia DR2 catalog. GaiaDR2 can also be used as a standalone process thanks to its graphical user interface, which you can see above. However, before using GaiaDR2 you have to install a set of database files, as described below.
Installing Local Gaia DR2 Database Files
The first step to use this important new feature in PixInsight is to install the required database files in XPSD format. Open the GaiaDR2 process from Process Explorer, where you'll find it under the StarCatalogs category:
On the GaiaDR2 tool window, click the Preferences button, which has a wrench icon and is located on the bottom control bar of the window. This will open the GaiaDR2 Preferences dialog:
On this dialog, click the Add button to select the required .xpsd files that you have downloaded. When you're done, click the OK button to install the selected database files.
Using the GaiaDR2 Process to Search the Gaia DR2 Catalog
Once you have installed the desired database files, let's see how GaiaDR2 can be used as a standalone process to perform search operations on the entire Gaia DR2 catalog. A search operation requires defining at least the following parameters:
Optionally, you can also define the following parameters:
ImageSolver version 5.4.8
The new version 5.4.8 of the ImageSolver script, written by PTeam member Andrés del Pozo with contributions from the author of this document, supports now the GaiaDR2 process as its preferred source of reference stars to compute astrometric solutions. This version of ImageSolver comes with the following main changes:
The new GaiaDR2 process and ImageSolver 5.4.8, as well as an updated version of the PhotometricColorCalibration tool that uses GaiaDR2 for plate solving, are now available as regular updates on all supported platforms.
Gaia DR2 Credits and Acknowledgments
This work has made use of data from the European Space Agency (ESA) mission Gaia (https://www.cosmos.esa.int/gaia), processed by the Gaia Data Processing and Analysis Consortium (DPAC, https://www.cosmos.esa.int/web/gaia/dpac/consortium). Funding for the DPAC has been provided by national institutions, in particular the institutions participating in the Gaia Multilateral Agreement.
__________
Thank you for you attention and support.
High-performance local star databases will play a crucial role in PixInsight from now on, making possible a new generation of astrometry and photometry based tools and scripts, such as astrometric image registration and mosaic construction, photometry-based image normalization and background modeling, and positional astronomy applications, along with the current integration of solar system ephemerides. As you'll see below, the new version 5.4.8 of our ImageSolver script can right now work with the GaiaDR2 process to compute astrometric solutions with unprecedented accuracy and flexibility.
As you probably know, the Gaia Early Data Release 3 catalog (Gaia EDR3) will be publicly available in December. Now that we have built the necessary infrastructure to implement these large catalogs as local databases efficiently, we'll release a new Gaia EDR3 local database and its associated GaiaEDR3 process very soon after the official Gaia release.
*** NOTE - The Gaia DR2 database files released for version 1.8.8-6 of PixInsight are no longer available. The latest version 1.8.8-7 of PixInsight has been released with new databases for the APASS DR10 and DR9 catalogs, plus Gaia EDR3 and DR2, which are now available for download on our software distribution system. See also the official 1.8.8-7 release announcement.
These files have been generated in a new database format that we have developed for fast and efficient access to large astrometric and photometric star catalogs: the XPSD (eXtensible Point Source Database) format. The architecture and organization of XPSD are similar to XISF, but adapted to implement efficient storage and fast search operations on spherical surfaces for huge sets of point sources. As is customary with our recent developments, the definition and implementation of XPSD are open-source, although a formal definition document is not yet available. A technical description of this format and its usage for script and module development in PixInsight is beyond the scope of this announcement. I'll cover all of the associated development topics in a forthcoming document.
Downloading Local Gaia DR2 Database Files
See the table above for direct links to the 16 Gaia DR2 database files. Note that each file except the last one provides about 110 million stars. You normally won't need to download all of the 16 files. You'll just need the files necessary to cover the appropriate range of magnitudes for the images that you acquire with your equipment, or the images for which you want to compute astrometric solutions. For most practical purposes, the first 10 database files should suffice. They allow you to plate solve images up to the 20th magnitude, which is reasonable for focal lengths up to 3000 or 4000 mm. If you only do wide-field work, the first two files provide stars up to 17.5 magnitude, which can be more than sufficient in these cases. At any rate, you should download a set of consecutive database files without gaps.
For good database search performance, you should store these files on SSD storage units, or much better on fast NVMe SSD disks. The XPSD format implementation that we are releasing here has been optimized for multithreaded, indexed random access to database files stored on solid-state drives. Avoid rotational hard disk drives for this task, as they can lead to very poor performance, especially when accessed from parallel execution threads.
The GaiaDR2 ProcessThe new GaiaDR2 process provides access to the local Gaia DR2 database. It works as a server for scripts and modules, allowing them to perform arbitrary, thread-safe search operations on the entire Gaia DR2 catalog. GaiaDR2 can also be used as a standalone process thanks to its graphical user interface, which you can see above. However, before using GaiaDR2 you have to install a set of database files, as described below.
Installing Local Gaia DR2 Database Files
The first step to use this important new feature in PixInsight is to install the required database files in XPSD format. Open the GaiaDR2 process from Process Explorer, where you'll find it under the StarCatalogs category:
On the GaiaDR2 tool window, click the Preferences button, which has a wrench icon and is located on the bottom control bar of the window. This will open the GaiaDR2 Preferences dialog:
On this dialog, click the Add button to select the required .xpsd files that you have downloaded. When you're done, click the OK button to install the selected database files.
Using the GaiaDR2 Process to Search the Gaia DR2 Catalog
Once you have installed the desired database files, let's see how GaiaDR2 can be used as a standalone process to perform search operations on the entire Gaia DR2 catalog. A search operation requires defining at least the following parameters:
- The spherical equatorial coordinates of the center of the search region: right ascension and declination. These coordinates can be specified with the Center right ascension and Center declination parameters.
- The radius of the search region. The GaiaDR2 process performs search operations on 'circular' regions (projected on the celestial sphere). The Search radius parameter allows you to define this value in degrees.
- The range of magnitudes for the objects that will be included in the search result. This range can be defined with the Low magnitude limit and High magnitude limit parameters. Obviously, the highest possible magnitude limit will be imposed by the database files that are currently installed.
Optionally, you can also define the following parameters:
- Inclusion flags. If non-zero, only sources with all of these flags set will be included in the search result. A number of flags (or individual bits) are available for each source stored in the database. These flags provide important information on the data available for each star—for example, not all stars have proper motions, or R/B magnitudes—, as well as information on the quality of astrometric and photometric data. Click the Define inclusion flags tool button (the icon with two check boxes) to define these flags easily on a dedicated dialog:
- Exclusion flags. Sources with any of these flags set will not be included in the search result. You can click the Define exclusion flags button to define these flags on the same dialog shown above.
- Source count limit. This is the maximum number of stars that will be included in the search result. You can set it to zero to only count the objects found, without storing or reporting them. The default value is sufficiently large as to not impose any practical limit.
ImageSolver version 5.4.8
- Added support for local XPSD server catalogs, with the corresponding changes to internal astrometry processes and graphical user interface elements.
- New GaiaDR2_XPSD catalog, which interfaces to the new GaiaDR2 process (local Gaia DR2 XPSD server).
- New automatic magnitude limit adaptive calculation algorithm based on fast XPSD search operations. This algorithm is always used when a valid XPSD server is available and the Automatic limit magnitude option is selected.
- New automatic catalog selection strategy. When a local XPSD server is available, the corresponding catalog (currently GaiaDR2_XPSD) is always selected when the magnitude limit is >= 7 and the Automatic catalog option is selected, irrespective of the field of view.
- New Restrict to high-quality stars option (Advanced Parameters), which is only applicable to local XPSD server catalogs. When this option is selected, only stars with high-quality astrometric parameters (positions, proper motions and parallaxes) are used in the astrometric solution. Gaia stars without proper motions are always excluded by the new GaiaDR2_XPSD catalog.
- The online GaiaDR2 catalog applies now parallax corrections to compute space motion vectors, as the new GaiaDR2_XPSD catalog already does.
- Removed support for the .bin StarGenerator database, which is now obsolete and no longer necessary.
The new GaiaDR2 process and ImageSolver 5.4.8, as well as an updated version of the PhotometricColorCalibration tool that uses GaiaDR2 for plate solving, are now available as regular updates on all supported platforms.
Gaia DR2 Credits and Acknowledgments
This work has made use of data from the European Space Agency (ESA) mission Gaia (https://www.cosmos.esa.int/gaia), processed by the Gaia Data Processing and Analysis Consortium (DPAC, https://www.cosmos.esa.int/web/gaia/dpac/consortium). Funding for the DPAC has been provided by national institutions, in particular the institutions participating in the Gaia Multilateral Agreement.
- Gaia Collaboration et al. (2016): Description of the Gaia mission (spacecraft, instruments, survey and measurement principles, and operations)
- Gaia Collaboration et al. (2018b): Summary of the contents and survey properties.
__________
Thank you for you attention and support.
Last edited: