New in PixInsight 1.8.8-6: Gaia DR2 Local Database

Juan Conejero

PixInsight Staff
Sep 2, 2004
8,283
408
57
Valencia, Spain
pixinsight.com
I am glad to announce an important new feature now available for PixInsight 1.8.8-6 on all supported platforms: The local Gaia DR2 database and its associated GaiaDR2 module and process. The Gaia DR2 database consists of a set of files providing positions, proper motions, parallaxes and three-band magnitudes, as well as data availability and quality flags, for the entire set of 1,692,919,135 sources in the Gaia DR2 catalog. The new GaiaDR2 process acts as a server to provide access to the Gaia DR2 local database. Basically, the GaiaDR2 process allows scripts and modules to perform extremely fast search operations on the entire Gaia DR2 catalog, among other tasks.

High-performance local star databases will play a crucial role in PixInsight from now on, making possible a new generation of astrometry and photometry based tools and scripts, such as astrometric image registration and mosaic construction, photometry-based image normalization and background modeling, and positional astronomy applications, along with the current integration of solar system ephemerides. As you'll see below, the new version 5.4.8 of our ImageSolver script can right now work with the GaiaDR2 process to compute astrometric solutions with unprecedented accuracy and flexibility.

As you probably know, the Gaia Early Data Release 3 catalog (Gaia EDR3) will be publicly available in December. Now that we have built the necessary infrastructure to implement these large catalogs as local databases efficiently, we'll release a new Gaia EDR3 local database and its associated GaiaEDR3 process very soon after the official Gaia release.

The Gaia DR2 database files are now freely available for download:

*** NOTE - The Gaia DR2 database files released for version 1.8.8-6 of PixInsight are no longer available. The latest version 1.8.8-7 of PixInsight has been released with new databases for the APASS DR10 and DR9 catalogs, plus Gaia EDR3 and DR2, which are now available for download on our software distribution system. See also the official 1.8.8-7 release announcement.

These files have been generated in a new database format that we have developed for fast and efficient access to large astrometric and photometric star catalogs: the XPSD (eXtensible Point Source Database) format. The architecture and organization of XPSD are similar to XISF, but adapted to implement efficient storage and fast search operations on spherical surfaces for huge sets of point sources. As is customary with our recent developments, the definition and implementation of XPSD are open-source, although a formal definition document is not yet available. A technical description of this format and its usage for script and module development in PixInsight is beyond the scope of this announcement. I'll cover all of the associated development topics in a forthcoming document.


Downloading Local Gaia DR2 Database Files

See the table above for direct links to the 16 Gaia DR2 database files. Note that each file except the last one provides about 110 million stars. You normally won't need to download all of the 16 files. You'll just need the files necessary to cover the appropriate range of magnitudes for the images that you acquire with your equipment, or the images for which you want to compute astrometric solutions. For most practical purposes, the first 10 database files should suffice. They allow you to plate solve images up to the 20th magnitude, which is reasonable for focal lengths up to 3000 or 4000 mm. If you only do wide-field work, the first two files provide stars up to 17.5 magnitude, which can be more than sufficient in these cases. At any rate, you should download a set of consecutive database files without gaps.

For good database search performance, you should store these files on SSD storage units, or much better on fast NVMe SSD disks. The XPSD format implementation that we are releasing here has been optimized for multithreaded, indexed random access to database files stored on solid-state drives. Avoid rotational hard disk drives for this task, as they can lead to very poor performance, especially when accessed from parallel execution threads.

The GaiaDR2 Process

GaiaDR2-process.png

The new GaiaDR2 process provides access to the local Gaia DR2 database. It works as a server for scripts and modules, allowing them to perform arbitrary, thread-safe search operations on the entire Gaia DR2 catalog. GaiaDR2 can also be used as a standalone process thanks to its graphical user interface, which you can see above. However, before using GaiaDR2 you have to install a set of database files, as described below.


Installing Local Gaia DR2 Database Files

The first step to use this important new feature in PixInsight is to install the required database files in XPSD format. Open the GaiaDR2 process from Process Explorer, where you'll find it under the StarCatalogs category:

GaiaDR2-process-explorer.png

On the GaiaDR2 tool window, click the Preferences button, which has a wrench icon and is located on the bottom control bar of the window. This will open the GaiaDR2 Preferences dialog:

GaiaDR2-preferences.png

On this dialog, click the Add button to select the required .xpsd files that you have downloaded. When you're done, click the OK button to install the selected database files.


Using the GaiaDR2 Process to Search the Gaia DR2 Catalog

Once you have installed the desired database files, let's see how GaiaDR2 can be used as a standalone process to perform search operations on the entire Gaia DR2 catalog. A search operation requires defining at least the following parameters:
  • The spherical equatorial coordinates of the center of the search region: right ascension and declination. These coordinates can be specified with the Center right ascension and Center declination parameters.

  • The radius of the search region. The GaiaDR2 process performs search operations on 'circular' regions (projected on the celestial sphere). The Search radius parameter allows you to define this value in degrees.

  • The range of magnitudes for the objects that will be included in the search result. This range can be defined with the Low magnitude limit and High magnitude limit parameters. Obviously, the highest possible magnitude limit will be imposed by the database files that are currently installed.
Be aware that if you define a large search radius and a wide range of magnitudes (say for example, a search radius of 10 degrees and a high magnitude limit of 20 on dense regions of the sky), the search operation can generate a huge result with data for many millions of stars.

Optionally, you can also define the following parameters:
  • Inclusion flags. If non-zero, only sources with all of these flags set will be included in the search result. A number of flags (or individual bits) are available for each source stored in the database. These flags provide important information on the data available for each star—for example, not all stars have proper motions, or R/B magnitudes—, as well as information on the quality of astrometric and photometric data. Click the Define inclusion flags tool button (the icon with two check boxes) to define these flags easily on a dedicated dialog:

inclusion-flags.png

  • Exclusion flags. Sources with any of these flags set will not be included in the search result. You can click the Define exclusion flags button to define these flags on the same dialog shown above.

  • Source count limit. This is the maximum number of stars that will be included in the search result. You can set it to zero to only count the objects found, without storing or reporting them. The default value is sufficiently large as to not impose any practical limit.
When used as a standalone process to perform a database search, GaiaDR2 writes information on the console and creates an output plain text file with information generated from the search result. Output files can be generated in standard CSV format, suitable for automated applications, or in tabular formats that can be selected to write coordinates either as scalars in degrees, or using compound angular representations where right ascension is represented as separate hours, minutes and seconds, and declination as degrees, arcminutes and arcseconds. In all cases output text files can contain optional headers, which you can control with the Text headers option. The rest of output options available are self-explanatory. Here is an example of output text file in tabular format, shown on the Script Editor window:

script-editor.png


ImageSolver version 5.4.8

ImageSolver.png
The new version 5.4.8 of the ImageSolver script, written by PTeam member Andrés del Pozo with contributions from the author of this document, supports now the GaiaDR2 process as its preferred source of reference stars to compute astrometric solutions. This version of ImageSolver comes with the following main changes:
  • Added support for local XPSD server catalogs, with the corresponding changes to internal astrometry processes and graphical user interface elements.

  • New GaiaDR2_XPSD catalog, which interfaces to the new GaiaDR2 process (local Gaia DR2 XPSD server).

  • New automatic magnitude limit adaptive calculation algorithm based on fast XPSD search operations. This algorithm is always used when a valid XPSD server is available and the Automatic limit magnitude option is selected.

  • New automatic catalog selection strategy. When a local XPSD server is available, the corresponding catalog (currently GaiaDR2_XPSD) is always selected when the magnitude limit is >= 7 and the Automatic catalog option is selected, irrespective of the field of view.

  • New Restrict to high-quality stars option (Advanced Parameters), which is only applicable to local XPSD server catalogs. When this option is selected, only stars with high-quality astrometric parameters (positions, proper motions and parallaxes) are used in the astrometric solution. Gaia stars without proper motions are always excluded by the new GaiaDR2_XPSD catalog.

  • The online GaiaDR2 catalog applies now parallax corrections to compute space motion vectors, as the new GaiaDR2_XPSD catalog already does.

  • Removed support for the .bin StarGenerator database, which is now obsolete and no longer necessary.
The good news is that the Gaia DR2 local database is now used automatically by default when available. So you just have to configure the GaiaDR2 process, as described above, and select the Automatic catalog and Automatic limit magnitude options in ImageSolver; everything else is optimized automatically. The accuracy and versatility of our thin-plate based astrometric solutions with distortion corrections are now spectacular, thanks to the availability of the entire Gaia DR2 catalog as a local database and the possibility to perform unlimited catalog search operations in milliseconds. This is an important step forward.

The new GaiaDR2 process and ImageSolver 5.4.8, as well as an updated version of the PhotometricColorCalibration tool that uses GaiaDR2 for plate solving, are now available as regular updates on all supported platforms.


Gaia DR2 Credits and Acknowledgments

This work has made use of data from the European Space Agency (ESA) mission Gaia (https://www.cosmos.esa.int/gaia), processed by the Gaia Data Processing and Analysis Consortium (DPAC, https://www.cosmos.esa.int/web/gaia/dpac/consortium). Funding for the DPAC has been provided by national institutions, in particular the institutions participating in the Gaia Multilateral Agreement.
  • Gaia Collaboration et al. (2016): Description of the Gaia mission (spacecraft, instruments, survey and measurement principles, and operations)
  • Gaia Collaboration et al. (2018b): Summary of the contents and survey properties.

__________

Thank you for you attention and support.
 
Last edited:

johnpane

Well-known member
Jan 13, 2015
191
18
58
Wexford, PA, USA
Juan,

This is great!

Please check the database links in the table. I started downloading them but the links all seem to point to the same file.

John
 

tommy_nawratil

Well-known member
Feb 28, 2012
83
5
gracias Juan,

this are great news, big steps forwards are possible. A big reliable database locally, wonderful.
Especially in underdeveloped rural areas where sky is good but net availability is problematic.
Here where I live, the data from Strasbourg often trickle in like casual drizzles of rain,
and working with Image solver and related processes sometimes is just impossible.

I am also very curious on photometry based image normalization and background modeling,
a topics we recently discussed with Hartmut. If it can be put to work, what a huge improvement!
And all the DBE tutorials will be obsolete :giggle: Thanks and wish you success!

Tommy
 
  • Like
Reactions: bilgebay

alanrock

Well-known member
Mar 16, 2018
121
5
I have downloaded the update and a few of the database files, but when I restart PI, I don't have the StarCatalogs option or the GAIADR2 process. What am I missing? I'm on a Mac and on the current version, of course.
 

Juan Conejero

PixInsight Staff
Sep 2, 2004
8,283
408
57
Valencia, Spain
pixinsight.com
Go to Resources > Updates > View Installed Updates. The file '20201013-macosx-x64-1-module.tar.gz' should be in the list of installed packages. If it is not, then either something strange has happened, or you haven't installed all updates available. Select Resources > Updates > Check for Updates.
 
  • Like
Reactions: bilgebay

Juan Conejero

PixInsight Staff
Sep 2, 2004
8,283
408
57
Valencia, Spain
pixinsight.com
Very strange. This cannot be reproduced, and has not been reported before. Select Process > Modules > Install Modules, then click the Search button. The GaiaDR2 module should be found. Then click Install.

Are you using a virus protection software in your Mac?
 

alanrock

Well-known member
Mar 16, 2018
121
5
I went to Process>Modules>Install Modules, and when I clicked on the Search button it DID find it, installed, and it seems ok now. Thanks!
Alan
 
  • Like
Reactions: Maudy

Boogie

Member
May 9, 2019
9
1
Dear Juan,
Thank you for all the effort and good work.
Just wanted to inform:
I experienced the same problem as alanrock, though on a Win10-64 system.
Proposed solution worked here as well.
Thanks.
Johan
 
  • Like
Reactions: alanrock

alanrock

Well-known member
Mar 16, 2018
121
5
BTW, I ran the image solver script followed by a photometric color calibration and it worked perfectly. Thanks again!
Alan
 

bvalente

Well-known member
Feb 17, 2017
66
0
wow those downloaded pretty quickly. loaded it all up, did an ImageSolve, worked great. thanks!
 

alanrock

Well-known member
Mar 16, 2018
121
5
Definitely, this cannot be reproduced under normal working conditions, neither on Windows nor on macOS. Anyway, if you have found a solution that works, that's fine!
Is there a log file of some sort that I can send you to help figure out what happened, since apparently it has happened to someone else as well?
 

Freeomano

Member
Mar 15, 2019
5
0
Question , I am not to computer savvy but you are saying that if you dont have a solid state drive dont download ?
 

rmarrash

Member
Aug 18, 2020
9
1
Question , I am not to computer savvy but you are saying that if you dont have a solid state drive dont download ?
he isn't saying that. He is simply saying that the lookups will be much quicker on a Solid State drive. It's a simple database, it can be stored anywhere
 

mileja

New member
Sep 30, 2020
1
0
Definitely, this cannot be reproduced under normal working conditions, neither on Windows nor on macOS. Anyway, if you have found a solution that works, that's fine!
Good Morning,
I experienced the same as Boogie and alanrock, with the same solution.
So maybe we could help figure it out what went wrong?
Regards,
Dominik
 

timtrice

Well-known member
Aug 21, 2013
47
0
To add, I had the same issue on Ubuntu 20. The fix as mentioned resolved it for me as well.