New Tool Released: SpectrophotometricColorCalibration (SPCC)

Juan Conejero

PixInsight Staff
Staff member
Hi all,

Today we are proud to announce a crucial milestone in the history of PixInsight: the release of SpectrophotometricColorCalibration (SPCC), a new tool that will change how we understand color in astrophotography.

Gaia Data Release 3 (Gaia DR3) was released on June 13, 2022. This Gaia release includes mean BP/RP spectra for more than 219 million point sources. This is the first massive, accurate spectrophotometric catalog in the history of astronomy, with unprecedented homogeneity and consistency thanks to the fact that it is based on measurements made from space, without the varying conditions of the atmosphere. Thanks to this vast database, the new SPCC tool implements color calibration by reproducing the exact characteristics of the equipment we use to acquire our images: the same filter transmission curves, the same sensor quantum efficiency curves, and the precise spectra of the selected white reference. This way, SPCC can outperform PCC (the PhotometricColorCalibration tool) by a 400% increase in precision. SPCC is the first of a series of tools we are developing to exploit the availability of Gaia mean spectrum data.

SPCC-1.jpg


SPCC is now available as an update for PixInsight 1.8.9-1 on all supported platforms. Along with SPCC, we are releasing WeightedBatchPreprocessing (WBPP) 2.5.4 and ImageSolver 5.6.5, along with new versions of our APASS, Gaia, ColorSpaces, ColorCalibration, and StarAlignment tools. The latest versions of these scripts and processes provide new features and improvements necessary to support SPCC.

The algorithm we have implemented in SPCC has been conceived by PTeam member Vicent Peris, who has contributed decisively to its development, along with the rest of our team, composed of Roberto Sartori, Edoardo Luca Radice, Alicia Lozano, and the author of this post. This first version of SPCC results from the intense, strenuous development activity we have been doing during the past five months.

We have authored an extensive article describing SPCC, its methodology, the implemented algorithms, and all the requirements to use it efficiently. We also include practical examples, usage recommendations, and a technical comparison with PCC. The article is now available on our corporate website:


After installing the SPCC update, you can also read this article directly on PixInsight's integrated web browser by selecting Resources > Technical Documents > Spectrophotometry–based Color Calibration in PixInsight from the main menu.

Quoted from the Conclusions section of the above article:

PCC was a temporary solution for us because we were waiting for the release of the Gaia spectrophotometric catalog. We knew this catalog would be the definitive solution for our needs. Now, PCC and APASS will be deprecated in most use cases, and we do think SPCC will be the flexible, precise and accurate tool we'll need in the upcoming years. Therefore, SPCC represents the culmination of nearly ten years of work.

During the following weeks we'll release, on our official YouTube channel, a dedicated series of video tutorials to describe and document adequately the new SPCC tool and everything it represents for PixInsight and astronomical imaging.

We hope you enjoy it. Thank you for your attention.
 
Thank you, it looks promising.

Now downloading Gaia's small catalog, but it seems everyone is on the same thing.... downloading at 130 kb/sec :(
 
Thank you, it looks promising.

Now downloading Gaia's small catalog, but it seems everyone is on the same thing.... downloading at 130 kb/sec :(

Thank you. That's very strange. Our CDN has a huge bandwidth. We've just downloaded these files at 45 MB/s, three in parallel. Should be a temporary issue in your local area networks.
 
Apologies for my ignorance, but are there instructions somewhere for how to download and install the new GAIA database?
 
Thank you. That's very strange. Our CDN has a huge bandwidth. We've just downloaded these files at 45 MB/s, three in parallel. Should be a temporary issue in your local area networks.

Accessing the updates and even the pi website is really slow right now, and errors out quite a bit. I was only able to download 2 of the 3 updates. It's timing out here as well
 
Congratulations Juan and to everyone who worked on this new tool! It looks like a potential major advance in color calibration and it sure sounds like the right way to do the calculation. I can't wait to give it a try.

BTW, the PI site must be getting slammed because I'm am experiencing the same slow response others mentioned here.

John
 
Apologies for my ignorance, but are there instructions somewhere for how to download and install the new GAIA database?

Never mind - I found it on the PI SW Distribution web site.

BTW - It took my 8 attempts to download the new PI updates, most likely due to heavy traffic. Trying to download the GAIA databases now...
 
After you update and then load a project that uses PCC you will see these warning:


** Warning: PhotometricColorCalibration: Skipped unknown process parameter: workingMode

** Warning: PhotometricColorCalibration: Skipped unknown process parameter: redFilterWavelength

** Warning: PhotometricColorCalibration: Skipped unknown process parameter: redFilterBandwidth

** Warning: PhotometricColorCalibration: Skipped unknown process parameter: greenFilterWavelength

** Warning: PhotometricColorCalibration: Skipped unknown process parameter: greenFilterBandwidth

** Warning: PhotometricColorCalibration: Skipped unknown process parameter: blueFilterWavelength

** Warning: PhotometricColorCalibration: Skipped unknown process parameter: blueFilterBandwidth

But don't think it is a big issue
 
I noticed this tool uses the DR3/SP database, while the ImageSolver script uses DR3, EDR3 or DR2. In the past I've had issues with different tools using different databases. Can the script be updated to use DR3/SP also?
 
Are there any comparisons available of the image results from applying SPCC vs. PCC to the same data?
 
After you update and then load a project that uses PCC you will see these warning:


** Warning: PhotometricColorCalibration: Skipped unknown process parameter: workingMode

** Warning: PhotometricColorCalibration: Skipped unknown process parameter: redFilterWavelength

** Warning: PhotometricColorCalibration: Skipped unknown process parameter: redFilterBandwidth

** Warning: PhotometricColorCalibration: Skipped unknown process parameter: greenFilterWavelength

** Warning: PhotometricColorCalibration: Skipped unknown process parameter: greenFilterBandwidth

** Warning: PhotometricColorCalibration: Skipped unknown process parameter: blueFilterWavelength

** Warning: PhotometricColorCalibration: Skipped unknown process parameter: blueFilterBandwidth

But don't think it is a big issue

It is normal, along with SPCC release, PCC had a major overhaul too, as you can read here.

These warning refers to the Narrowband mode that is disappeared in PCC and now is only available in SPCC.
 
I noticed this tool uses the DR3/SP database, while the ImageSolver script uses DR3, EDR3 or DR2. In the past I've had issues with different tools using different databases. Can the script be updated to use DR3/SP also?

What's the difference between DR3 and DR3/SP


DR2 and EDR3 database are now considered obsolete.

the current official database for astrometry is DR3 that counts more than 1.800.000.000 sources.

DR3/SP is a subset of DR3 containing more than 220.000.000 sources with spectroscopic data, this is why, even if DR3/SP contains astrometric data, it cannot be considered a replacement dor DR3: DR3 contains much more sources needed for robust astrometric measurements.
 
This is awesome, and I'm very excited about it. My immediate question is how one might find and convert the filter curves for their own DSLR camera that isn't included in the default set?
 
Ok, what am I doing wrong?, I followed the instructions and I get the following error:

Or do I need the full Gaia 3 catalog?, I have only downloaded the small one.



Extracting catalog star spectrum data: done​
2401 catalog sources found.​
* Extracting PSF signal samples​
Fitting 203 stars: done​
Fitting 11 stars: done​
* Integrating spectrum data​
Robust linear regression (R/G): 0%​
*** Error: Insufficient data: only 1 sample(s) are available; at least 5 are required.
Reading swap files...​
16172.327 MiB/s​
<* failed *>​
 
Ditto. Sure I'm doing something wrong (Gaia installed on external drive??) , but I get the same error as above. I'm guessing it might have something to do with my file structure but who knows. I had just recently gotten PCC to work reliably (with APASS data), but now I get the same errors with it. Any help appreciated!
 
I'm answering myself, in case anyone else finds it helpful. SPCC works correctly when increasing the value of Saturation threshold in Signal Evalaution.
 
Ok, what am I doing wrong?, I followed the instructions and I get the following error:

Or do I need the full Gaia 3 catalog?, I have only downloaded the small one.



Extracting catalog star spectrum data: done​
2401 catalog sources found.​
* Extracting PSF signal samples​
Fitting 203 stars: done​
Fitting 11 stars: done​
* Integrating spectrum data​
Robust linear regression (R/G): 0%​
*** Error: Insufficient data: only 1 sample(s) are available; at least 5 are required.
Reading swap files...​
16172.327 MiB/s​
<* failed *>​

I'm getting something similar using as OSC image. Should tghis be done on stretched or unstretched data?
 
Back
Top