New Tool Released: SpectrophotometricColorCalibration (SPCC)

Juan Conejero said:
Hi all,

Today we are proud to announce a crucial milestone in the history of PixInsight: the release of SpectrophotometricColorCalibration (SPCC), a new tool that will change how we understand color in astrophotography.

Gaia Data Release 3 (Gaia DR3) was released on June 13, 2022. This Gaia release includes mean BP/RP spectra for more than 219 million point sources. This is the first massive, accurate spectrophotometric catalog in the history of astronomy, with unprecedented homogeneity and consistency thanks to the fact that it is based on measurements made from space, without the varying conditions of the atmosphere. Thanks to this vast database, the new SPCC tool implements color calibration by reproducing the exact characteristics of the equipment we use to acquire our images: the same filter transmission curves, the same sensor quantum efficiency curves, and the precise spectra of the selected white reference. This way, SPCC can outperform PCC (the PhotometricColorCalibration tool) by a 400% increase in precision. SPCC is the first of a series of tools we are developing to exploit the availability of Gaia mean spectrum data.

View attachment 16495

SPCC is now available as an update for PixInsight 1.8.9-1 on all supported platforms. Along with SPCC, we are releasing WeightedBatchPreprocessing (WBPP) 2.5.4 and ImageSolver 5.6.5, along with new versions of our APASS, Gaia, ColorSpaces, ColorCalibration, and StarAlignment tools. The latest versions of these scripts and processes provide new features and improvements necessary to support SPCC.

The algorithm we have implemented in SPCC has been conceived by PTeam member Vicent Peris, who has contributed decisively to its development, along with the rest of our team, composed of Roberto Sartori, Edoardo Luca Radice, Alicia Lozano, and the author of this post. This first version of SPCC results from the intense, strenuous development activity we have been doing during the past five months.

We have authored an extensive article describing SPCC, its methodology, the implemented algorithms, and all the requirements to use it efficiently. We also include practical examples, usage recommendations, and a technical comparison with PCC. The article is now available on our corporate website:

PixInsight Reference Documentation | Spectrophotometry–based Color Calibration in PixInsight

A description of the new spectrophotometry–based color calibration tools and techniques implemented in PixInsight. Published: 2022 November 21 | Last update: 2022 December 13
pixinsight.com pixinsight.com

After installing the SPCC update, you can also read this article directly on PixInsight's integrated web browser by selecting Resources > Technical Documents > Spectrophotometry–based Color Calibration in PixInsight from the main menu.

Quoted from the Conclusions section of the above article:

PCC was a temporary solution for us because we were waiting for the release of the Gaia spectrophotometric catalog. We knew this catalog would be the definitive solution for our needs. Now, PCC and APASS will be deprecated in most use cases, and we do think SPCC will be the flexible, precise and accurate tool we'll need in the upcoming years. Therefore, SPCC represents the culmination of nearly ten years of work.​

During the following weeks we'll release, on our official YouTube channel, a dedicated series of video tutorials to describe and document adequately the new SPCC tool and everything it represents for PixInsight and astronomical imaging.

We hope you enjoy it. Thank you for your attention.
Click to expand...
Thanks for the new SPCC. However, your server is dramatically slow. I have checked my speed at this end and getting 50mbs.
Anything from 8hrs to "failed - Network error" and this is for only one file gdr3sp-1.0.0-01.xpsd. What will happen to the other 19 files to down load.
All other websites are fine!
An answer on what is the trouble and when it will be fixed?
 
The default S/N was set at 40. For a Rayleight Rice distribution (steady target in gaussian noise) I've always used 13db for detection (Pd ~.9). Set the S/N threshold here to 10 and it ran fine.
 
Hello everybody,

Thank you so much for you interest in the new tool. I'm giving you some quick advices on how to use it:

- Both PCC and SPCC are independent of the astrometric solving process. We decided to move this process to the preprocessing stage where it can be easily integrated and automated with WBPP. This means that in most cases you will be able to forget this work. If you want to apply PCC or SPCC to an image without an astrometric solution, just go to menu SCRIPT > Image Analysis > ImageSolver. Configure approximate coordinates of center, approximate date, focal length and pixel size (divide pixel size by 2 in case you used drizzle x2) and check the Distortion Correction checkbox.

- Both PCC and SPCC will only work with linear images. Don't apply the process after stretching. The problem you solve by raising the Saturation threshold to 1 is because you're applying SPCC to an already stretched image.

- SPCC is not only a valid tool for the specific set of filters and cameras listed in the tool. In any case we advice you to use SPCC instead of PCC. Even in the case you select a different set of filters / sensors you are currently using, you will get always a better color calibration than in PCC. This is mostly thanks to the Gaia spectrophotometric catalog, which is game changer. Don't see you filters / camera listed? Open the Curve Explorer in SPCC and navigate the curve list to find which curves are more similar to the ones you're using.

- We're are already developing new learning resources for you. Be patient, please, we're working hard to have it ready as soon as possible.


I hope you'll enjoy playing with this tool as much as we are.
Best regards,
Vicent.
 
vicent_peris said:
Hello everybody,

- Both PCC and SPCC will only work with linear images. Don't apply the process after stretching. The problem you solve by raising the Saturation threshold to 1 is because you're applying SPCC to an already stretched image.
Click to expand...

Thanks for the answer, but that leads me to a question.

I do mostly narrow band and like to stretch each filter separately to bring out the details of each, then I merge these images together to form the final color image.

Is it better then to form the final image non-linearly, use SPCC and then stretch?

Thank you,
 
My image was linear and reducing the threshold S/N worked for me. Let me just say, one more time, that the analysis, models and coding you folks do is world class. The newest WBPP has been a major game changer for me and enables me to do parametric image analysis to my hearts content. I'm having major fun now that I don't spend a couple hours on preprocessing of each image. I don't like just pushing buttons to see what happens, but sometimes it's better than nothing. Like in SPCC checking out the Filters and Curve Explorer and seeing the filter response curves for my specific filters and being able to verify them. Little things like that go a long way. Attention to detail requires lots of work and I know you folks do the work. Thanks for all your efforts.
Rich Gibbons
 
Hi Juan. I can't wait to test this new feature.
I want to use this occasion however to mention an idea I had some time ago, but that I could not pursuit earlier.
Since we have so much precise data regarding the best calibration coefficients for each star on the image, would it be doable, in your opinion, for SPCC to derive a 2D linear (or more?) model of the weights to best-fit all areas on the image?
This could, maybe, fix both CC and gradients in an extremely precise and scientific way!
 
Hi Luca, thank you!

Luca Amerio said:
Hi Juan. I can't wait to test this new feature.
I want to use this occasion however to mention an idea I had some time ago, but that I could not pursuit earlier.
Since we have so much precise data regarding the best calibration coefficients for each star on the image, would it be doable, in your opinion, for SPCC to derive a 2D linear (or more?) model of the weights to best-fit all areas on the image?
This could, maybe, fix both CC and gradients in an extremely precise and scientific way!
Click to expand...

The answer is yes, although probably not in the way you expect. Star catalogs don't provide information on the local background of each star, so they don't provide any data useful to model additive gradients in the image.

However, Gaia spectra can be used to model multiplicative gradients with unprecedented accuracy and robustness. This means that we can use our Gaia DR3/SP databases to model local variations of normalization scale factors with the LocalNormalization tool, among other exciting possibilities. We already have the algorithms designed and an implementation planned for a future update.
 
vicent_peris said:
...
- SPCC is not only a valid tool for the specific set of filters and cameras listed in the tool. In any case we advice you to use SPCC instead of PCC. Even in the case you select a different set of filters / sensors you are currently using, you will get always a better color calibration than in PCC. This is mostly thanks to the Gaia spectrophotometric catalog, which is game changer. Don't see you filters / camera listed? Open the Curve Explorer in SPCC and navigate the curve list to find which curves are more similar to the ones you're using.
...
Click to expand...

Can we anticipate more cameras to be added to the SPCC tool? If not, you mentioned looking through the Curve Explorer to find similar curves. Is there a good source for that information so we know what curve should match a particular camera that isn't listed?

I have been playing with the SPCC tool since the download was available. I am really impressed with it so far. It would be nice to have plate-solving as part of the tool.

A good addition to PixInsight overall!

-Crimsus
 
Hi Crimsus,

Thank you!

Crimsus said:
Can we anticipate more cameras to be added to the SPCC tool? If not, you mentioned looking through the Curve Explorer to find similar curves. Is there a good source for that information so we know what curve should match a particular camera that isn't listed?

I have been playing with the SPCC tool since the download was available. I am really impressed with it so far. It would be nice to have plate-solving as part of the tool.

A good addition to PixInsight overall!

-Crimsus
Click to expand...

Yes, we are working on that and new filter curves will be released as successive updates.

Unfortunately, manufacturers are of little help here, since in most cases they refuse to publish the actual transmission curves of the filters they produce. I mean real numeric data, not pretty pictures in their advertising material (which are often inaccurate and only show curves for a small part of the required range of wavelengths).

In a forthcoming document, we'll show how we have been analyzing the spectral response of several filters very accurately with relatively simple and cheap equipment.
 
Marcelofig said:
Thanks for the answer, but that leads me to a question.

I do mostly narrow band and like to stretch each filter separately to bring out the details of each, then I merge these images together to form the final color image.

Is it better then to form the final image non-linearly, use SPCC and then stretch?

Thank you,
Click to expand...

Hi,

For a Hubble's palette, I advice you to use ColorCalibration and BackgroundNeutralization instead. Follow these steps before stretching:

- Create a preview selecting a background area. Configure it as region of interest in BackgroundNeutralization. Apply the background neutralization process.

- Create a preview covering the nebula. The nebula is going to be the white reference; this way, you'll multiply each different emission to have the same brightness in the three components of the nebula. In ColorCalibration, configure this preview as the region of interest for the white reference. Uncheck the Structure detection checkbox. Configure the background region of interest as the same preview you used for BackgroundNeutralization. Apply the ColorCalibration process.

Readjust STF by linking the RGB channels and pressing again the Auto Stretch button. Magic, you won't need to rescue the S-II emission areas. :)

You can then proceed with stretching.


Best regards.
 
Can anyone remind me where I have to store the GAIA database? I've completely forgotten and I can't find it in any of the usual file locations. Thanks
 
StuartT said:
Can anyone remind me where I have to store the GAIA database? I've completely forgotten and I can't find it in any of the usual file locations. Thanks
Click to expand...

Hi,

You should store in any location in your drive. Then, go to the Gaia tool (File Explorer > Astrometry category > Gaia). Press the Preferences button (bottom right in the tool interface), then select the DR3/SP data release. Now you can select the database files from your drive.
 
vicent_peris said:
Hi,

You should store in any location in your drive. Then, go to the Gaia tool (File Explorer > Astrometry category > Gaia). Press the Preferences button (bottom right in the tool interface), then select the DR3/SP data release. Now you can select the database files from your drive.
Click to expand...
Thanks. Currently I have these installed. Can I delete those now?
1669113497356.png
 
StuartT said:
Thanks. Currently I have these installed. Can I delete those now?
View attachment 16520
Click to expand...

APASS is used in PCC, to preserve it depends whether you want to use PCC or not. Gaia DR3 is used for the plate solving; you should preserve this database because it has 1.8 billion stars, much more than DR3/SP (220 million). We cannot use DR3/SP for plate solving for this reason.
 
With the other catalogs used with the Gaia process, the instructions were to download "just need the files necessary to cover the appropriate range of magnitudes", for example, "the first 10 database files". Is that true here, too, or is the small catalog the only valid subset of the full DR3/SP catalog?
 
bulrichl said:
If you intend to never use PCC any more, you can delete the APASS database files.

In my view, the gedr3 files are not necessary any more if gdr3 files are installed.

Bernd
Click to expand...

Yes, sorry. Actually, if you want to use SPCC, you need only DR3 and DR3/SP.
 
Also, can you please help to quantify what the boundary is between "normally sufficient for calibrating wide to moderate field images" and "necessary for the calibration of narrow field images" in terms of field of view?
 
The range of covered field of view is specified in the download area of the specific database files in Software Distribution.

Bernd
 
You must log in or register to reply here.
Back
Top