New Tool Released: SpectrophotometricColorCalibration (SPCC)

For those having trouble with download failing, you might try to do the downloads in batches.

When I try to download all the files simultaneously, it always fails somewhere in the middle. Not only does it fail, but it fails in a way that the downloads cannot simply resume, and I have to start over again (lots of wasted bandwidth).

I found that I could safely kick off 4 downloads at a time with no problems. I would occasionally get some server-side error (usually an HTTP 500 error trying to start the individual download), but once each download starts, it runs to completion. The downside to this, is that I can't just start the downloads and forget them. I have to come back periodically to kick off another 4 downloads. I was able to get all of the DR3 and DR3/SP files in one day without much trouble.

I do like the idea of distributing them via torrent, but I appreciate Juan's reasons for not doing so.
 
It fails, because the download link contains a Token and a timestamp for how long the link is valid.
I was lucky last night and could download the complete set in a little bit more than 2 hours (100Mbit only).

Cheers
Tom
 
tom886 said:
It fails, because the download link contains a Token and a timestamp for how long the link is valid.
I was lucky last night and could download the complete set in a little bit more than 2 hours (100Mbit only).

Cheers
Tom
Click to expand...
Interesting. I have 50Mbit, so 4 hours would have been about right for downloading the entire set - if I could have done it in one shot.

If that's the case, I wonder if they could prevent a significant number of failed downloads by increasing the time for which the token is valid...
 
Good luck, you have to calculate the time, and I am not sure whether the Token stays valid with a modified time.
Just as an example: I don't mean a standard UTC or GMT time, the time is coded like this
Code: 
"expire=1670024196"
.
What I had success with was, going into my Distribution area, started the download, cancelled it and copied the link into a download manager.
One file at a time. It's a little bit more complicated, but it worked out.
Use also the "checksums-sha256" to confirm that the downloaded files didn't get damaged. I used 7-Zip to confirm that there were no errors.

Cheers
Tom
 
Thank you, was not so sure about that anymore.
Fun Fact, my download link was valid for 3 hours.

Cheers
Tom
 
Marcelofig said:
Ok, what am I doing wrong?, I followed the instructions and I get the following error:

Or do I need the full Gaia 3 catalog?, I have only downloaded the small one.



Extracting catalog star spectrum data: done​
​
​
2401 catalog sources found.​
​
​
* Extracting PSF signal samples​
​
Fitting 203 stars: done​
​
Fitting 11 stars: done​
​
​
​
* Integrating spectrum data​
​
​
​
Robust linear regression (R/G): 0%​
​
*** Error: Insufficient data: only 1 sample(s) are available; at least 5 are required.​
​
Reading swap files...​
​
16172.327 MiB/s​
​
<* failed *>​
Click to expand...
I found the same problem the I realised I was using NON-Linear or stretched data. SPCC must use linear data
 
I've been trying to download the DR3 files for at least a week and a half. So far I've only managed to grab 4. It doesn't matter where I start, how many I simultaneously download, even tried one at a time, it keeps failing. Even tried over multiple computers.

I'm in Australia. I don't know if that has anything to do with it.

EDIT: If there's a time token of three hours, my downloads start at about 10 to 14 hours, so I have no hope. :-(
dr3_download.png
 
Last edited:
Don't download all at once, I managed to download the complete set in roughly 2 hours.
What helped me, was using a download manager which used 9 chunks for each file.
You could also run a ping to pixinsight.kxcdn.com to find from which IP address you will get the downloads. Should point to a location in Australia.

Cheers
Tom
 
tom886 said:
Don't download all at once, I managed to download the complete set in roughly 2 hours.
What helped me, was using a download manager which used 9 chunks for each file.
You could also run a ping to pixinsight.kxcdn.com to find from which IP address you will get the downloads. Should point to a location in Australia.

Cheers
Tom
Click to expand...
Thanks Tom. You're right: Pinging a-au00.kxcdn.com

I've been trying one file at a time for days. Only just started trying 4 at a time since a member reported 4 was his limit.

Can you recommend a good download manager? I tried one, but it looked like it only did torrent files.
 
I use jDownloader, pretty universal and worked fine for me.
Started the download in Firefox, cancelled it and copied the download link to jDownloader.

Cheers
Tom
 
Terry Danks said:
Not sure I am posting this in the correct place but it does have to do with my experience to date with SPCC.

First off, I experienced no difficulty implementing it on my PC, nor downloading the complete 60GB database to do so.
Initially, I was delighted as results seemed satisfactory and were not difficult to obtain.

But, when I started experimenting with adding Ha data into RGB galaxy images . . . it seemed the wheels fell off SPCC.

As I was using PM expressions developed by others, I at first blamed my procedure as I admittedly didn't really know what I was doing.

Several attempts adding Ha to the R channel of my M31 data, using different PM expressions, kept turning up a greenish turquoise hue when doing the final combine.

I finally began to suspect it was actually SPCC causing this and resorted to a simple use of the Background Neutralization tool and, bam!, the problem went away and colours were more as they should be. After experimenting further, it was determined this had nothing to do with the PM expressions I was experimenting with. The greenish cast seems to originate with SPCC in this instance. A simple application of BN to the RGB combine shows more appropriate colours than does doing cc with SPCC.
I attach screenshots. The top panel is the SPCC result. The bottom a simple BN application to the RGB combine. All images are linear with STF.

No idea as to the source of this behaviour. My filters are entered correctly in the SPCC pane and the selection of Average Spiral Galaxy for white reference is surely appropriate in this instance?

Have there been any others experiencing "wonky" colours after using PMCC?

PS: I redid SPCC using G2V for white balance and was much more satisfied with the results. But it is surprising that the choice of white refereance should be so critical. I had hoped SPCC would be more, for want of a better word, "objective." It appears, at least in this case, that choosing the white reference is critical to good results. I do not recall PCC ever delivering a "bad" colour correction.
Click to expand...
Hello Terry,
I am just sharing my thought with you but I don't think it is relevant to perform SPCC again AFTER adding Ha data. All the reference of the Red filter you used will be wrong because you add data to the R channel.

You should perform SPCC on the linear RGB channels and then add Ha like you want,
normally there is no reason to perform even background neutralisation after adding Ha data, unless you do it in a way which change the mean background value of the R channel
 
I picked up the updates with the new SPCC process and downloaded the catalogs back when they first came out, but yesterday was the first time I had a chance to try it.

I used some recent data for IC410. I had R, G, B, Ha, OIII and SII data available. I made RGB and HOO color images and used SPCC to calibrate them. I have to say that the result is noticeably better than I typically get with PCC. This is a brilliant new addition to PixInsight.

Normally, I use PixelMath to build a color image from the Ha, OIII and SII, but I wanted to see how this would do. Naturally, SPCC doesn't use the SII data, so I think that I'll use the color calibrated HOO and then blend the SII in afterwards to enhance the red channel for areas with strong SII.

Anyway, thanks very much for this new process. It's awesome!
 
Djibi said:
Hello Terry,
I am just sharing my thought with you but I don't think it is relevant to perform SPCC again AFTER adding Ha data. All the reference of the Red filter you used will be wrong because you add data to the R channel.

You should perform SPCC on the linear RGB channels and then add Ha like you want,
normally there is no reason to perform even background neutralisation after adding Ha data, unless you do it in a way which change the mean background value of the R channel
Click to expand...
I agree. This seems to run against the grain of what doing photometric/spectroscopic color calibration is all about. You're telling SPCC the data is one thing and then giving it something else (and something almost by definition "false").
 
Terry Danks said:
PS: I redid SPCC using G2V for white balance and was much more satisfied with the results. But it is surprising that the choice of white refereance should be so critical. I had hoped SPCC would be more, for want of a better word, "objective." It appears, at least in this case, that choosing the white reference is critical to good results. I do not recall PCC ever delivering a "bad" colour correction.
Click to expand...
From the documentation:
It's also important to point out that color calibration will be more accurate if the white reference (signaled by two cross lines in each graph) is well within the point cloud and in its linear range.
Click to expand...

Regarding the idea of "objective" calibration (which was true before and after SPCC):

In PixInsight, we have always favored a relativistic view of color in astrophotography. This is well described in the book Fotografiar lo invisible, by Vicent Peris:

Under the relativistic perspective, the same natural object does not have a single authentic balance of color, since this color is always relative to its frame of reference. And this frame of reference, in turn, depends on the physical phenomenon that you want to convey in the image. Therefore, color in astronomical photography acquires a meaning in its own right, since in these images we can discuss in a specific and accurate way what the white balance and each of the color hues physically represent.
From this point of view, an image can have multiple color perspectives, depending on the natural content we want to convey.
Click to expand...
 
I just upgraded from trial to commercial license. While I was on the trial license I downloaded and was able to use SPCC. Now on the commercial license I don't see the SPCC process (expected) but can't figure out where to download SPCC from. I don't see it in PixInsight software downloads link. What am I overlooking?
 
1. Check the version, it should be:
PixInsight Standard Edition, Core version 1.8.9-1, build 1556, released 2022-05-18

2. Check whether SPCC resides in 'PROCESS/Color Calibration'.

3. Click in the menu 'RESOURCES/Updates/Check for Updates'. If updates are found, confirm that the updates shall be downloaded. After successful download, you'll have to quit PixInsight. Then restart PixInsight and check whether SPCC is available now.

Bernd
 
bulrichl said:
1. Check the version, it should be:
PixInsight Standard Edition, Core version 1.8.9-1, build 1556, released 2022-05-18

2. Check whether SPCC resides in 'PROCESS/Color Calibration'.

3. Click in the menu 'RESOURCES/Updates/Check for Updates'. If updates are found, confirm that the updates shall be downloaded. After successful download, you'll have to quit PixInsight. Then restart PixInsight and check whether SPCC is available now.

Bernd
Click to expand...
And check that that pix repository URLs in the Manage Repositories dialog say 1.8.9-1 and not 1.8.9-0. In some cases, the repo URLs aren't updating and the 1.8.9-1-compatible updates never get a chance to come through.
 
Last edited:
I don't understand why there is a white balance/reference dropdown.
If I wanted it to be made 'pretty' I wouldn't be doing the calibration in the first place.
from what I know white balance/reference is only in case the light source on a object you photograph isn't what you want it to be, so you change that in post, but never when the light source is what you photograph like in astrophotography.

When calibrated I want them calibrated not be 'post pretty' again.
So what do I need to select to make the white balance an empty action?
 
neverfox said:
And check that that pix repository URLs in the Manage Repositories dialog say 1.8.9-1 and not 1.8.9-0. In some cases, the repo URLs aren't updating and the 1.8.9-1-compatible updates never get a chance to come through.
Click to expand...
I confirmed that I had 1.8.9-1, build 1556, date 2022-05-18. However, the Manage Repositories dialog showed 1.8.9-0 for all four updates. I deleted them and checked for updates and was able to download the 1.8.9-1 repositories. I now have SPCC back. Thanks for the assist! Not sure I would have figured that out on my own.
 
You must log in or register to reply here.
Back
Top