PixInsight Forum

PixInsight => Release Information => Topic started by: Juan Conejero on 2017 May 19 12:29:07

Title: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Juan Conejero on 2017 May 19 12:29:07
Hi everybody,

Further news on the next version of PixInsight. Today we have been working on an exciting new tool: PhotometricColorCalibration (PCC). We have a first working version with really outstanding results that I want to share with you all.

PCC performs automatic plate solving and photometry (both aperture and PSF photometry are selectable) to calculate RGB white balancing factors based on measured star fluxes, with respect to a user-selectable white reference. Photometric data are retrieved online from the APASS survey (https://www.aavso.org/apass) through the VizieR server (http://cdsarc.u-strasbg.fr/) and mirrors. The new PCC tool uses internally the latest versions of the ImageSolver and AperturePhotometry scripts, authored by Andrés del Pozo and Vicent Peris. Vicent is also the author of the internal PCC calculation algorithms. I am writing the C++ implementation, making the most of the new functionality of version 1.8.5 that allows us to execute JavaScript scripts from PCL-based C++ code to write hybrid PixInsight modules. This means that PCC is a regular PixInsight tool, not a script, with all its inherent benefits.

Here are a few examples that Vicent and I have been preparing to show you the kind of results you may expect from PCC.

First a wide-field DSLR image of the Milky Way, courtesy of Georg Viehoever:

(http://forum-images.pixinsight.com/20170519/PCC/MilkyWay-tn.jpg) (http://forum-images.pixinsight.com/20170519/PCC/MilkyWay.png)
Click for a full-size version (http://forum-images.pixinsight.com/20170519/PCC/MilkyWay.png)

On the left, the original RGB debayered image. On the right, the result after applying PhotometricColorCalibration. The good news is that PCC is really easy to use. If your image has approximate center coordinates and image scale metadata (as FITS header keywords), PCC is basically a one-button tool in most cases (or, speaking in pure PixInsight terms, a one-blue-triangle tool). The next example is a two-frame mosaic of the region around NGC 2080, in the Large Magellanic Cloud (data by Vicent Peris):

(http://forum-images.pixinsight.com/20170519/PCC/LMCMosaic-tn.jpg) (http://forum-images.pixinsight.com/20170519/PCC/LMCMosaic.png)
Click for a full-size version (http://forum-images.pixinsight.com/20170519/PCC/LMCMosaic.png)

This is an image of the LBN 552 region acquired with the 1.2 m Zeiss telescope of Calar Alto Observatory:

(http://forum-images.pixinsight.com/20170519/PCC/LBN552-tn.jpg) (http://forum-images.pixinsight.com/20170519/PCC/LBN552.png)
Click for a full-size version (http://forum-images.pixinsight.com/20170519/PCC/LBN552.png)

The original combined RGB image, shown on the top-left, had already been white balanced using the average color of the stars in the frame as white reference. The newly calibrated image with PCC is on the bottom-right of the screenshot. The difference speaks by itself.

The last example is an image of M51, also acquired through the 1.2 m telescope of Calar Alto Observatory:

(http://forum-images.pixinsight.com/20170519/PCC/M51-tn.jpg) (http://forum-images.pixinsight.com/20170519/PCC/M51.png)
Click for a full-size version (http://forum-images.pixinsight.com/20170519/PCC/M51.png)

After applying PCC, the image shown on the right has been stretched nonlinearly with HistogramTransformation, processed with HDRMultiscaleTransform, and its color saturation has been increased with CurvesTransformation.

Besides its underlying high-accuracy astrometric and photometric analysis implementations, the most innovative and powerful feature of our new PCC tool is, in my opinion, the fact that it allows you to select one among numerous predefined, carefully generated white references. By default, PCC applies a white reference based on the average of the characteristic fluxes of Sb, Sc and Sd galaxies. This reference, which has been used in all of the examples shown above, is in our opinion truly representative of the deep space, and hence an unbiased, neutral white reference quite close to our documentary color philosophy. If you want to persist in making common conceptual mistakes, you will be able to use the G2V spectral type as a white reference—but PCC will allow you to select virtually any spectral type, along with several galaxy types, to calibrate the color of your images automatically and accurately in PixInsight.

A huge kudos to Andrés del Pozo and Vicent Peris, who are the authors of the excellent implementations and algorithms behind the new PhotometricColorCalibration tool. Thank you for your continued support and contributions, which make PixInsight an exciting platform in constant evolution.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Harry page on 2017 May 19 13:58:49
Hi Juan

Look very interesting , But "One Button " where's the fun in that  >:D

Harry
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Nocturnal on 2017 May 19 18:57:34
Exciting stuff!
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: RickS on 2017 May 19 19:16:51
I'm really looking forward to this... and also not spending an hour per image using PixelMath to overwrite satellite trails  :)
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Niall Saunders on 2017 May 20 06:13:31
Hi Juan (and, as always, Hi to all those who have made these significant contributions),

Yet again PixInsight raises the bar in the field of astroimage processing, with yet another totally scientific and statistical apporach to the solution of a problem that has, so often, previously been attacked 'artistically' or 'subjectively'.

As Harry says, whilst half the fun of PixInsight is spending hours fiddling with sliders and buttons whilst trying to figure out what combination best suits the image, if this truly is a 'one-blue-triangle' approach, then at least it will give us more time to fiddle with those pesky settings in all the other processes  :)

Thanks again.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Ignacio on 2017 May 20 10:19:44
This is amazing! I was just working on something along these lines, using the AperturePhotometry script!

Now, my motivation was driven by a DSLR issue with color calibration, that stems from the fact that the CFA has no clean frequency cut-offs and there is significant cross-talk between color channels. Proprietary digital development software inside the camera processors, and in software like adobe´s camera raw and dcraw, partly correct this by means of a color calibration matrix that maps the camera raw data onto some standard color space (ie, XYZ). This is camera and illumination dependent. Although one can find sources for such matrix data for different camera models and illuminations,  these are not useful with modded DSLRs in astroimaging situations. Hence, I was working on the idea of using catalog photometric data to determine such matrix via a multilinear regression approach, including pedestals in the model that could take care of background neutralization at the same time:

[RGB]_xyz = M [RGB]_raw + [RGB]_bias

where the elements of the 3x3 matrix M, and 3x1 vector [RGB]_bias, are obtained by fitting the detected data to the catalog photometric data.

Is this something that could be accomplished by the new module, or white balance will be handled in the standard three (RGB) multiplicative factor?

best,
Ignacio
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Herbert_W on 2017 May 20 10:36:41
Hi Juan!

That's an exiting announcement.

Just a question - don't be angry:
"Photometric data are retrieved online from the APASS survey - do you also think of including the SDSS-DR9 catalog?

Best regards!

Herbert, Austria
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Juan Conejero on 2017 May 20 10:41:51
Thank you so much to all.

Quote
But "One Button " where's the fun in that

Don't worry, Harry, the PCC tool will have lots of parameters to control the astrometric solution and the photometric analysis, along with the background reference, so you'll have plenty of numbers to play :) However, the tool will work fine with default parameters in most cases, if the image has the required metadata.

Quote
...not spending an hour per image using PixelMath to overwrite satellite trails

Hi Rick. You'll *never* have to do this again in PixInsight. The new large-scale pixel rejection feature of ImageIntegration works flawlessly. It works so well and is so robust, that you'll be able to integrate a data set simply ignoring all plane and satellite trails, etc. Just as if they didn't exist in virtually all cases.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Juan Conejero on 2017 May 20 10:50:50
Hi Ignacio. Thank you.

Quote
[RGB]_xyz = M [RGB]_raw + [RGB]_bias

The current (first) version of the PCC tool computes scaling factors for the individual RGB components. The mean background is sampled and evaluated through robust statistics from a user-defined region of interest, just as the current BackgroundNeutralization tool does now. However, your idea looks very interesting, and I think it could be implemented without problems. I also think your approach is sound and should work well.

The PCC tool will be released as an open-source product (under PCL license), so it will be available at our open-source GitHub repositories, and hence open to collaborations. Our PCL development framework has all of the necessary resources to implement what you want (and in case it lacks something, we can implement it). This would be an excellent improvement.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Juan Conejero on 2017 May 20 11:15:09
Hi Herbert,

Thank you.

Quote
do you also think of including the SDSS-DR9 catalog?

The new PCC tool uses the latest release of the APASS survey as its source of photometric data by default. However, PCC is very flexible, so you can use basically what you want. You can select any catalog currently supported by the AperturePhotometry script (which, by the way, should be renamed to Photometry IMO, since it now performs both aperture and PSF photometry).

As I have said before, PCC will be released as an open-source module, so it will be open to collaborations from external developers. With the new version 1.8.5 of PixInsight, we'll release almost all of the new tools as open source products, including LocalNormalization (previously announced as FrameAdaptation), SubbandBlending, and of course the new features that will be available in ImageIntegration and DrizzleIntegration, among many others. I am betting very hard on open source releases, as the best way to reinforce the required dynamism on the PixInsight platform.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Harry page on 2017 May 20 11:23:27
Hi Juan

What is SubbandBlending going to be ?  please let it be image blender !!!!!!!!!!!

Harry
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: pfile on 2017 May 20 11:31:30
so what's the timeframe like for release of 1.8.5? i have a boatload of my usual crappy LP data that i have been holding off processing because i'd like to try the new normalization stuff.

rob
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Juan Conejero on 2017 May 20 11:31:45
The final name is still subject to discussion :) It will be a new tool to generate seamless mosaics using, among others, the sub-band blending (SBB) algorithm. SBB was first described, AFAIK, by P.J. Burt and E.H. Adelson in A Multiresolution Spline With Application to Image Mosaics (http://persci.mit.edu/pub_pdfs/spline83.pdf). My implementation introduces significant changes and uses powerful multiscale techniques that we already have implemented in PixInsight, instead of Gaussian pyramids, but follows the same basic idea. SBB is also used with great success in well-known panorama generation applications, such as AutoStitch for example.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Juan Conejero on 2017 May 20 11:46:32
Quote
so what's the timeframe like for release of 1.8.5?

I still don't dare to anticipate a release date. It should happen during the first half of June. Version 1.8.5 is a very complex release, with very significant changes to the platform (as usual, much more and much deeper changes than what the user will see on the surface), many new tools, and many and very important improvements and new features.

For example DrizzleIntegration can now work with CFA monochrome data directly (direct Bayer drizzle), which involves a completely redesigned drizzle data format (the new XML-based XDRZ format, which includes LZ4-compressed rejection maps and local normalization data). Another example of complexity is the possibility to write hybrid modules that mix C++ code with JavaScript scripts (PhotometricColorCalibration is a good example). There is also a new HTML5 painting engine available on the JavaScript and C++ runtimes. etc, etc...

The new version 5.8.0 of Qt is fantastic, but as usual, it generates new problems and incompatibilities with our code base, which I have to fix. I already have a 1.8.5 version working extremely well on Linux, which I am using to concentrate all of my work on the new tools and improvements for now. So the macOS and Windows versions still require a lot of work. So I ask for (even more!) patience; 1.8.5 will be the best, most powerful and most stable version of PixInsight ever.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Andres.Pozo on 2017 May 20 12:04:04
Just a question - don't be angry:
"Photometric data are retrieved online from the APASS survey - do you also think of including the SDSS-DR9 catalog?
Hi,
AperturePhotometry can already use SDSS-R8. Supporting the release 9 should be very easy.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: pfile on 2017 May 20 14:12:19
Quote
so what's the timeframe like for release of 1.8.5?

I still don't dare to anticipate a release date. It should happen during the first half of June. Version 1.8.5 is a very complex release, with very significant changes to the platform (as usual, much more and much deeper changes than what the user will see on the surface), many new tools, and many and very important improvements and new features.


OK, that's enough precision - was just wondering if it was "soon" or late in the year, or what.

rob
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: bob_franke on 2017 May 20 16:58:35
Juan, you said...
This reference, which has been used in all of the examples shown above, is in our opinion truly representative of the deep space, and hence an unbiased, neutral white reference quite close to our documentary color philosophy.

Several years ago you said…
The concepts of "true color" and "natural color" are illusions in deep-sky astrophotography. Such things don't exist. The main reason is that a deep-sky image represents objects far beyond the capabilities of the human vision system.

So which is it, you seem to have change your mind.

Then you said in this announcement…
If you want to persist in making common conceptual mistakes, you will be able to use the G2V spectral type as a white reference—but PCC will allow you to select virtually any spectral type, along with several galaxy types, to calibrate the color of your images automatically and accurately in PixInsight.

I find this statement incredibly arrogant. Who are you to say that your color philosophy is better than anyone else's? Also, you are again stating that PCC provides "accurate" color, which you previously stated does not exist. At least one astrophysicist and many of the best astrophotographers on the planet accept the G2V and/or eXcalibrator methods. eXcalibrator's Linear Regression routine uses stars of multiple colors and gets the same result as the "white-star only" routines.

I am a PixInsight user and believe it is an exceptional and powerful program. But I find the arrogance of the developers and many in the user community to be astounding. The general consensus is that if you don't understand the math... that's your problem. What little help there is, is written at a level only a mathematician can understand. This is not the general astrophotography community.

It is unfortunate that the PixInsight developers are too lazy to write complete and easy to understand documentation. You should not be relying on others to write books and provide tutorials.

Regards,
Bob
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: msmythers on 2017 May 20 21:22:05
Bob

I respect your opinion now respect mine, you come across sounding very mean to me. I suspect to your friends and family you are a very nice man though.

I bought PI almost 5 years ago with my eyes wide open as far as the documentation situation. I still purchased it and don't regret it. I understood when I bought it that the documentation would most likely always remain that way. I am still fine with that because I see great value in my purchase. The PixInsight team has a business model that seems to be working fine for them, you might do it different and I might do it different but it is their business. The users who write tutorials do it because they want to. I believe you write tutorials for others software also so are those developers lazy too?

Also when I purchase software I don't care if the developer 'sounds' arrogant, what their stated opinions are, I care does the software do the job I expect of it. I inspect the goods before purchasing and I decide. If the answer is yes then the developer can 'say' the moon is bright green it doesn't change the fact that the software does what I need it to do. If it doesn't I don't buy it and move on. It's very simple but I am very simple. 



Mike
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: vicent_peris on 2017 May 21 00:09:46
Hello everybody,

Thank you very much for your comments. At a personal level, this tool represents an important milestone to me. I have been working on my own color theory for almost ten years from now. As you may know, my work for the ALHAMBRA survey (http://pixinsight.com/tutorials/alhambra-rgb/index.html) was the key to develop this theory. This theory is based on these simple statements:


As you know, we implemented points 1 to 4 in PixInsight through the ColorCalibration tool. But, up to now, it was not possible to demonstrate point 5 since in most cases you are not able to reference the color to a face-on spiral galaxy. My hypothesis was that a spectral model of an intermediate spiral galaxy could serve as a kind of "universal" white reference in astrophotography.
Up to now, PixInsight hadn't a solution to demonstrate this hypothesis. So I do think this new tool is an important milestone for PixInsight as well. Right now, you have all the freedom to choose your white balance in PixInsight: you can choose a source in your own picture, or you can choose an absolute reference.

This is the aesthetic point of view, which we feel contributes to the artistic development of the user community. It's never arrogant to establish you point of view. We simply don't like a sun-like star as white reference because we think it doesn't represent the nature of the photographed object. Moreover, we understand that we should give the user the freedom to choose his/her own white reference, so we included the G2V spectra, together with almost any kind of star spectral type to make your choice. In the future, we'll be adding more white reference models to the list.


Best regards,
Vicent.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: bob_franke on 2017 May 21 01:14:56
Vicent, I agree that a face-on spiral galaxy makes a nice white reference.

However, what if there is intervening galactic extinction? How do you show a galaxy with its intrinsic color and still correctly display the foreground stars?

I think it's better to show the foreground stars correctly and let the color of the galaxy include the extinction. I call this the café doctrine... that is Color As From Earth. Or probably more correctly, color from Earth orbit. :-)

Also, I am a bit fuzzy with your basic color theory. Can you give us a definition of a "documentary goal"?

Regards,
Bob
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: msmythers on 2017 May 21 01:37:23
Bob

I would like to apologize for the mean comment and others. I was taught better then this by my parents.

I just finished watching the video of the car rampage in Times Square and that brings things like software issues back into perspective.

Again sorry Bob.


Mike
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: vicent_peris on 2017 May 21 01:44:22
Hi Bob,

However, what if there is intervening galactic extinction? How do you show a galaxy with its intrinsic color and still correctly display the foreground stars?

I think it's better to show the foreground stars correctly and let the color of the galaxy include the extinction. I call this the café doctrine... that is Color As From Earth. Or probably more correctly, color from Earth orbit. :-)

We are applying the white point to the picture by measuring the star fluxes. From those fluxes, we calculate the RGB weights that would have an unreddened face-on spiral galaxy. This means that, if your spiral galaxy is reddened by galactic extinction, it will show up in the calibrated picture as a redder galaxy than the spiral galaxy model that we are applying as white reference.

Quote
Also, I am a bit fuzzy with your basic color theory. Can you give us a definition of a "documentary goal"?

Please, take a look at this document. (http://astro-photographer.org/dsa/statement.html)

Best regarfds,
Vicent.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Ignacio on 2017 May 21 08:27:30
Color calibration is always an engaging topic, that can easily bring out passionate views and opinions.
The way I see it (no pun intended), color is about spectral information within a certain bandwidth. So if one is able to piece together a procedure from detection to final reproduction (say, through a display), that reproduces accurately such information, then that is the ideal case. Now, in astrophotography we like to go beyond human vision capabilities so that frequent astrophysical phenomena is not left out (ie, H-alpha emissions). Then the question arises as how to remap such part of the spectrum into the visible range. One possible approach is to compress the spectral information around some middle point in the greens, so that all information is preserved (and could be recovered by an inverse transformation). This, again, is the ideal case.

In practice, we detect color through three components (typically RGB), that result from passing the signal through three different filters before hitting a detector. Such filters and detectors are not standardized among amateurs, and they add their own idiosyncrasies to color registration. In fact, the problem becomes infinite dimensional if different filter/detector profiles are to be taken into account (to match those used in photometric catalogs). On top of this, we like to filter out unwanted additive signals, such as LP and air-glow, and the question of spectral information becomes even more intractable.

What I really like about color calibration using photometric data from different spectral type stars, is that it deals with many of the these problems at the same time, if a rich enough model if fitted.

So, the question remains if a model with three scaling parameters is good enough to recover accurate color balance, given the infinite dimensional nature of the problem. I know for a fact this is not the case with DSLRs (whose filters have significant cross-talk), where at least a matrix transformation is required. But given the unstandardized nature of color filters transmittance, and detectors spectral QE profiles, I wonder... And now that we will have access to hundreds of data points via photometric catalogs, maybe is time to get more ambitious!

best,
Ignacio
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Niall Saunders on 2017 May 21 09:16:40
Quote
1.8.5 will be the best, most powerful and most stable version of PixInsight ever

Not strictly true, Juan - the 'best' version, IMHO, was that original version of PixelInsight LE - it was the pebble that you dropped in the ocean that started the tsunami that we all now know and enjoy today as PixInsight.

Yet, somehow, you constantly seem to be able to amaze us!
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: vicent_peris on 2017 May 21 10:27:09
So, the question remains if a model with three scaling parameters is good enough to recover accurate color balance, given the infinite dimensional nature of the problem. I know for a fact this is not the case with DSLRs (whose filters have significant cross-talk), where at least a matrix transformation is required. But given the unstandardized nature of color filters transmittance, and detectors spectral QE profiles, I wonder... And now that we will have access to hundreds of data points via photometric catalogs, maybe is time to get more ambitious!

Hi,

This is a different problem. With this tool or any other that calibrates a white point in your picture, you multiply the RGB channels by a calculated ratio. This operation neutralizes the white reference, so you're sure that the white reference will be white in your picture. On the other hand, objects differing from that white point will have different colors on each image depending on several characteristics of the optical system and the acquisition conditions: the atmospheric extinction, the QE curve, or the filter transmission curve. This implies a higher order correction that, at this moment, cannot be done because we would need standard stars measured with RGB filters. As of today, all the photometric catalogs are built with photometric filters. So, there is still a long way to go in this field...

Anyway, the main issue is to have a solid white reference in your picture. Now, you have a complete solution in PixInsight since it let's you choose between a white reference relative to your picture (by using ColorlCalibration) or an absolute reference by using photometric models.

Our implementation has some very strong points:

- The calculated astrometry is extremely accurate thanks to the ImageSolver script by Andrés del Pozo. This is far superior to any other astrometry solution because it uses splines to correct geometrical distorsions. Far superior to WCS or any other solution using polynomials (I could show you some examples that are *impossible* to solve by using polynomials).

- The AperturePhotometry script I designed with Andrés is also very powerful and flexible, thus allowing us to implement a great variety of new tools based on photometry. In the next version you'll be able to use PSF photometry as well, that can be very powerful for crowded fields or in images with heavy optical aberrations.

- Trust me, the linear fit algorithm in PixInsight is magic. It is far better than a linear regression, and this tool woudn't work at all without a truly robust linear fitting. The Milky Way and the LMC cases are really difficult to calibrate. We really didn't expect any good result from these two images... And it worked! :)

Beyond this tool, Juan evolved the development platform, so now you can build new modules that use JavaScript scripts. This will be very important in the future. A direct benefit is that you'll be able to use scripts with non modal windows simply by developing the UI in C++.


Best regards,
Vicent.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: akulapanam on 2017 May 21 20:16:10
Thanks for adding this feature!  Color is one of the tougher areas of PixInsight to get good results in.  Will the new DynamicBackground tool be in 1.8.5 too?
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Juan Conejero on 2017 May 22 00:28:24
Thanks for adding this feature!  Color is one of the tougher areas of PixInsight to get good results in.  Will the new DynamicBackground tool be in 1.8.5 too?

DynamicBackground, which will replace DBE with a much more advanced and flexible interactive background modeling tool, will be available during the 1.8.5 cycle. The new tool is already designed but partially implemented, so it still requires work and a lot of testing. I am not sure if I'll be able to include it in the initial 1.8.5 version. I won't include it if that means delaying the release too much; in such case, I'll try to release it as soon as possible as an update, hopefully during June/July.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: copello on 2017 May 22 01:24:32
That is a great new feature and will attract even more people from the scientific community. Please continue
with the great work, even when some seems to come with strange and weird statements:

Quote
bob_franke: "It is unfortunate that the PixInsight developers are too lazy to write complete and easy to understand documentation. You should not be relying on others to write books and provide tutorials."

Please look around in the IT and scientific literature, why are there thousands of books on software such as: Maple, Mathematica, Matlab, ...., Photoshop, Corel Draw, ..., OS X, ... etc.
Are all those companies just to stupid to provide proper and easy to understand documentation. I am sorry, but your statement is pure non-sense.

Quote
bob_franke:"The general consensus is that if you don't understand the math... that's your problem. What little help there is, is written at a level only a mathematician can understand."

I personally enjoy the documentation and especially seeing some formulas for deeper understanding the algorithm used in the tool I am applying. I am frequently
prototyping something in JavaScript and the formulas help me guiding.

There is the beautiful statement of Lewin: "Nothing is more practical than a good theory."

Cheers
 Thomas
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Juan Conejero on 2017 May 22 01:54:56
Hi Bob,

Quote
Several years ago you said…
The concepts of "true color" and "natural color" are illusions in deep-sky astrophotography. Such things don't exist. The main reason is that a deep-sky image represents objects far beyond the capabilities of the human vision system.

So which is it, you seem to have change your mind.

Not at all. I think exactly the same today: color is purely conventional in astrophotography. The best example of this is narrowband imaging, where one has to use an arbitrary color mapping convention or palette in order to represent different wavelengths outside the RGB band as an RGB image. As long as your rendition is consistent throughout the whole image, any palette is valid, although some palettes will allow you to represent the data better than others. The same is true for RGB data. For example, nothing stops you from exchanging the red and blue components if you have a good reason to do so, either from a purely aesthetic perspective, or for the sake of information representation in a particular case.

For conventional RGB color representations, where the R, G and B components are to be represented as red, green and blue colors respectively, any white reference is applicable for the same reason. This is why we provide a large set of selectable white references in PCC, including most spectral types and a number of galaxy types, among other options. However, the choice of a white reference may have a strong impact on the documentary value of the image in our opinion, and this is a very important point for us. We think that no spectral type—including G2V—is suitable as a white reference because, in general, no particular star is representative of the objects being shown in a deep sky image. On the contrary, the integrated light from a spiral galaxy may provide a combined source of all of the existing spectral types and deep-sky objects, which makes it an excellent neutral, unbiased white reference for RGB deep-sky data. An unbiased reference is essential to generate a rendition that can maximize information representation, which is a crucial goal for us. For this reason, the default white reference in PCC has been generated from the average fluxes of Sb, Sc and Sd galaxies, or what we call the average spiral galaxy reference.

Quote
Then you said in this announcement…
If you want to persist in making common conceptual mistakes, you will be able to use the G2V spectral type as a white reference—but PCC will allow you to select virtually any spectral type, along with several galaxy types, to calibrate the color of your images automatically and accurately in PixInsight.

I find this statement incredibly arrogant. Who are you to say that your color philosophy is better than anyone else's? Also, you are again stating that PCC provides "accurate" color, which you previously stated does not exist. At least one astrophysicist and many of the best astrophotographers on the planet accept the G2V and/or eXcalibrator methods. eXcalibrator's Linear Regression routine uses stars of multiple colors and gets the same result as the "white-star only" routines.

Sorry if that sounds arrogant to you, but it's just a concise description of what I think. If you prefer, I can prepend an IMHO token to say IMHO, using the G2V spectral type as a white reference for deep-sky images is a common conceptual mistake, or even polish it to say IMHO, using the G2V spectral type as a white reference for deep-sky images is not the best choice, but I am not a big fan of palliative formalisms. In part this is probably because my mother tongue is Spanish. We tend to say things more directly and less sweetened in Spanish. At any rate, my intention has not been to put myself above anybody.

As for the rest of your post, I prefer to not comment more on that.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Juan Conejero on 2017 May 22 02:09:46
Thank you, Thomas.

Quote
There is the beautiful statement of Lewin: "Nothing is more practical than a good theory."

A nice quote, and with many practical applications in PixInsight! ;)
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: bob_franke on 2017 May 23 20:21:15
For future PCC users, I strongly recommend using the SDSS-DR9 data instead of the APASS whenever possible. The SDSS data are acquired with a 2.5m telescope and the APASS data with a 3.15cm.  Also, the SDSS folks are more experienced. Additionally, the APASS staff have freely admitted that some of their data are problematic.

I recently encountered a Southern Hemisphere field-of-view where the APASS data had obviously highly inaccurate (b-v) values. Also, the Sloan g' and r' filters produced poor RGB color correction. In the Northern Hemisphere, the APASS data have always agreed with the Sloan for color correction.

BTW Juan, will the PCC user interface return the RGB values used for the pixel math? I would like to compare with my results. I have no doubt that the PI routines are more sophisticated than eXcalibrator's white-star and linear regression.  Also, with direct access to plate solving, the PI process will be easier to use.

But will the results provide a significant difference in the final color? It can take a surprisingly large change in RGB factors to be noticeable in the final image.

Like I said earlier, I am a PixInsight user and find the program exceptional and powerful. I'm looking forward to taking PCC for a spin.

Regards,
Bob
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Philippe B. on 2017 May 30 05:23:47
Hi Juan
Very exciting features for next release


If you need some "direct integrated" images (modded DLSR A7s) with gradients have more images to test new algorithms, let me know, it will be a pleasure to help.


Anyway, again, again, many thanks for your work. 
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: kkretzsch on 2017 May 31 00:19:19
I still don't dare to anticipate a release date. It should happen during the first half of June...
Looking forward to it  ;)

If you find time, I have a feature request: O:) ... many webserrvice api use content type json. Would be great if the NetworkTransfer could support json as content types as well.

Btw, what is the javascrip member/method for the EnablePasswordMode method of the pcl::Edit class. Couldn't figure it out ...

Klaus
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Juan Conejero on 2017 May 31 01:05:00
Hi Klaus,

Thank you :)

NetworkTransfer::ContentType() (https://pixinsight.com/developer/pcl/doc/html/classpcl_1_1NetworkTransfer.html#aeb92cb7acca45e8ce99b54856b997fa7) will return whatever the server has reported as content type for the latest download operation. Unfortunately, in many cases servers and/or network applications are not properly configured and report invalid content types such as text/plain for JSON data, instead of application/json. There is nothing we can do to solve this.

Edit::EnablePasswordMode() (https://pixinsight.com/developer/pcl/doc/html/classpcl_1_1Edit.html#a1cb81988a3161da6fc8112f638ed10ec) is what you are looking for.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Juan Conejero on 2017 May 31 01:10:52
By the way, if you are requesting passwords using PCL controls such as Edit, consider calling String::SecureFill() (https://pixinsight.com/developer/pcl/doc/html/classpcl_1_1GenericString.html#aca549f0262e095a6b9931a885b353cd9) to wipe out the password securely when it is no longer needed. The corresponding string in the server-side control is always destroyed securely when the password mode is enabled.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Juan Conejero on 2017 May 31 01:23:51
Oh, sorry, I didn't read your post correctly (too many things in my head I guess). The PJSR Edit object also has a passwordMode property:

Boolean Edit.passwordMode

which works just as pcl::Edit::EnablePasswordMode().
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Juan Conejero on 2017 May 31 01:27:19
Hi Philippe,

Thank you so much.

Quote
If you need some "direct integrated" images (modded DLSR A7s) with gradients have more images to test new algorithms, let me know, it will be a pleasure to help.

Having more images for testing is always a good thing, so yes, if you can upload some of them I'll appreciate it ;)
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: kkretzsch on 2017 May 31 01:30:29
Hi Juan,
Oh, sorry, I didn't read your post correctly (too many things in my head I guess). The PJSR Edit object also has a passwordMode property:

Boolean Edit.passwordMode

which works just as pcl::Edit::EnablePasswordMode().
Ah, thanks, yes this is what I am  searching for ...

Quote from: Juan Conejero

NetworkTransfer::ContentType() (https://pixinsight.com/developer/pcl/doc/html/classpcl_1_1NetworkTransfer.html#aeb92cb7acca45e8ce99b54856b997fa7) will return whatever the server has reported as content type for the latest download operation. Unfortunately, in many cases servers and/or network applications are not properly configured and report invalid content types such as text/plain for JSON data, instead of application/json. There is nothing we can do to solve this.
My problem is that I want to send data in JSON form via POST request. With curl I would express it as follows:

curl -H "Content-Type: application/json" -X POST -d '{"user":"xyz", "password":"XYZ"}' url

AFAIK the POST operation of the NetworkTransfer POST method only send form data ... 

Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Juan Conejero on 2017 May 31 02:23:40
Hi Klaus,

This is not possible with the current version. I am going to implement support for custom HTTP headers in version 1.8.5 (hopefully I'll upload a new development version for Linux today, along with a new version of PCL).
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: kkretzsch on 2017 May 31 02:31:03
Hi Juan
I am going to implement support for custom HTTP headers in version 1.8.5 (hopefully I'll upload a new development version for Linux today, along with a new version of PCL).
This is great news! Can I use/download such a development version? Then I can also continue on my telescope pointing stuff ...
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Juan Conejero on 2017 May 31 03:17:25
Quote
Can I use/download such a development version?

Sure! I'll let you know when it's ready.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: mmirot on 2017 May 31 13:32:39
Lot's of cool new tools this summer to play with! I can't wait.

Thank in advance Juan.

Max
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: ChoJin on 2017 June 03 07:19:57
Are we there yet? ;-)  :P

(In all seriousness I'm about to process an image and I'd like to reprocess one I'm not fully satisfied with, and I'm sure this new release would help. Hence I'm pondering whether or not I should wait for it ^^ )
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Juan Conejero on 2017 June 04 02:52:27
Quote
Are we there yet?

Almost there... I'd bet that yes, you'll be satisfied ;)
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: kayronjm on 2017 June 13 15:39:36
This looks like an absolutely wonderful tool. Well done guys! Looking forward to playing with it. I will be updating a tutorial of mine to add this in once it's out. :)
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: mtrungadi on 2017 July 01 05:32:33
Juan Conejero
I've seen some people are using version 1.8.5 test, could I even become a beta tester?
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Niall Saunders on 2017 July 01 05:54:24
Hi,

Perhaps I can answer on Juan's behalf:

The latest version is still very much in early Beta revision, and has only been distributed to PTeam members. It cannot (and will not) be distributed outside of this small group as it does not yet have even something as fundamental as a fully-compatible installation procedure (which also likely to be all-new).

I know that users will very likely be super-keen to get their hands on the latest version and, trust me, "Good things will come to those that wait  :D "

The very last thing that Juan needs on his plate right now is an all-new version in circulation, being used by people who may then swamp the system with standard "great stuff, but how can I nake PixInsight Proces XYZfoo do bar" queries. Right now he needs to be given the time and freedom to eliminate the bugs that are already being identified by the PTeam users - so that when it does finally get released the queries are most likely being caused by genuine user-inexperience than by unidentified bugs.

Becoming part of the group of beta testers has always been 'by invitation only' - not that we are anything special, it is just that we have all worked hard to support PixInsight (many of us from the very beginning), which is why it is rapidly becoming the de facto astroimage processing software of choice - at all levels of the science.

However, I also know full-well that, if Juan feels that you would be a worthy contributor, he will be in touch (and, if he isn't, then don't worry or feel bad - it is probably because he is too busy trying to get the non-Beta version into your Update Repository as soon as possible  ;) ).
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: mmirot on 2017 July 02 21:22:57
Early beta?
Niall are you sure about that ?   
Juan put the teasers on the release out six weeks ago.
(He was hoping for a June release originally) 

I would think it would be fairly well along by now.


Max
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Niall Saunders on 2017 July 03 07:29:34
Reply Censored  :police:  :sealed:
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: plawrence1 on 2017 August 04 04:43:07
Very excited about this feature as I see it just appeared on the latest Pixinsight.

Are there any notes on how to use it? I'm getting plate solving errors and don't really know what to do to fix it

Pete
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: johnpane on 2017 August 04 14:34:09
Should PCC be run before or after background extraction? I would have thought after but am getting better results if I run it before.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: Niall Saunders on 2017 August 04 14:59:59
Hi John,

It may well be that you choose to use PCC instead of DBE  ;)

Probably best to wait for a clear write-up to be released.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: msmythers on 2017 August 04 15:30:13
In my mind PCC should be used in the same flow as you would with BN/CC, that is after gradient corrections. Gradients can affect different colors different ways. DBE and ABE deal with that. If you color correct an image with a color gradient how could you end up with the correct color balance across the image.

As far as 'it looks better before' you still might need to play with the settings. A slight change to the Saturation threshold maybe but not likely. Change in the White reference can make a real difference. Also remove the Automatic limit magnitude and try a step up or down but generally no more then that. This is what I found while testing PCC.

Also remember if you are using the background Neutralization as in previous versions of PI with a preview, try without and the defaults. And the converse is worth a try also.

Mike
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: bob_franke on 2017 August 06 11:16:18
By using aperture photometry, PCC should, and does, give the same color correction factors before and after color gradient removal.

My preference, with DBE and ABE, is to correct the image with Normalize unchecked. This way a good background model will essentially produce a neutral background. Then the PCC background tweak will probably work better.

Cheers,
Bob
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: pfile on 2017 August 06 12:08:55
bob, is this simply because the gradients are so tiny relative to the signal in the stars? it makes sense that gradients would not matter if that's the case.

but for background neutralization gradients are still important, right?

rob
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: bob_franke on 2017 August 06 13:36:50
Hi Rob,

Yes, the large relative difference in the signal is a factor. However, aperture photometry is essentially doing an ABE correction for each star. The process subtracts the local background, actually foreground, for each star before taking the measurements. This gives normalized readings across the entire image.

As for background neutralization gradients… Yes, of course, we must still apply DBE and ABE corrections to get a neutral flat background.

Bob
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: pfile on 2017 August 06 15:21:38
gotcha - i know little about aperture photometry so thanks for the info!

rob
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: gnomus on 2017 August 09 23:12:52
For future PCC users, I strongly recommend using the SDSS-DR9 data instead of the APASS whenever possible. The SDSS data are acquired with a 2.5m telescope and the APASS data with a 3.15cm.  Also, the SDSS folks are more experienced. Additionally, the APASS staff have freely admitted that some of their data are problematic.

I recently encountered a Southern Hemisphere field-of-view where the APASS data had obviously highly inaccurate (b-v) values. Also, the Sloan g' and r' filters produced poor RGB color correction. In the Northern Hemisphere, the APASS data have always agreed with the Sloan for color correction.

BTW Juan, will the PCC user interface return the RGB values used for the pixel math? I would like to compare with my results. I have no doubt that the PI routines are more sophisticated than eXcalibrator's white-star and linear regression.  Also, with direct access to plate solving, the PI process will be easier to use.

But will the results provide a significant difference in the final color? It can take a surprisingly large change in RGB factors to be noticeable in the final image.

Like I said earlier, I am a PixInsight user and find the program exceptional and powerful. I'm looking forward to taking PCC for a spin.

Regards,
Bob

Sorry to be a bit slow.  How does one acquire the SDSS data and 'enable' it in PI?  In my installation the only Photometry catalogue available in the drop down menu is APASS.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: vicent_peris on 2017 August 10 00:31:07
Hi Bob,

For the moment, the only photometric catalog available in PCC is APASS. SDSS could be available in the future, but a few updates of this tool have higher priority.

Best regards,
Vicent.
Title: Re: New in PixInsight 1.8.5: PhotometricColorCalibration
Post by: bob_franke on 2017 August 10 01:09:25
For only 30% of the Northern Hemisphere, it may not be worth the extra download time to check for SDSS availability. :-)

-bob