Author Topic: New in PixInsight 1.8.5: PhotometricColorCalibration  (Read 36178 times)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Hi everybody,

Further news on the next version of PixInsight. Today we have been working on an exciting new tool: PhotometricColorCalibration (PCC). We have a first working version with really outstanding results that I want to share with you all.

PCC performs automatic plate solving and photometry (both aperture and PSF photometry are selectable) to calculate RGB white balancing factors based on measured star fluxes, with respect to a user-selectable white reference. Photometric data are retrieved online from the APASS survey through the VizieR server and mirrors. The new PCC tool uses internally the latest versions of the ImageSolver and AperturePhotometry scripts, authored by Andrés del Pozo and Vicent Peris. Vicent is also the author of the internal PCC calculation algorithms. I am writing the C++ implementation, making the most of the new functionality of version 1.8.5 that allows us to execute JavaScript scripts from PCL-based C++ code to write hybrid PixInsight modules. This means that PCC is a regular PixInsight tool, not a script, with all its inherent benefits.

Here are a few examples that Vicent and I have been preparing to show you the kind of results you may expect from PCC.

First a wide-field DSLR image of the Milky Way, courtesy of Georg Viehoever:


On the left, the original RGB debayered image. On the right, the result after applying PhotometricColorCalibration. The good news is that PCC is really easy to use. If your image has approximate center coordinates and image scale metadata (as FITS header keywords), PCC is basically a one-button tool in most cases (or, speaking in pure PixInsight terms, a one-blue-triangle tool). The next example is a two-frame mosaic of the region around NGC 2080, in the Large Magellanic Cloud (data by Vicent Peris):


This is an image of the LBN 552 region acquired with the 1.2 m Zeiss telescope of Calar Alto Observatory:


The original combined RGB image, shown on the top-left, had already been white balanced using the average color of the stars in the frame as white reference. The newly calibrated image with PCC is on the bottom-right of the screenshot. The difference speaks by itself.

The last example is an image of M51, also acquired through the 1.2 m telescope of Calar Alto Observatory:


After applying PCC, the image shown on the right has been stretched nonlinearly with HistogramTransformation, processed with HDRMultiscaleTransform, and its color saturation has been increased with CurvesTransformation.

Besides its underlying high-accuracy astrometric and photometric analysis implementations, the most innovative and powerful feature of our new PCC tool is, in my opinion, the fact that it allows you to select one among numerous predefined, carefully generated white references. By default, PCC applies a white reference based on the average of the characteristic fluxes of Sb, Sc and Sd galaxies. This reference, which has been used in all of the examples shown above, is in our opinion truly representative of the deep space, and hence an unbiased, neutral white reference quite close to our documentary color philosophy. If you want to persist in making common conceptual mistakes, you will be able to use the G2V spectral type as a white reference—but PCC will allow you to select virtually any spectral type, along with several galaxy types, to calibrate the color of your images automatically and accurately in PixInsight.

A huge kudos to Andrés del Pozo and Vicent Peris, who are the authors of the excellent implementations and algorithms behind the new PhotometricColorCalibration tool. Thank you for your continued support and contributions, which make PixInsight an exciting platform in constant evolution.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1458
    • http://www.harrysastroshed.com
Re: New in PixInsight 1.8.5: PhotometricColorCalibration
« Reply #1 on: 2017 May 19 13:58:49 »
Hi Juan

Look very interesting , But "One Button " where's the fun in that  >:D

Harry
Harry Page

Offline Nocturnal

  • PixInsight Jedi Council Member
  • *******
  • Posts: 2727
    • http://www.carpephoton.com
Re: New in PixInsight 1.8.5: PhotometricColorCalibration
« Reply #2 on: 2017 May 19 18:57:34 »
Exciting stuff!
Best,

    Sander
---
Edge HD 1100
QHY-8 for imaging, IMG0H mono for guiding, video cameras for occulations
ASI224, QHY5L-IIc
HyperStar3
WO-M110ED+FR-III/TRF-2008
Takahashi EM-400
PIxInsight, DeepSkyStacker, PHD, Nebulosity

Offline RickS

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1298
Re: New in PixInsight 1.8.5: PhotometricColorCalibration
« Reply #3 on: 2017 May 19 19:16:51 »
I'm really looking forward to this... and also not spending an hour per image using PixelMath to overwrite satellite trails  :)

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
Re: New in PixInsight 1.8.5: PhotometricColorCalibration
« Reply #4 on: 2017 May 20 06:13:31 »
Hi Juan (and, as always, Hi to all those who have made these significant contributions),

Yet again PixInsight raises the bar in the field of astroimage processing, with yet another totally scientific and statistical apporach to the solution of a problem that has, so often, previously been attacked 'artistically' or 'subjectively'.

As Harry says, whilst half the fun of PixInsight is spending hours fiddling with sliders and buttons whilst trying to figure out what combination best suits the image, if this truly is a 'one-blue-triangle' approach, then at least it will give us more time to fiddle with those pesky settings in all the other processes  :)

Thanks again.
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline Ignacio

  • PixInsight Old Hand
  • ****
  • Posts: 375
    • PampaSkies
Re: New in PixInsight 1.8.5: PhotometricColorCalibration
« Reply #5 on: 2017 May 20 10:19:44 »
This is amazing! I was just working on something along these lines, using the AperturePhotometry script!

Now, my motivation was driven by a DSLR issue with color calibration, that stems from the fact that the CFA has no clean frequency cut-offs and there is significant cross-talk between color channels. Proprietary digital development software inside the camera processors, and in software like adobe´s camera raw and dcraw, partly correct this by means of a color calibration matrix that maps the camera raw data onto some standard color space (ie, XYZ). This is camera and illumination dependent. Although one can find sources for such matrix data for different camera models and illuminations,  these are not useful with modded DSLRs in astroimaging situations. Hence, I was working on the idea of using catalog photometric data to determine such matrix via a multilinear regression approach, including pedestals in the model that could take care of background neutralization at the same time:

[RGB]_xyz = M [RGB]_raw + [RGB]_bias

where the elements of the 3x3 matrix M, and 3x1 vector [RGB]_bias, are obtained by fitting the detected data to the catalog photometric data.

Is this something that could be accomplished by the new module, or white balance will be handled in the standard three (RGB) multiplicative factor?

best,
Ignacio

Offline Herbert_W

  • PixInsight Addict
  • ***
  • Posts: 204
    • Skypixels - Astrophotography by Herbert Walter
Re: New in PixInsight 1.8.5: PhotometricColorCalibration
« Reply #6 on: 2017 May 20 10:36:41 »
Hi Juan!

That's an exiting announcement.

Just a question - don't be angry:
"Photometric data are retrieved online from the APASS survey - do you also think of including the SDSS-DR9 catalog?

Best regards!

Herbert, Austria

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: New in PixInsight 1.8.5: PhotometricColorCalibration
« Reply #7 on: 2017 May 20 10:41:51 »
Thank you so much to all.

Quote
But "One Button " where's the fun in that

Don't worry, Harry, the PCC tool will have lots of parameters to control the astrometric solution and the photometric analysis, along with the background reference, so you'll have plenty of numbers to play :) However, the tool will work fine with default parameters in most cases, if the image has the required metadata.

Quote
...not spending an hour per image using PixelMath to overwrite satellite trails

Hi Rick. You'll *never* have to do this again in PixInsight. The new large-scale pixel rejection feature of ImageIntegration works flawlessly. It works so well and is so robust, that you'll be able to integrate a data set simply ignoring all plane and satellite trails, etc. Just as if they didn't exist in virtually all cases.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: New in PixInsight 1.8.5: PhotometricColorCalibration
« Reply #8 on: 2017 May 20 10:50:50 »
Hi Ignacio. Thank you.

Quote
[RGB]_xyz = M [RGB]_raw + [RGB]_bias

The current (first) version of the PCC tool computes scaling factors for the individual RGB components. The mean background is sampled and evaluated through robust statistics from a user-defined region of interest, just as the current BackgroundNeutralization tool does now. However, your idea looks very interesting, and I think it could be implemented without problems. I also think your approach is sound and should work well.

The PCC tool will be released as an open-source product (under PCL license), so it will be available at our open-source GitHub repositories, and hence open to collaborations. Our PCL development framework has all of the necessary resources to implement what you want (and in case it lacks something, we can implement it). This would be an excellent improvement.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: New in PixInsight 1.8.5: PhotometricColorCalibration
« Reply #9 on: 2017 May 20 11:15:09 »
Hi Herbert,

Thank you.

Quote
do you also think of including the SDSS-DR9 catalog?

The new PCC tool uses the latest release of the APASS survey as its source of photometric data by default. However, PCC is very flexible, so you can use basically what you want. You can select any catalog currently supported by the AperturePhotometry script (which, by the way, should be renamed to Photometry IMO, since it now performs both aperture and PSF photometry).

As I have said before, PCC will be released as an open-source module, so it will be open to collaborations from external developers. With the new version 1.8.5 of PixInsight, we'll release almost all of the new tools as open source products, including LocalNormalization (previously announced as FrameAdaptation), SubbandBlending, and of course the new features that will be available in ImageIntegration and DrizzleIntegration, among many others. I am betting very hard on open source releases, as the best way to reinforce the required dynamism on the PixInsight platform.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1458
    • http://www.harrysastroshed.com
Re: New in PixInsight 1.8.5: PhotometricColorCalibration
« Reply #10 on: 2017 May 20 11:23:27 »
Hi Juan

What is SubbandBlending going to be ?  please let it be image blender !!!!!!!!!!!

Harry
Harry Page

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: New in PixInsight 1.8.5: PhotometricColorCalibration
« Reply #11 on: 2017 May 20 11:31:30 »
so what's the timeframe like for release of 1.8.5? i have a boatload of my usual crappy LP data that i have been holding off processing because i'd like to try the new normalization stuff.

rob

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: New in PixInsight 1.8.5: PhotometricColorCalibration
« Reply #12 on: 2017 May 20 11:31:45 »
The final name is still subject to discussion :) It will be a new tool to generate seamless mosaics using, among others, the sub-band blending (SBB) algorithm. SBB was first described, AFAIK, by P.J. Burt and E.H. Adelson in A Multiresolution Spline With Application to Image Mosaics. My implementation introduces significant changes and uses powerful multiscale techniques that we already have implemented in PixInsight, instead of Gaussian pyramids, but follows the same basic idea. SBB is also used with great success in well-known panorama generation applications, such as AutoStitch for example.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: New in PixInsight 1.8.5: PhotometricColorCalibration
« Reply #13 on: 2017 May 20 11:46:32 »
Quote
so what's the timeframe like for release of 1.8.5?

I still don't dare to anticipate a release date. It should happen during the first half of June. Version 1.8.5 is a very complex release, with very significant changes to the platform (as usual, much more and much deeper changes than what the user will see on the surface), many new tools, and many and very important improvements and new features.

For example DrizzleIntegration can now work with CFA monochrome data directly (direct Bayer drizzle), which involves a completely redesigned drizzle data format (the new XML-based XDRZ format, which includes LZ4-compressed rejection maps and local normalization data). Another example of complexity is the possibility to write hybrid modules that mix C++ code with JavaScript scripts (PhotometricColorCalibration is a good example). There is also a new HTML5 painting engine available on the JavaScript and C++ runtimes. etc, etc...

The new version 5.8.0 of Qt is fantastic, but as usual, it generates new problems and incompatibilities with our code base, which I have to fix. I already have a 1.8.5 version working extremely well on Linux, which I am using to concentrate all of my work on the new tools and improvements for now. So the macOS and Windows versions still require a lot of work. So I ask for (even more!) patience; 1.8.5 will be the best, most powerful and most stable version of PixInsight ever.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Andres.Pozo

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 927
Re: New in PixInsight 1.8.5: PhotometricColorCalibration
« Reply #14 on: 2017 May 20 12:04:04 »
Just a question - don't be angry:
"Photometric data are retrieved online from the APASS survey - do you also think of including the SDSS-DR9 catalog?
Hi,
AperturePhotometry can already use SDSS-R8. Supporting the release 9 should be very easy.