ICC color profile transformation: 1.8 gamma ProPhoto to linear ProPhoto & back

[Elle]

Which gives the most accurate ICC color profile transformation?  Photoshop CS2 or PI Core?

Fair warning - I am not a mathematician - feel free to tell me where I'm wrong!  Also, there are five questions at the very bottom of this rather long post.



1)Linear and gamma = 1.8 ProPhoto working spaces:

Mathematically the conversion from a gamma = 1.8 space to a gamma = 1 space is a straightforward and totally lossless transformation, using the following conversion equation:

rgb# divided by 255 (to normalize, assuming 8-bit file - use other value as appropriate), result raised to the 1.8 power, then multiplied by 255 (or other value as appropriate).  

And to get back from gamma 1.8 space to gamma 1.0 space would involve going the other way, and again, the transformation should be totally lossless.

In Photoshop CS2 I created a "linear ProPhoto, gamma = 1.0" working space by opening up the PS "Color Settings" dialog, selecting ProPhoto as the RGB Working Space, then selecting "Custom", which brings up the ProPhoto color space information.  Then I changed the gamma from 1.8 to 1.0 and saved the resulting color space as "linear ProPhoto".  

The 16-bit ProPhoto tiff that ACR 3.7 produces after converting a raw file is in the standard Adobe gamma = 1.8 ProPhoto working space (assuming you choose ProPhoto rather than sRGB for output from ACR).  The whole point of this exercise is I would like the option of working in a gamma=1 working space, without paying a heavy "conversion" penalty of loss of information by doing what OUGHT to be a lossless operation.



2)Photoshop conversion from gamma 1.8 to gamma 1 and back shows gapping:

The problem with using Photoshop to convert from gamma 1.8 space to gamma 1.0 space and back to gamma 1.8 space is that the histogram of the resulting file demonstrates gapping and combing - information has been lost, probably because Photoshop actually uses 15-bit integer math rather than true 16-bit floating point math.  The gapping and combing is evident regardless of which "engine" or "intent" is used and whether or not "black point compensation" is used.  The "use dither" option would eliminate the banding, but that is cheating, as "use dither" randomizes the last significant decimal point (probably not phrased right) of information for the precise purpose of eliminating banding - thus guaranteeing a less accurate though bandless and combless conversion.  

The Photoshop histogram shows the loss after the above-described double conversion.  But to really see the loss, use the Reindeer Graphics Optipix Wide Histogram plug-in, available at http://www.reindeergraphics.com/index.php?option=com_content&task=view&id=31&Itemid=54.

So can PixInsight Core be used to create a more accurate, less "lossy" conversion from a gamma = 1.8 space to a gamma = 1.0 space and then back again?



3)Comparing results from using PS and PI Core to do ICC profile transforms:

Working strictly in 16 bits, and starting with the same 16-bit ProPhoto gamma = 1.8 file, I made conversions to linear ProPhoto and back to gamma 1.8 ProPhoto, using Photoshop CS2 and using PI Core - seeking the particular combination of intent, bit-depth, floating (for PI Core) or not floating (for Photoshop), and with or without black point compensation - that would give the most accurate round-trip conversion.

I eliminated "black point compensation" from further testing, because PI Core consistently completely chokes up the shadows when the option "black point compensation" is used.  As "use black point compensation" is supposed to eliminate rather than produce choked up shadows, there seems to be a problem with the PI Core implementation of "use black point compensation".  For Photoshop, the "use black point compensation" option, at least for the conversion into and back out of linear ProPhoto working space, made no visible difference in the shadow areas of the resulting tiff.  I used the Adobe conversion engine in PS, as previous testing within PS CS2 had shown no visible difference in the final tiff when using the Microsoft engine instead of the Adobe engine.  

All test tiff conversions began with the same 16-bit gamma 1.8 ProPhoto output from ACR 3.7.  And for my initial testing, all files were saved as 16-bit files, with no interim conversion to 32-bits.  My final set of converted test tiffs included:  

Photoshop conversions from gamma 1.8 to gamma 1, and back to gamma 1.8, using relative colorimentric, absolute colorimetric, and perceptual (three test tiffs). plus

PI Core conversions from gamma 1.8 to gamma 1, and back to gamma 1.8, using relative colorimetric, absolute colorimetric, and perceptual (three more test tiffs, for a total of 6).

All six test tiffs, plus the original 16-bit tiff created by ACR 3.7, were pulled into a Photoshop *.PSD file as layers.  I then used the Photoshop color sampler tool to take spot color sample RGB readings (four at a time, that being the Photoshop CS2 limit for the number of simultaneous color samples).  I took a total of 24 readings.  The first four readings were enough to show that the most accurate conversion by Photoshop was from using perceptual rendering intent.  And the most accurate conversion by PI core was from using relative colorimetric rendering intent (with floating point math - I didn't try unchecking the floating point option).  

Of the 72 individual channel readings from my 24 test color samples, Photoshop perceptual had 5 errors and PI Core relative colorimetric had 0 errors.  

More important than the number of errors (which were really taken as a way of confirming the following observation) was the fact that the Reindeer Graphics Wide Histogram showed significantly less combing and banding for the PI Core transformed tiff, than for the Photoshop transformed tiff, when comparing the PS perceptual rendering to the PI Core relative colorimetric rendering.

The obvious conclusion is that if someone wants to work in a linear (gamma = 1) working space and is starting with a file that was created with a gamma>1 ICC profile, PI Core will create a more accurate ICC profile conversion than Photoshop CS2.



4)32-bit interim conversion?

Now what about an interim conversion from 16-bit to 32-bit for the original file, with all subsequent conversions being done in 32-bit floating?  Can't be done in Photoshop, other than the original conversion to 32-bit.  I converted my original 16-bit test file to 32 bits in Photoshop and saved as "floating" (Photoshop and PI cannot read each other's respective "32-bit integer" files).  I opened the file in PI Core and converted to linear ProPhoto and saved as a 32-bit floating point tiff.  I then converted the linear ProPhoto tiff back to gamma 1.8 ProPhoto and saved the file as a 32-bit floating point tiff.  Photoshop opened both 32-bit PixInsight-produced tiffs, but it displayed the second tiff, the tiff that had gone from gamma 1.8 to gamma 1, to gamma 1.8 incorrectly, showing very washed out colors, like what you'd expect if a gamma 1.8 profile had been assigned to a gamma 1.0 file.  I went back into PI and saved the same file as a 16-bit tiff, and Photoshop opened and displayed it correctly.  

I did a spot check on four color sample points on the file with the interim 32-bit conversions - they matched the original file.  But I didn't feel like checking more points.  

The "acid test" of ICC profile transform accuracy is to transform the "www.brucelindbloom.com/RGB16Million.html" tiff.  I did so, using 32-bit processing as far as possible, and using relative colorimetric floating no black point compensation for PI core and perceptual no black point compensation for Photoshop.  Both round-trip transformations showed the resulting histogram "cut in half" as shown by the Photoshop histogram.  The Wide Histogram showed the shadow areas comby for Photoshop, and several random spots of slight combiness for PI.  I couldn't find any inaccurate color samples for either, but I didn't try very hard.  



5)Questions

I have some questions regarding PI Core's ICC profile transform function.

First, which set of parameters - bit-depth, rendering intent, use or don't use black point compensation, bit depth for saving - OUGHT to give the most accurate ICC profile transform, assuming that the transform in question is merely going from one gamma to another, rather tthan going from say, ProPhoto to WideGamut?  

Second, is there any mathematical accuracy to be gained (or lost?) by doing an interim conversion to 32 bits?

Third, why does Photoshop display the second conversion, saved in 32 bit floating, incorrectly (washed out, lightened, like it is using the a profile with the wrong gamma), when it correctly displays the first conversion, also saved in 32 bit floating.  A Photoshop problem?  a PI problem?  Or user error on my part?

Fourth, why the choked up shadows when using black point compensation with PI Core?

And finally, does PI Core use dithering or some such equivalent?  Or is the histogram from PI core really showing a better, more accurate, less lossy ICC profile transformation?


Thanks! for responding
Elle

[Juan Conejero]

Hi Elle,

Excellent test and description. This is an amazing study, and I am very interested in your results.

PixInsight's color management is based on the wonderful LittleCMS engine. This is an open-source, beautiful piece of code that has been released under the MIT license. LittleCMS makes PixInsight's color management completely portable and cross-platform. So PixInsight does not depend on any platform-specific or third-party color engine.

An important consequence of this is the fact that PixInsight is the first advanced image processing application that implements professional-grade color management under Linux.

In fact, we still have to exploit most of LittleCMS's power. We have just implemented a small fraction of its rich functionality.

About Bruce Lindbloom's 16 million color image. Take into account that this is an 8-bit image. For better evaluation of PI's capabilities, this image should be transformed to 32-bit floating point before testing. I'll write a script to generate a 32-bit image with more than 16 million colors.

Now I'll try to answer your five questions.

First, which set of parameters - bit-depth, rendering intent, use or don't use black point compensation, bit depth for saving - OUGHT to give the most accurate ICC profile transform, assuming that the transform in question is merely going from one gamma to another, rather tthan going from say, ProPhoto to WideGamut?

As for bit depths and data types, obviously the 32-bit floating point format provides much higher accuracy than the 16-bit integer format. Since ICC profile transforms involve relatively complex numerical calculations, and color accuracy is critical, the ICCProfileTransform process should be applied in floating point mode to a 32-bit image, when possible.

In the current PixInsight platform, the 32-bit floating point format provides between 10^6 to 10^7 effective discrete numerical values for representation and storage of pixel values, depending on the complexity of applied processes.

An interesting alternative is the 32-bit integer format, which provides a huge range of 2^32 discrete values. However, many processes can be slow working with 32-bit integer images.

The 64-bit floating point format is reserved to deal with extremely large dynamic ranges and very complex transformations. It is usually not necessary and of course ICCProfileTransform doesn't justify its use.

Now the problem is if you are going to export images generated with PixInsight to other applications that don't support some advanced formats, as floating point or large integers. Then you must save as 16-bit files. However, if the conversion to 16-bit is performed as the very last step, there should be no major problems in practice, at least for normal daylight photography (not too large dynamic ranges).

After saving a true 16-bit image with PixInsight, beware of Photoshop's "16 bits", which are actually 15 bits. You already know this, but it is worth to mention, because it means that you lose one half of the existing levels in your image (I can demonstrate this if you want/need).

Having said that, to perform a simple gamma transform you don't need an ICC transform at all in PixInsight. You can do this with PixelMath instead, which is an extremely accurate (research grade) processing tool. The gamma function is just the power (raise) function. Consider the following:

Code: [Select]

f is the linear version (gamma=1) of your image.
f^1.8 is your image with gamma=1.8 applied.
f^2.2 is your image with gamma=1.8 applied.

where the ^ sign denotes exponentiation (the common raise operation).

Now assume that you have f^1.8, but you want f^2.2. This can be achieved very easily as follows:

Code: [Select]

f = (f^1.8)^(1/1.8)
f^2.2 = ((f^1.8)^(1/1.8))^2.2

So the problem reduces to returning your current image to the linear space, and then applying the desired gamma function. Of course, you have to know which the current gamma is, since there is no (direct) way to guess it.

A PixelMath expression that implements the above equation is simply:

Code: [Select]

($target^(1/1.8))^2.2

Just type this expression in the R/K field of PixelMath's interface (other parameters by default) and apply it to your image, preferably in 32-bit floating point format. Now you have converted it from gamma=1.8 to gamma=2.2.

If you want to transform to gamma=1.0, it's as simple as:

Code: [Select]

$target^(1/1.8)

again assuming that yout current gamma is 1.8; change the numbers otherwise. If somebody knows another, simpler or better way to do this, please let us know.

Second, is there any mathematical accuracy to be gained (or lost?) by doing an interim conversion to 32 bits?

I've already answered this question above. The answer is yes, of course.

Third, why does Photoshop display the second conversion, saved in 32 bit floating, incorrectly (washed out, lightened, like it is using the a profile with the wrong gamma), when it correctly displays the first conversion, also saved in 32 bit floating. A Photoshop problem? a PI problem? Or user error on my part?

I'd need to have a look at the images to give you an answer. PixInsight's floating point support works without flaws, as far as we know (and we indeed have tested this, as you can imagine). Otherwise we'd be having a lot of problems, which we don't have!

Fourth, why the choked up shadows when using black point compensation with PI Core?

This is something I have to investigate; I really don't know why is this happening. However, if you apply ICCProfileTransform in floating point mode to a 32-bit floating point image, I think you don't actually need black point compensation. But of course I may be wrong. I'll try to learn more about this topic and will return here with more accurate information.

And finally, does PI Core use dithering or some such equivalent? Or is the histogram from PI core really showing a better, more accurate, less lossy ICC profile transformation?

Of course not. We don't use dithering simply because we don't need it. Our platform is plenty of numerical accuracy (up to 64-bit floating point) so we invest our time in more interesting things than dithering tricks. Others say 16 when they have just 15, but this is not our case :lol:

Thank you for giving raise to such interesting topics.

[Carlos Milovic]

Just a quick (mathematical) note:

You don't have to write an expression like this:
(f^(1/1.8))^2.2
to go from gama 1.8 to 2.2 (just an example).

Instead, you may write:

f^(2.2/1.8)

since PixelMath's optimizations will locate 2.2/1.8 as a constant, it will be calculated only once (or this is the supposed behavior ;) )

[Juan Conejero]

Code: [Select]

f^(2.2/1.8)

Oh yes, this is primary school math  :oops:

This happens when one is up working at 4 am ...  Thanks for pointing!

since PixelMath's optimizations will locate 2.2/1.8 as a constant, it will be calculated only once

Of course. PixelMath will identify 2.2/1.8 as an invariant subexpression, which will be calculated only once upon execution start.

[Elle]

Juan, Carlos,

Thanks so much for responding.  And especially thanks about the tip about the pixelmath way of directly going from gamma 1.8 to gamma 1.0.  No need to invoke an ICC transform at all!  Just assign the correct profile after doing the math in PixInsight - yes?  

Again, thank you thank you thank you!  I was quite distressed by the information loss in Photoshop from simply going from a gamma 1.8 space to the same space with gamma 1, and back - it should be a totally lossless transformation, but quite evidently from the histograms a lot of information disappeared.


I will give a try (though not today) to transforming the Bruce file in PI to 32 bit floating, and proceeding through the transforms - I'll let you know how it turns out.

Elle

[Juan Conejero]

Hi Elle,

Some information about black point compensation. This is the corresponding entry excerpted from LCMS version 1.17 documentation:

************************************

9.1 Black point compensation

  The black point compensation feature does work in conjunction with relative colorimetric intent. Perceptual intent should make no difference, although it affects some profiles.

  The mechanics are simple. BPC does scale full image across gray axis in order to accommodate the darkest tone origin media can render to darkest tone destination media can render. As a such, BPC is primarily targeting CMYK.

  Let's take an example. You have a separation (CMYK image) for, say, SWOP. Then you want to translate this separation to another media on another printer. The destination media/printer can deliver a black darker that original SWOP. Now you have several options.

  a) use perceptual intent, and let profile do the gamut remapping for you. Some users complains about the profiles moving too much the colors. This is the "normal" ICC way.

  b) use relative colorimetric.This will not move any color, but depending on different media you would end with "flat" images, taking only a fraction of available grays or a "collapsed" images,
  with loss of detail in dark shadows.

  c) Use relative colorimetric + black point compensation. This is the discussion theme. Colors are unmoved *except* gray balance that is scaled in order to accommodate to  the dynamic range of new media. Is not a smart CMM, but a fist step letting the CMM to do
  some remapping.

 The algorithm used for black point compensation is a XYZ linear scaling in order to match endpoints.

************************************

From this explanation, it seems that enabling BPC for ICC profile transforms applied to RGB images does not make too much sense. I think this is why you are obtaining poor results with BPC enabled in your tests. I'd guess PS is circumventing BPC automagically in some way when you convert RGB images. But of course I may be wrong. Anyone knows more about this?