Possible bug in calls to dcraw

[johnpane]

Summary: When producing CFA greyscale or RGB bayer output, calls to dcraw do not respect user inputs regarding white balance. The -D parameter is used when -d may be the correct parameter.

Details:

I'm working with Canon CR2 files with the following camera parameters for white balance, reported by dcraw:
    Daylight multipliers: 2.096035 0.929390 1.315425

If I create a VNG debayered image, the color balance is shifted (as expected) when I request camera white balance  versus no white balance. However, when producing CFA or RGB bayer output, color balance is unaffected by white balance settings.

For example, BatchFormatConversion produces identical output with the following two sets of format hints:
    1. camera-white-balance bayer-drizzle
    2. no-camera-white-balance no-auto-white-balance bayer-drizzle

It produces the following calls to dcraw in these two cases:
    1. dcraw -4 -o 0 -w -D -t 0
    2. dcraw -4 -o 0 -r 1 1 1 1 -D -t 0

The problem is the -D parameter, which specifies no scaling (disabling white balance scaling). If the -d parameter were used instead, color balance scaling could take effect. (I do not know if other scaling factors are affected by -d vs. -D).

The same issues occur if these preferences are manipulated in DSLR_RAW preferences and the images are opened in the GUI.

[Juan Conejero]

I agree completely. Thank you for detecting this issue.

Neglecting -d has not actually been a bug, but an auto-imposed limitation inherited from the first versions of the DSLR_RAW module. The idea was that allowing white balancing for raw output is a bad idea, since in this way the loaded data are not actually raw, or not exactly the image as it is stored in the device. However, this was plausible when the no white balance parameter didn't exist. Now that we have this parameter, we have a secure way to force -D, so we can allow -d.

I have already fixed this problem in the DSLR_RAW module that will be released with the next version of PixInsight. It will honor all white balancing options with and without interpolation. For uninterpolated output (super pixel, Bayer RGB and Bayer CFA options), -D will be used when no white balance is true (just as happens now), -d when no white balance is false. Note that the "raw" format hint, as well as the "Pure RAW" option of the RAW Format Preferences dialog, will continue working as usual, since they force no white balance = true. You'll have the option to specify "raw camera-white-balance" to output raw data with camera multipliers applied.

[johnpane]

Excellent, thank you Juan!

[johnpane]

I have already fixed this problem in the DSLR_RAW module that will be released with the next version of PixInsight.

Hi Juan,

I have been keenly awaiting this fix, but noticed it was not included with the DrizzleIntegration fixes released this weekend. Do you know approximately when we should expect this to be released?

Thanks,
John

[Juan Conejero]

Hi John,

Since the next version is being delayed, I'm going to release an update to the DSLR_RAW module today. Sorry for the inconvenience.

[Juan Conejero]

Hi John,

The update to the DSLR_RAW module is now available to all supported platforms. Let me know if it works as expected.

[johnpane]

Thanks very much, Juan! I will test it and let you know.

[johnpane]

Hello, Juan. I tested this and it is now working as expected. I can now perform a bayer drizzle workflow while still having dcraw apply camera white balance. Thank you for taking care of this.

In the process of testing, I found something interesting about deBayered images generated directly by dcraw versus generated indirectly by first creating an RGBBayer image and then deBayered. I suspect there might be an error in dcraw's VNG algorithm.

First, my DSLR_RAW preferences depart from the default (clicking the de-Bayer RGB button) as follows:
    No image flip is checked
    No highlights clipping is checked

I opened the RAW file (CR2 format) directly, which called dcraw with these arguments:
    dcraw -4 -o 0 -w -q 1 -t 0 -H 1

Then I did a BatchFormatConversion with the input format hints:
    bayer-drizzle no-cfa camera-white-balance
This called dcraw with these arguments:
    dcraw -4 -o 0 -w -d -t 0 -H 1
As expected, "-q 1" (VNG deBayer) was replaced with "-d" (Show the raw data as a grayscale image with no interpolation) and the bayer-drizzle hint caused the cfa images to be combined into an RGBBayer image, which I then deBayered using BatchDeBayer.

The two resulting debayered images are very similar but on close inspection subtle differences can be seen. These may be differences in the VNG implementation built into dcraw versus the one in PixInsight. In fact, the version generated directly by dcraw may be incorrect.

To illustrate, I found a green hot pixel and here I attach three small images:
    1. the rgbbayer image showing the hot pixel and surrounding pixels
    2. the image debayered directly by dcraw
    3. the image first converted to RGBBayer, then debayered by PixInsight's VNG algorithm.

Image 2 shows a central bright pixel of neutral color and bright green extensions in three orthogonal directions, forming an asymmetric T shape. Image 3 is symmetric, with green extensions in all for directions, forming an X shape.

Beyond this example, I found other places where the direct conversion image (2) has stars with one-pixel extensions around the edge that make them appear slightly less symmetric with than in the indirect conversion image (3).

Looking at the entire images (not just the small crops posted here), almost all of the pixels differ by a small amount between the two images, in two of the three colors. I took the absolute value of the difference between the two images. Here are the statistics, and I attach a crop (from a different section of the image than where the hot pixel is).
            R          G          B
count (%)   73.42334   51.10786   73.69855
count (px)  14809722   10308619   14865233
mean        0.0000711  0.0001250  0.0000654
median      0.0000076  0.0000089  0.0000076
avgDev      0.0000672  0.0001209  0.0000620
MAD         0.0000038  0.0000056  0.0000038
minimum     0.0000000  0.0000000  0.0000000
maximum     0.8579233  0.4873732  0.4191005

John