Any bias removed from Canon raw file reading ?

[bitli]

Hello,
I am trying to make some simple statistics and comparisons with my darks and bias from a Canon 350D. To my surprise reading the files with different software results in quite different values. Especially reading a bias with pixinsight results in an image with very low ADU values (0 to 5 on 4096). I read the same file in Iris, save it in TIFF and read the tiff file in pxixinsight I get an ADU of 31 to 33.

My understanding is that Canon add a bias and that the lowest value is not zero (although I am not sure this is the case on the 350).

Does the dcraw remove such an offset ? Does the load raw somehow rebase the image ? On images with saturated stars the scale goes up to 4095, so the subtract hypothesis does not seem right.

Any suggestion for the cause of the discrepancies ?

Bitli
PS. As an aside, I did not find a very easy way to calculate the mean of a set of images. With PixelMath I have to enter the names of the images. I did not find a way to read an ImageContainer in Javascript. Neither to apply a process that would return a single image to an ImageContainer. Any hint?

[Juan Conejero]

Hi Bitli,

In the Format Explorer window, select the RAW format and click Edit Preferences. On the RAW Format Preferences dialog, make sure both White Balance options are disabled, and the three Color Adjustment sliders are set to 1.0. With these settings dcraw will load just the raw data without any transformation. You may also want to select the "Create RAW Bayer Picture" option in order to load the true raw image without any interpolation. You can then obtain statistics for each channel with the Statistics tool.

Any suggestion for the cause of the discrepancies ?

Different decoders may apply different transformations to the raw data. AFAIK, In the case of dcraw, if you select the above settings the raw data are not changed in any way.

I did not find a very easy way to calculate the mean of a set of images.

Do you refer to the mean of each pixel in a stack of images? Of course this can be done with a script. However, we are now working on a new tool: ImageIntegration. This tool will perform the integration of a set of images using the mean or median with common pixel rejection algorithms (sigma clipping, min/max, etc.). I hope it will be ready in the next version.

If you are interested in a script, I can write it for you; it's quite easy (though a bit slow, of course). Please let me know.

[bitli]

Thanks for your prompt answer Juan.

According to Luc, DSS does not use bias either (http://tech.groups.yahoo.com/group/DeepSkyStacker/message/473 and following), but I was not able yet to find a comparable value. I will continue to investigate.
For Iris, it may be that it scales the image to the full 16 bits (from the 12 bits, or from 15 bits unsigned), which may cause discrepancies. All thisis fine when working in a single program, but more annoying when trying to combine results if multiple programs. Well, maybe one day I will do everything in Pixinsight:

Thanks for the proposal for the script, but I can write it . I guess I am lazy and I wanted to know if there is a way to access the list of images of an ImageContainer from Javascript, to avoid writing  the code to select image (which would be easy, I can just use all views... - but I am very lazy)... If not I'ill have to do the not so hard work...

Cheers,bitli

[bitli]

I have played a while with dcraw, and found that in fact there is a parameter (-k) that sets the black offset.
By default dcraw calculates a black and subtract it. from the image. Also dcraw convert to 15 or 16 bits in some way apparently.

If I decode a raw bias with dcraw -d -r 1 1 1 1 -k 0  -T -4 -o 0 (note the -k 0), then the generated tiff have a lowest value around 4000 when I load the tiff to Pixinsight (displayed on 16 bits, in  gray)
When the raw is directly loaded in Pixinsight (in create raw bayer, no WB, all color to 1.0), the lowest value is 0 (and most values 0 or very small).

So I guess that normally Pixinsight does not use the option -k (which makes sense for real images, as dcraw use hidden pixels to get the best black levels).I can always convert with dcraw if I want to use 0 based raws.

I am not sure I understand the implication of having dcraw rebasing the image on combining or inspecting darks/offsets with Pixinsight. or using the same master darks/offset with different software.

As an aside it would be nice if the log did mention the parameters use to load a raw, to have a trace if it.

Is my understanding correct ?
bitli

[Juan Conejero]

Hi Bitli,

In the next version, there will be an option on the RAW Format Preferences dialog to send the "-k 0" argument to dcraw. In this way you'll be able to load DSLR raw images with their original black offsets. The raw module will also write the employed dcraw arguments to the console.

You may want to suggest a similar feature to Luc Coiffier, perhaps this also makes sense for DSS.

Maybe I'll add also a second option to use the -n argument for wavelet denoising (I have to experiment with this dcraw's feature).

Thank you for sharing your insights and knowledge. User feedback is what really helps us to improve our software.

[Simon Hicks]

Hi Bitli and Juan,

This is really interesting for me....and maybe quite a few others. A while back I tried to investigate my Canon RAW CR2 images (darks, bias, etc) as a function of temp, exposure time and ISO. The results were not what I expected at all and I ended up giving up as I could not make sense of the trends.

Having read the results of Bitli's investigation it makes me think that the results I got could all have been due to a false background removal....something I did not expect from a raw.

So I will follow this thread, and any changes to PI that come out of it, with great interest. Great work guys.

Cheers
              Simon