DSLR_Raw Settings: CFA or not

[georg.viehoever]

Hi,

Astropixel's excellent tutorial on how to calibrate DSLR images http://pixinsight.com/forum/index.php?topic=2276 suggests to use "Create RAW debayer image" in the Format Explorer options for RAW. Other sources, such as the posting by Hans Pleijsier http://pixinsight.com/forum/index.php?topic=1868.0 suggest to use "Create Raw Bayer CFA" and even claims better results.

I am doing Astropixel's flow now, but from what I understand about the calibration procedure, it does not really matter which of the RAW options I use. For basic operations (integration and calibration without normalization, weighing and rejection magic), the results should be almost identical. But from what I see, I should think that using "Create Raw Bayer CFA" would be better for the advanced stuff.

1. The resulting (monochrome) images consume only 1/3rd of the memory  space (without CFA, the resulting images have 3 channels, and each pixel has only one channel value !=0). This also saves file space and speeds up operations.
2. The monochrome images produce useful statistics in the HistogramTransformation window (without CFA, 2/3rd of all values are always 0, resulting in large peaks in the histogram)
3. Pixel rejection statistics in Image Integration are more useful (without CFA, usually >66% of all pixel values are rejected)
4. I am not sure, but I guess that all those 0 pixels values are also not really helpful for the magic (e.g. normalization, weighting, ...) done by ImageCalibration and ImageIntegration.

What's the opinion of the Guru's?

Georg

[Hans Pleijsier]

Hi Georg,

Just a short reply before the Guru's tune in ...

My use of the Create_Raw_Bayer_CFA_option (with no more than acceptable results) was before Juan and others modified and enhanced the Debayer Script. This script is now able to debayer DSLR RAW images!

So I abandoned the CFA-workflow and now I am using AstroPixel's tutorial and the new Debayer script. The results are much better compared to what I used to produce (which is - by the way - still <below standard> for the most of you, I guess  ).

regards,
Hans Pleijsier.

[astropixel]

Hi Georg and Hans. I used the information provided by Juan, but I'd be interested to know your thoughts otherwise. Always happy to make changes to the thread once we're all satisfied that they are OK. I tried the CFA image option before Juan released the modified debayer script, but I wasn't happy with the results.
The tutorial really is a compilation of input from many sources, so I make no special claims - leave it open to improvement.

[pauljv]

I tried using the method of Astropixel and when I finally got it to work it produced identical images as the monochrome method, since this takes half of the time  I have gone back to using this method.

Regards Paul

[Juan Conejero]

Hi Georg,

Monochrome CFA images will work nicely and you're right with just one exception: dark frame optimization. Dark scaling will be more accurate with RGB Bayer images because an independent scaling factor will be computed for each channel. With a monochrome CFA, only one scaling factor will be computed as an "average" for the three channels, which also will work, but less accurately. Whether there are practical differences in the final calibrated images depends on the images ^TM

Note that this is actually a limitation of our dark optimization routine, due to the fact that it has been designed to work with each channel as an independent image (which is what they are for calibration purposes). I'll try to implement a special routine to compute independent RGB scaling factors from monochrome CFA images as soon as possible. When that happens, then there will be no reason to calibrate with RGB Bayer images anymore.

1. The resulting (monochrome) images consume only 1/3rd of the memory  space (without CFA, the resulting images have 3 channels, and each pixel has only one channel value !=0). This also saves file space and speeds up operations.

Agreed 100%.

2. The monochrome images produce useful statistics in the HistogramTransformation window (without CFA, 2/3rd of all values are always 0, resulting in large peaks in the histogram)

Also agreed. However you usually don't need to inspect the histograms of a mosaiced image, do you?

3. Pixel rejection statistics in Image Integration are more useful (without CFA, usually >66% of all pixel values are rejected

Pixel rejection will work equally well in both cases. However I agree the extra rejected pixels may be a bit distracting.

4. I am not sure, but I guess that all those 0 pixels values are also not really helpful for the magic (e.g. normalization, weighting, ...) done by ImageCalibration and ImageIntegration.

Hey, there's no magic in PixInsight (we leave all the magic for the 'other' applications)

Not helpful except for the dark optimization routine, as I've explained above.

[georg.viehoever]

Hi Juan,

thanks a lot for the answer. I can now see that I was thinking along the correct lines.

...
Monochrome CFA images will work nicely and you're right with just one exception: dark frame optimization. Dark scaling will be more accurate with RGB Bayer images because an independent scaling factor will be computed for each channel. With a monochrome CFA, only one scaling factor will be computed as an "average" for the three channels, which also will work, but less accurately. Whether there are practical differences in the final calibrated images depends on the images ^TM
...

Thinking a little bit about the dark optimization: Why would it be necessary in a DSLR to have different dark scaling factors for the channels RGB? All 3 channels always integrate light and noise for the same time. So in theory (I did not really measure my Canon ...), noise levels for all 3 channels should be identical. Since ImageCalibration uses these noise levels to estimate the scaling factors in dark optimization, all 3 factors should always be the same. In fact, considering the larger number of samples, the estimates should actually be more stable. Is that right? 

Thanks again,

Georg

[Carlos Milovic]

Hi Georg
I was thinking the same thing... since it is the same sensor chip, I see no reason why the noise should be different along it, following different functions (or scalations) for the CFA pattern.

[Yuriy Toropin]

Guys, forgive me for asking this simple question here, but anyway (while it's active thread)

In astropixel's DSLR_RAW Workflow Procedure (that is closed and that's why I'm asking here) step "2. Create master calibration files" refers to the great tutorial of Vicent Peris. The only real problem here is that ImageIntegration tool used for master bias and master dark creation accepts ONLY fits files, not DSLR raw ones.

With this the question is - did I miss something or I have to convert my individual bias and dark shots from DSLR (save in RAW, in NEF specifically) anyway before creation of the masters?

What's the point of NOT supporting DSLR RAW files as input for ImageIntegration then if they're accepted directly by ImageCalibration later...

Hopefully, I miss something... PLEASE, HELP! 

[NKV]

Hopefully, I miss something... PLEASE, HELP!  

Yuriy, maybe you missed the message about "incremental file reading".

[Yuriy Toropin]

NKV, thanks for pointing me to that post... I'm using mainly bw camera and was not following all discussions around OSC.
Anyway, looks like situation was not changed since that time and initial RAW -> FIT conversion is the only way to go. It's a pity...

Now - back to CFA or not. CFA being grayscale is much faster to work with vs Bayer (RGB, 3 layers) ones. Juan mentioned in some posts that for dark scale optimization you need in Bayer to optimize per channel. I don't understand why this is REALLY needed. If we have option "Force CFA" in ImageCalibration then, if it is selected, why not treat each 2x2 set of pixels as R,G1,G2,B (even if the real sequence is unknown) and IMDEPENDENTLY optimize dark for 4 "sublayers" presented in grayscale??? Bayer (RGB) is not really needed in principle... 3 times acceleration of preprocessing.

BTW, during BatchFormatConversion from RAW to Bayer/CFA, does it matter in which format (16 bit int vs 32 bit float) is to save FITS? Again 16 bit will be smaller and faster to work. Is it Ok to use 16 bit while majority of DSKR RAW are 12-bit or 14-bit now?

[pfile]

Hi Georg
I was thinking the same thing... since it is the same sensor chip, I see no reason why the noise should be different along it, following different functions (or scalations) for the CFA pattern.

i don't know the answer, but i can see that the noise in the red channel of all of my DSLR integrations is always higher than the green and blue channels. i can understand why the green channel might end up having lower noise, since the bayer pattern is RGGB, but that does not explain the lower noise in the blue channel. i can only guess that even though the camera is modified, the CMOS sensor is somehow less sensitive to red wavelengths than it is to blue wavelengths. or perhaps the red filter on the sensor has a narrower bandpass than the blue filter?