[ Solved ]V1.8.5 'Auto' Bayer does not work

[RobD]

NB: I originally started this as a bug report for version 1.8.4, having installed 1.8.5 I see that a Bayer 'Auto' option has been added, however this does not work for my Canon 1000d nor my Pentax K5 so I have left report as is.

I usually use a Canon 1000D for imaging but whilst on holiday I used my Pentax K5.
Resulting image colour was rubbish. After much digging and loading of daylight image I realised that the bayer on the K5 is BGGR  not GBBR as per the Canon.
I of course had left things set to GBBR. Changed to BGGR and all was fine.
OK my fault, tutorials do say to check bayer format etc.
However when looking at the log - LOG A (NB Dummy run for report only using GBBR in Batch Pre Processing Script.)

As can be seen PI is reporting Filter Pattern of BG/GR.
Given that PI has read the correct information from the camera should it not use it? I accept that possibly not all DSLRs will present the data but if available then use it rather than relying on the user selection.
If preferred then add a flag to override camera in case inerpretation is wrong.

OK in 1.8.5 this is covered by 'Auto'

I Checked the 'DeBayer Process Module' and noticed that it had an Auto Pattern selection ( Is this new in 1.8.5 - yes just found in BPP?)
Used the same image and got an error - see below LOG C.
Interestingly double click on image name in DPM and image opens with the correct colour - 1st part of LOG C, confirmed with day time image.

To summarise PI has at least four ways to open DSLR raw data:-
a) File->Open
b) BPP - now has auto.
c) DPM - now has auto.
d) Double Click on name in file list of process.

a and d correctly open the file using the correct debayer without user input.
b and c use user specified debayer even if wrong.
b and c on auto miss-interprets the debayer and throws an error - *** Error: Unsupported or invalid CFA pattern ''. NB same happens when using Canon 1000D.

The original bug/request has been covered in 1.8.5 by adding 'Auto' option to Bayer pattern selection.
Auto does not work for 1000D and K5 although file open does.

Thanks Rob
Apologies for the long post.

LOG A
[2017-08-05 10:18:58] Loading target frame:
[2017-08-05 10:18:58] T:/RawData/2017-06-25/Scorpius/d/RD_14388.PEF
[2017-08-05 10:18:58]
[2017-08-05 10:18:58] Timestamp: Sun Jun 25 23:04:55 2017
[2017-08-05 10:18:58] Camera: Pentax K-5
[2017-08-05 10:18:58] Owner: ROB DIPPLE
[2017-08-05 10:18:58] ISO speed: 400
[2017-08-05 10:18:58] Shutter: 29.9 sec
[2017-08-05 10:18:58] Aperture: f/2.4
[2017-08-05 10:18:58] Focal length: 35.0 mm
[2017-08-05 10:18:58] Embedded ICC profile: no
[2017-08-05 10:18:58] Number of raw images: 1
[2017-08-05 10:18:58] Thumb size:  4928 x 3264
[2017-08-05 10:18:58] Full size:   4992 x 3284
[2017-08-05 10:18:58] Image size:  4950 x 3284
[2017-08-05 10:18:58] Output size: 4950 x 3284
[2017-08-05 10:18:58] Raw colors: 3
[2017-08-05 10:18:58] Filter pattern: BG/GR
[2017-08-05 10:18:58] Daylight multipliers: 2.155461 0.926686 1.305787
[2017-08-05 10:18:58] Camera multipliers: 19552.000000 8192.000000 11392.000000 8192.000000
[2017-08-05 10:18:58]
[2017-08-05 10:18:58] Invoking: dcraw -4 -o 0 -r 1 1 1 1 -D -t 0 -k 0 -H 1
[2017-08-05 10:18:58] Decoding Pentax K-5 file (4950x3284 pixels, ISO=400, Exposure=29.93s): done
[2017-08-05 10:18:59] Loading raw image: done
[2017-08-05 10:19:00]


LOG C     -  Cut and Paste from 'Process Console'
Reading image:
T:/RawData/2017-06-25/Scorpius/d/RD_14388.PEF

Timestamp: Sun Jun 25 23:04:55 2017
Camera: Pentax K-5
Owner: ROB DIPPLE
ISO speed: 400
Shutter: 29.9 sec
Aperture: f/2.4
Focal length: 35.0 mm
Embedded ICC profile: no
Number of raw images: 1
Thumb size:  4928 x 3264
Full size:   4992 x 3284
Image size:  4950 x 3284
Output size: 4950 x 3284
Raw colors: 3
Filter pattern: BG/GR
Daylight multipliers: 2.155461 0.926686 1.305787
Camera multipliers: 19552.000000 8192.000000 11392.000000 8192.000000

Invoking: dcraw -4 -o 0 -w -q 1 -t 0
Decoding Pentax K-5 file (4950x3284 pixels, ISO=400, Exposure=29.93s): done
Loading raw image: done

Debayer: Global context

Demosaicing of 2 target files.
* Using 2 worker threads.
[000] T:/RawData/2017-06-25/Scorpius/d/RD_14388.PEF
[001] T:/RawData/2017-06-25/Scorpius/d/RD_14391.PEF

* Waiting for running tasks to terminate...

Loading target file:
T:/RawData/2017-06-25/Scorpius/d/RD_14388.PEF

*** Error:
*** Error: Unsupported or invalid CFA pattern ''

* Applying error policy: Continue on error.
Loading target file:
T:/RawData/2017-06-25/Scorpius/d/RD_14391.PEF

*** Error:
*** Error: Unsupported or invalid CFA pattern ''

* Applying error policy: Continue on error.
*** Error: No image could be demosaiced.
<* failed *>

[Alejandro Tombolini]

Hi Rob,

b and c use user specified debayer even if wrong.

if I understand correctly this is the expected behavior

b and c on auto miss-interprets the debayer and throws an error - *** Error: Unsupported or invalid CFA pattern ''. NB same happens when using Canon 1000D.

Lets wait for Juan opinion but may be useful if you can upload one image of each camera to test. 

Saludos, Alejandro.

[Juan Conejero]

Hi Rob,

This is not reproducible with our test images. Please upload one of these raw frames. I'll investigate and fix this issue as soon as I have data to make tests.

[RobD]

Hi
Alejandro - re 'expected behavior' - yes I agree, as I said the overall post was put together regarding 1.8.4. was in a round about fashion suggesting what has been implemented as 'Auto' in 1.8.5.

Juan - unable to upload pef and cr2 files due to size, hopefully you will be able to download them from my web site -  http://www.dipplestars.org.uk/PI.php

Thanks in advance

Rob

[Juan Conejero]

Hi Rob,

Thanks for the images. As expected, they work without any problem with the new Auto CFA pattern mode. Here is a copy of the process console:


Reading 1 image(s):
/Users/juan/Pictures/RD_14388.PEF

Timestamp: Sun Jun 25 23:04:55 2017
Camera: Pentax K-5
Owner: ROB DIPPLE
ISO speed: 400
Shutter: 29.9 sec
Aperture: f/2.4
Focal length: 35.0 mm
Embedded ICC profile: no
Number of raw images: 1
Thumb size:  4928 x 3264
Full size:   4992 x 3284
Image size:  4950 x 3284
Output size: 4950 x 3284
Raw colors: 3
Filter pattern: BG/GR
Daylight multipliers: 2.155461 0.926686 1.305787
Camera multipliers: 19552.000000 8192.000000 11392.000000 8192.000000

Invoking: dcraw -4 -o 0 -r 1 1 1 1 -D -t 0 -k 0 -H 1
Decoding Pentax K-5 file (14104672605014x3284 pixels, ISO=400, Exposure=29.93s): done
Loading raw image: done

Reading 1 image(s):
/Users/juan/Pictures/L_2016-11-28_20-27-27_0019_ISO800_300s__8C.CR2

Timestamp: Mon Nov 28 20:22:25 2016
Camera: Canon EOS 1000D
Owner: Rob Dipple
ISO speed: 800
Shutter: 300.0 sec
Aperture: f/inf
Focal length: 0.0 mm
Embedded ICC profile: no
Number of raw images: 1
Thumb size:  1936 x 1288
Full size:   3948 x 2622
Image size:  3906 x 2602
Output size: 3906 x 2602
Raw colors: 3
Filter pattern: RG/GB
Daylight multipliers: 2.362489 0.919100 1.182562
Camera multipliers: 2153.000000 1024.000000 1570.000000 1024.000000

Invoking: dcraw -4 -o 0 -r 1 1 1 1 -D -t 0 -k 0 -H 1
Decoding Canon EOS 1000D file (11175504908098x2602 pixels, ISO=800, Exposure=300.00s): done
Loading raw image: done

Debayer: Processing view: RD_14388
CFA pattern (detected): BGGR
VNG demosaicing: done
Noise evaluation: done
Gaussian noise estimates:
s0 = 1.908e-04, n0 = 0.3500 (MRS)
s1 = 1.906e-04, n1 = 0.4639 (MRS)
s2 = 2.128e-04, n2 = 0.5440 (MRS)
6.162 s

Debayer: Processing view: L_2016_11_28_20_27_27_0019_ISO800_300s__8C
CFA pattern (detected): RGGB
VNG demosaicing: done
Noise evaluation: done
Gaussian noise estimates:
s0 = 8.731e-05, n0 = 0.6198 (MRS)
s1 = 9.080e-05, n1 = 0.6353 (MRS)
s2 = 9.531e-05, n2 = 0.4945 (MRS)
2.783 s



Note in red color how the Debayer tool has detected the correct Bayer patterns automatically. The same works if I save the images as XISF files, then open them and apply Debayer.

Are you writing the images as FITS files? In such case, you're getting the expected behavior, since FITS cannot store the required image properties to support the new Auto CFA pattern mode.

[RobD]

Perhaps I should have given system info - so here it is.

Rob

OS Name   Microsoft Windows 7 Home Premium
Version   6.1.7601 Service Pack 1 Build 7601
Other OS Description    Not Available
OS Manufacturer   Microsoft Corporation
System Type   x64-based PC
Processor   AMD A8-7600 Radeon R7, 10 Compute Cores 4C+6G, 3100 Mhz, 2 Core(s), 4 Logical Processor(s)
BIOS Version/Date   American Megatrends Inc. P1.00, 07/11/2014
SMBIOS Version   2.7
Hardware Abstraction Layer   Version = "6.1.7601.17514"
Time Zone   GMT Daylight Time
Installed Physical Memory (RAM)   16.0 GB

[Juan Conejero]

Hi Rob,

OK, I know what happens. You are loading already debayered images:

Invoking: dcraw -4 -o 0 -w -q 1

Please go to File Explorer > DSLR_RAW (double clic), then clic Pure Raw in RAW Format Preferences. This will load monochrome CFA frames, which you can demosaic with Debayer. If you use the BatchPreprocessing script, all of this is done automatically and you don't need to change any format settings.

So there is no bug, just the expected behavior of the Debayer tool for already debayered images.

[RobD]

Juan,
many thanks - problem solved.

Just one thought though, would it solve future problems, for the easily confused like me, if the 'Pure Raw' setting was checked when using Auto?

Rob.

[VoidPointer]

Please go to File Explorer > DSLR_RAW (double clic), then clic Pure Raw in RAW Format Preferences. This will load monochrome CFA frames, which you can demosaic with Debayer. If you use the BatchPreprocessing script, all of this is done automatically and you don't need to change any format settings.

Little Typo here: It's Format Explorer not File Explorer. After realizing this, this also solved my issue with bias creation. Thanks!