Debayer Generating Magenta Images

wrichards

Active member
Jun 22, 2020
33
0
I'm still pretty new to PI, but I've used it to stack images from 7-8 sessions and have never had a problem with the Debayer process, which is pretty straightforward. But I must have changed something and now I'm getting a strong magenta cast to all my debayered images (see attached example). The calibrated lights that I'm feeding into the Debayer process are fine.

The images were shot with a Canon 6D Mark II. I've selected RBBG for the Bayer/mosaic pattern. In RAW Format Preferences, I've tried this using "Pure RAW" (No white balance) and Camera white balance - same results.

What's odd is that I processed this same stack earlier in the week and the output was fine - I don't know what I changed. Does anyone know what I messed up?
 

Attachments

fredvanner

Well-known member
Apr 17, 2019
79
4
Wells, Somerset, UK
I'm puzzled if it's worked before with the same settings. "RBBG" is not a valid matrix - I guess this is a mistype for "RGGB". The mosaic pattern depends on the "offset" of the pixel matrix; if this was the first time you had tried, I would reccommend trying each of the eight debayer matrix options (starting with "RGGB"), until you find the "right" one for your camera / software mix. Since you're using a DSLR, you can check the debayer settings by debayering an ordinary daytime terrestrial photograph.
 

wrichards

Active member
Jun 22, 2020
33
0
I'm puzzled if it's worked before with the same settings. "RBBG" is not a valid matrix - I guess this is a mistype for "RGGB". The mosaic pattern depends on the "offset" of the pixel matrix; if this was the first time you had tried, I would reccommend trying each of the eight debayer matrix options (starting with "RGGB"), until you find the "right" one for your camera / software mix. Since you're using a DSLR, you can check the debayer settings by debayering an ordinary daytime terrestrial photograph.
Yeah, sorry - I meant RGGB which is the correct pattern for a Canon DSLR. It's worked for me every time prior to now and I just don't know what I changed to cause this.

I stacked the same images using DSS and the result came out fine.
 

wrichards

Active member
Jun 22, 2020
33
0
Terrible - I attached an example in the very first post. The un-debayered calibrated frames look fine.
 

fredvanner

Well-known member
Apr 17, 2019
79
4
Wells, Somerset, UK
Ah! I hadn't realised that was a single frame. What sort of STF was applied for that image (channels linked or unlinked). My raw images often have a strong colour cast with a linked STF, but look quite reasonable with an unlinked STF. It might be worth seeing what PCC makes of one of your frames - these days it has become my "first stop" for colour balance if the image is good enough (it has to plate solve the image) - but yours looks fine.
 

wrichards

Active member
Jun 22, 2020
33
0
That was just an auto-stretched image. But attached is a screen capture of the unstretched, Debayered image - you can clearly see there is a magenta cast even without a stretch.
 

Attachments

fredvanner

Well-known member
Apr 17, 2019
79
4
Wells, Somerset, UK
Hmm! I'd expect an unstretched image to be virtually black. Could you post a link to the original raw image file for a single frame? I might understand better if I can see the actual file.
 

wrichards

Active member
Jun 22, 2020
33
0
Ha! Anticipating this request, I had already uploaded the Master SuperBias, Master Dark, Master Flat, and a calibrated Light frame HERE.

BTW - I appreciate you taking the time to help me resolve this.
 

bulrichl

Well-known member
Nov 2, 2016
728
41
La Palma, Canary Islands
Obviously the dark frames (or the integrated dark frames) were precalibrated with the superbias - this is not recommended. The MasterDark is heavily clipped (in Statistics: count (%) = 86.5 %, i.e. 13.5 % of the pixels in the MasterDark are clipped).

The MasterFlat (mean about 1200 ADU) shows that the flat frames were underexposed. For the uncalibrated flat frames in CR2 format, a mean target value of 7500 ADU is appropriate.

The flat field correction is strongly overcorrecting.


Recommendations:
1. How did you prepare the MasterFlat? Did you calibrate the individual flat frames with the superbias only and then integrate the calibrated flat frames to the MasterFlat? If not, you should redo the MasterFlat.

2. Delete your current MasterDark.

3. Integrate the dark frames.

Use this new MasterDark (and if applicable the new MasterFlat) for the calibration of the light frames.

4. Calibrate the light frames with one of the following set of parameters:
a) with dark frame optimization
Enable section 'Master Bias', select the MB, disable option 'Calibrate'.
Enable section 'Master Dark', select the MD, enable both options 'Calibrate' and 'Optimize'. Set 'CFA Pattern Detection' to 'Force CFA'.
Enable section 'Master Flat', select the MF, disable option 'Calibrate'.

b) witout dark frame optimization
Disable section 'Master Bias'.
Enable section 'Master Dark', select the MD, disable both options 'Calibrate' and 'Optimize'.
Enable section 'Master Flat', select the MF, disable option 'Calibrate'.

Compare the calibration results (with / without dark frame optimization) and in future use the approach which yields the superior result.


The Bayer pattern 'RGGB' is correct for the Canon EOS 6D MarkII. When using frames in CR2 format, you can leave the default setting in the Debayer process, 'Auto'.

Bernd
 

fredvanner

Well-known member
Apr 17, 2019
79
4
Wells, Somerset, UK
Health warning: I am a raw beginner at astrophotograpy - most of my images have been taken this year during COVID lockdown. I do have degrees in maths, and a background in signal processing (radar; acoustics), but astro imaging is a new hobby. This being understood...
Those files are useful, but I also need the raw image (before calibration), in case the problem is in the cal process.
I note that the bias has already been subtracted from the master dark - just need to make sure we only do this once.
The calibrated image looks "flat overcorrected" with a sort of "reverse vignette" effect. - can you recall exactly what your calibration workflow was?
Using PixelMath to "back off" the flat correction, followed by a standard RGGB debayer and an autostretch, I got the attached image. I suspect that more careful flat correction will do better with the remaining vignette effects. I rather suspect that the standard PI workflow sometimes gets OSC images wrong - but PI also has all the tools to fix the problems!
 

Attachments

wrichards

Active member
Jun 22, 2020
33
0
Thanks for the detailed insights.

I followed this LightVortex tutorial for creating the 3 master files. When I used them on these very same files earlier in the week, they yielded normal-looking images. When I went back to re-process this stack to try a few different things, the Debayered images suddenly showed up with this strong magenta cast. So I'm really baffled.

A couple questions:

1) Is the LightVortex tutorial incorrect? It instructs you to use the Master SuperBias to create the Master Dark, but you guys seem to think that's a bad idea. I'm too much of a newbie to know (which is why I followed the LightVortex tutorial).

2) Where are you getting the stats on my images and what values should I be looking for in each master? I'd like to be able to use that information in the future.

I uploaded one of the light frames to the Google Drive site. Note that this particular stack of lights was intentionally underexposed using a step-down ring to reduce the aperture without introducing refraction spikes so they can be used to create a lower intensity layer of mostly stars. Then after processing the nebula (from Starnet++), I'll add this star layer back in.

I don't mind starting over from scratch to recreate the masters, but I don't understand why they worked a few days ago and don't work now.

P.S. Fred - I also have a degree in digital signal processing, but mine was mostly applied to sonar back in the 80's and early 90's. I did a little bit of radar, but not much. And I only started doing astrophotography about a year ago. I've only been using PI for about a month, so I'm very green in that regard.
 

wrichards

Active member
Jun 22, 2020
33
0
> BTW - it really is a very nice image!

The actual stack of lights is much better, as they were captured without the step-down ring and with twice the exposure time. I just uploaded 2 JPEGs resulting from different processing steps. In one, I got a lot of blue but very little red. In the other, I managed to get some red, but lost most of the blue. I'm trying to find a way to retain both!
 

fredvanner

Well-known member
Apr 17, 2019
79
4
Wells, Somerset, UK
OK, we now have .CR2 (debayered colour image) file. All we are missing is a raw, unprocessed greyscale image file (i.e. the file to which greyscale calibration has yet to be applied).

PS: although I've done many things since I left university in 1972, I've been working for Thales UK underwater systems (historically Plessey) for the last 10 years as a maths consultant for sonar processing. Astro imaging brings up all the same issues - at the end of the day it all comes down to SNR!
 

bulrichl

Well-known member
Nov 2, 2016
728
41
La Palma, Canary Islands
The Lightvortex tutorial about preprocessing recommends to calibrate the dark frames with the superbias and then integrate the calibrated dark frames to the MasterDark. It is not unusual that clipping occurs with this approach. The better approach is, to simply integrate the dark frames (like you did with the bias frames), and thereby usually clipping is totally avoided.

The Lightvortex tutorial also recommends to calibrate the flat frames with MasterDark and superbias. In my view that is not good either. As I wrote I would calibrate the flat frames with the MasterBias (or superbias).

Important note: when you decide to NOT precalibrate the dark frames, also the settings for the calibration of the light frames have to be altered compared to the Lightvortex recommendation, see my post #10, point 4a) and b).

The statistics of an image are shown by the Statistics process (PROCESS/Image/Statistics). Leave the default for the option 'Unclipped', i.e. disabled. Clipping can also be detected in the histogram of an image (PROCESS/IntensityTransformations/HistogramTransformation). The horizontal zoom should be increased to 40 - 100 in order that the region at zero can be inspected closely.

The strong color cast is probably caused by the flat field correction, but that is not alarming. In ScreenTransferFunction (STF) please disable the option 'Link RGB color channels'. Then the color cast should be gone. Later on in the workflow (after integration of the calibrated light frames), a color calibration (e.g. with PhotometricColorCalibration) is necessary. Thereafter the option 'Link RGB color channels' can be enabled in STF again.

Bernd
 
Last edited:

fredvanner

Well-known member
Apr 17, 2019
79
4
Wells, Somerset, UK
Apologies for my last post. I've just found out that the .CR2 file you posted has got the raw data - its just that my default PI preferences for this format automatically debayered the file when opening it!
 

wrichards

Active member
Jun 22, 2020
33
0
Bernd - Thanks for the excellent advice. Later this weekend, I'll start from scratch and try your suggestions. Maybe that will help retain some of the color that I seem to be missing in the final images. But I don't think the color cast is from STF, as it is readily apparent in the image even without using STF. It's not there in the calibrated XISF image, but is definitely there in the Debayered XISF image.

Fred - The CR2 image is the raw image from the camera - there is no other unprocessed image file. The RAW images are calibrated with the master files, then registered and stacked.

PS: Small world. I worked with Thomson-Sintra (Paris) and FTSS (Manchester) on the ALFS (AN/AQS-22) dipping sonar system back in the early 90's. Probably a competitor of Plessey at the time.​
 

wrichards

Active member
Jun 22, 2020
33
0
Apologies for my last post. I've just found out that the .CR2 file you posted has got the raw data - its just that my default PI preferences for this format automatically debayered the file when opening it!
Ah, that explains it. You have to check that box for "Pure Raw" in the RAW Format Preferences...