Colors not correct after Flatdivision

[Simon Großlercher]

Fellow PixInsight-Friends!

I'm having a problem with processing my data. First off all, I'm on PI version 1.7 and I take my astroimages with a Full Spectrum Modded 600D.

On the 18th of January I exposed 4 hours on M81 M82 - weather was just ok. I also took a set of flats and bias at ISO 200 (also the setting for my lights) for calibration.
It was the first light with my new camera.

I put everything into the BatchPreprocessing script, set it up correctly and started it. The final stack already looked a bit "yellowish" and like blue color would be missing. My guess was confirmed in further steps of processing:



I used both the LinearFit and the ColorCalibration process for color calibration. RGB-channels are nicely aligned to each other, though the image is lacking in blue/purple.
I have quite some experience with the Software and I'm aware of how to use most of the processes properly.

I decided to ignore the vignetting and stack my lights without flats. This time the Masterlight looked very different straight away. I could tell that the colors were correct immediately. And they were:



I tried different DSLR_RAW settings in the Format Explorer  (CFA monochrome input, no blackpointcorrection) ; (VNG interpolation, no blackpointcorrection); (VNG interpolation, blackpointcorrection) but basically the final stacks all looked the same when I had calibrated my lights with flats. I am very sure there is nothing wrong with the Format Explorer settings either.

So it looks like the cause of the problem is the flat. Here is a (properly exposed) flat I used for calibration (8192 Adu):



I can also upload a RAW, if that helps. The flat panel I'm using is a high quality Aurora Flatpanel from Gerd Neumann.


I posted this problem in a german Astronomy forum in hope for a solution but nobody was able to fix my problem. Really hope you can help me I'm stacking now since 4 days, I also tried 10 different flat-sets with different exposures and of course I also did the whole Data Reduction manually without the script... - but same result.

Cheers,
Simon

[Simon Großlercher]

Here is the Download-Link for a Flat RAW-file:

https://www.dropbox.com/s/31p7e8ueroyo82r/RAW_Flat.CR2

[pfile]

how did you measure the flat's ADU count?

i don't have any answers for you but i have struggled with the same thing using a canon 50D. all i know is that the histogram of the flat looks very different depending on how it was processed.

for instance, if i set DSLR_RAW to do RAW and no black point correction and debayer your file using the Debayer process, the histogram appears as in the first screenshot. if i set DSLR_RAW to debayer the file, it looks like the 2nd screenshot. of course, using DSLR_RAW to debayer the file makes no sense in an astronomical processing context, but i show that to say that the same data can have wildly different histograms depending on what tool was used to process the CR2 file.

in the end i did this. i exposed a flat until it was completely overexposed, then opened the file with DSLR_RAW set to RAW+no black point, debayered with Debayer. inspecting the histogram shows what the full-well capacity looks like. i then exposed my flats long enough so that the histogram was 1/2 of the way to full-well saturation. only when i started doing this did my calibration work properly.

so, i suppose it is possible that your flats are just underexposed. you might try the above process and see what happens.

[Simon Großlercher]

Yes, all you are saying is correct - the histogramm looks different depending on how you import the RAW.

But I think its the monochrome 14bit CFA image which counts. At least that's what a very experienced guy in the german forum said. He also said you could never reach 50% of the histogramm (undebayered) with a 14Bit DSLR because that would equal 15bit. So he recommended exposing up to a value of 0.1250 on the Histogram which is equal to 13bit and perfect exposrure for a 14bit DSLR Flat.

[pfile]

this is correct, you can't reach 50% due to the 14-bit nature of the data.

i looked at some of my old flats, and although i thought that they had histograms farther to the right than yours, in fact they are almost the same.

here is another thing - did you use a CLS or IDAS filter? i see the same problem as my flats - the red channel is comparatively underexposed.

while in theory the color balance of a flat does not matter, in practice (with my 50D anyway) the red channel was underexposed enough that i had some camera artifacts in the red channel (like banding). toward the end of my DSLR career i started using a slightly pink T-shirt to make flats so that each channel would be similarly exposed.

[Simon Großlercher]

Hi there,

So actually it seems like you had the exact same problem.

I just found a process in PI called AssistedColorCalibration, which allows you to apply a manual whitebalance using a preview. Works pretty well!

Still I would appreciate a proper solution (within Software) for the root problem ! Thanks for the pink Tshirt idea, that was a start!

Cheers,
Simon

[pfile]

well, to be clear i'm not sure i've ever had a problem with color calibration that was caused by flats. it was more that the calibrated files had a lot of noise in the red channel. since it is a modified camera that is the last thing i'd expect.

but since you are using an EL panel it might be a little harder to tweak the color of the flat...

[Ignacio]

I came across a function in fitswork (free software), that balances the RGB channels of a fits master flat, returning the same CFA fits format.

I tried it, and it works fine, avoiding the color cast after flat division. SNR seems to be preserved too, when doing it with or without color balancing the master flat. I was hoping to see an improvement, thou.

I don't think I will use it, since I don't have problems balancing colors even when the master light is strongly unbalanced due to flat division.

I use my laptop's led screen as a flat panel, since the aperture of my imaging scope aperture is small enough to fit in the screen (5 inches). What I have done lately, is adjust the color balance of the flat image on the screen, so that the modded camera sees similar ADU values in all channels.

Ignacio

[Christoph Puetz]

Hi  - we had exactly the same problem with a modified Canon DSLR and the reason that you explaned is quite true  8)
We also found a workaround for this:

Before you start the preprocessing, do the following steps only with your flat frames:

1. Convert them from RAW to FITS with the BatchConversion Tool
2. Debayer them
3. Convert them to Grayscale

Use the converted grayscale flat frames as input to your preprocessing as usual !

We also found that (esp if we use a flatfield utilitym i.e. from Geoptik or similar) the median value of the flat (scaled to the range from 0...1)
is good around the value 0.125.

Hope this helps

Christoph (and friends from the Peterberg Observatory).

[Phil Leigh]

I don't see how this could work correctly because you are debayering BEFORE calibrating the flat frames with Bias/Darks...? Calibration absolutely requires undebayered RAW data.

[Christoph Puetz]

Dear Phil,

ok. Maybe this is not the correct solution for this topic. As I saw in the history of this thread, the proper solution is still
missing.
Our workaround indeed worked for us and we had very satisfiying flats after that (with all traces of dust and vignetting).

Can you give us a proposal for the correct solution ?
Any help would be appreciated.

[Phil Leigh]

Hi Christoph,

For an OSC image you need to to bias and/or dark subtraction from each flat (i.e. calibrate the flats) before integrating them to create a master flat. The master flat must be divided into the light frames before any debayering takes place in order to ensure that the pixels in the flats and lights correspond exactly. Debayering loses the original luminance data of each Red Green or Blue pixel in order to create an RGB value for each one.

If you debayer either the flats or the lights or both before flat division then the results will scientifically be incorrect. They might still look OK visually...

The original problem in this thread was that it appeared that a flat with a colour cast was influencing the colour balance of the flattened light frame.

Quite how this can happen is a mystery to me. The flat division works on each RGB channel independently. A strong colour cast on the flats would give a different overall median level for each channel - but the channels are individually normalised before the division so in theory the colour balance of the original light frame should remain unchanged.

I'm going to conduct some experiments to find out what is really happening... I'd like to answer this once and for all.

[pfile]

i don't think that the 3 channels are normalized during flattening… i think the statistics for each channel are computed independently, and 3 scaling factors are computed independently. since the scaling factors can be different depending on the signal strength in each channel, the final image can have a color cast.

but some experiments are always in order

rob

[Christoph Puetz]

@pfile and Phil
Thanks for your support.
Here are some original files for testing and download (> 550 MByte!).
https://sd2.1und1.de/guest?path=flat_test%20von%2073306335&token=EB97084C110F2804&mandant=06&locale=de&viewType=0

Klick on the above link and accept "1&1 Smartdrive Gastzugang".
Then download the file "testpics_flat_colour.7z"

This is a 7-zip compressed file with the following content:
8 Lightframes/Darkframes/Bias and Flatframes from a Canon 1100D DSLR

I hope this will help to clarify this issue.

Kind regards !

[Phil Leigh]

So... I've just take a well-exposed but rather blue flat  from an EL panel and made it very red (using assisted colour calibration to drop the green and blue to 0.5 and the red to 1.5) then calibrated a light frame with both versions...

There is no difference to the colour balance of the light frame with either flat and the calibration has worked perfectly in both cases.

What else can I try to prove that flat colour balance does not affect light frame colour balance?