Author Topic: Flat frame colour influencing colour of calibrated frames  (Read 2264 times)

Offline SteveJP

  • Newcomer
  • Posts: 10
Flat frame colour influencing colour of calibrated frames
« on: 2018 November 04 21:51:21 »
When processing OSC images with ImageCalibration to counter the effects of Bias, Noise and Non-Flatness, I find that the colour of the calibrated image is influenced by the colour of the flat frame. This seems most obvious if the flat frame is a blue Sky flat. (I'm debayering after calibration which I believe is the right thing to do.)

Why is the colour of the flat frame taken into account by the ImageCalibration process. It would seem unnecessary and/or undesirable?

As an experiment, I debayered my master flat and then used that in ImageCalibration of raw frames. This seemed to work and the colour (after debayer) didn't appear influenced by the flat frame colour.

What is the correct procedure? Is it correct to debayer the master flat? Could I just extract its luminance and use that perhaps?

I'm sure there is a well known answer for this probably naive question. Thanks

Offline bulrichl

  • PixInsight Guru
  • ****
  • Posts: 524
Re: Flat frame colour influencing colour of calibrated frames
« Reply #1 on: 2018 November 05 01:47:36 »
The correct approach for OSC cameras is to do the calibration with monochrome CFA images and debayer the calibrated (and if applicable CosmeticCorrected) images just before image registration.

The color shift that you observe is normal, it results from different intensities of the channels in the MasterFlat. A weak channel in the MasterFlat will amplify, a strong channel in the MasterDark will attenuate the corresponding channel of the light frame. However, this is unimportant. The color balance has to be accomplished in a separate step, by ColorCalibration (e.g. BackgroundNeutralization and ColorCalibration or PhotometricColorCalibration). If you want to evaluate the integration before ColorCalibration, apply a STF auto stretch with unlinked RGB channels.

Bernd

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
Re: Flat frame colour influencing colour of calibrated frames
« Reply #2 on: 2018 November 05 09:28:18 »
Hi Steve,

Bernd is absolutely correct when he describes how each individual channel in the CFA (Bayered) OSC image is directly affected by the corresponding channel in the CFA Flat Frame.

I go so fed up having to then tackle the extra Colour Correction caused by the resultant colour cast that I started to look at the issue from a different perspective.  First, we have to remember that Flat Field Correction relies on'division' of the Light by the Flat, and so we need to get the divisor to be as close to maximum as possible. However, we cannot over-expose the Flat frame, otherwise we get a saturate, and therefore non-linear image - which is not what we want.

So, we avoid saturating the image, but that leaves us with a non-maximal divisor. But the non-maximal divisor is a penalty that we have to pay, and which doesn't really affect us because PixInsight is quite capable of retaining numeric accuracy after the division because it natively works with 'real numbers' (Float variables - at up to 64-bit resolution, if you so wish).

And, if our imager had been a monochrome device, our work here would be done!

But, we effectively have a separate imager for each of the colour channels, where we can think of the 'gain' of each of the channels being directly related to both the colour of the incoming light and the sensitivity of a given channel to that colour of light. Even 'perfect white' light is likely to have a different effect on the pixels of each CFA to a different extent.

My first approach to the problem was to split the Master Flat Frame image into images representing each of the individual CFA filters. Using PixelMath, I then subtracted the maximum ADU value of each of the images from 1.0, and then added the result back to all the pixel ADU values for that image. This gave me an 'additively maximised' result for each channel. I then recombined these new images to give me a Master Flat Maximised image for use in subsequent calibration stages.

This worked, with some success, but still left a colour cast that needed to be eliminated during Post Processing - something that became easier to accomplish as PixInsight improved over the years.

This made me realise that the source of the problem was actually the 'colour' of the light used to expose the Flats - often too blue when using sky-flats, or too red when using artificial light. I needed a 'variable-colour' light source, and decided to try the now more readily available RGB leds, where each LED colour, and intensity, could be set, and memorised, by a master controller - on an LED by LED basis, if required.

I could finally now visualise the Histogram peaks more or less in real-time, and could tweak the LED controls to get the channels to line up, and to also give me a non-saturated ADU reading for my chosen exposure time.

Of course, all of this may be un-necessary detail for some, but it was a challenge that I enjoed tackling. Hopefully my ideas may inspire someone.
« Last Edit: 2018 November 06 16:03:11 by Niall Saunders »
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline sharkmelley

  • PTeam Member
  • PixInsight Addict
  • ***
  • Posts: 241
    • Mark Shelley Astrophotography
Re: Flat frame colour influencing colour of calibrated frames
« Reply #3 on: 2018 November 05 11:42:14 »
The first step I perform after stacking my DSLR images is to open the master flat and measure the RGB values at the centre.  I then scale RGB channels of stacked image using PixelMath to undo the colour mangling before I apply the camera's daylight white balance.

Definitely don't try to use a debayered master flat for image calibration of all your light frames.

The other thing that sometimes alters the colour balance of a DSLR image is Normalization in ImageIntegration.  I always switch this to "No normalization".   This option is in the "Image Integration" section.   The normalization in the "Pixel Rejection" section should be left switched on.

Mark
« Last Edit: 2018 November 05 22:18:35 by sharkmelley »
Takahashi Epsilon 180ED
H-alpha modified Sony A7S
http://www.markshelley.co.uk/Astronomy/

Offline SteveJP

  • Newcomer
  • Posts: 10
Re: Flat frame colour influencing colour of calibrated frames
« Reply #4 on: 2018 November 13 01:20:16 »
The correct approach for OSC cameras is to do the calibration with monochrome CFA images and debayer the calibrated (and if applicable CosmeticCorrected) images just before image registration.

The color shift that you observe is normal, it results from different intensities of the channels in the MasterFlat. A weak channel in the MasterFlat will amplify, a strong channel in the MasterDark will attenuate the corresponding channel of the light frame. However, this is unimportant. The color balance has to be accomplished in a separate step, by ColorCalibration (e.g. BackgroundNeutralization and ColorCalibration or PhotometricColorCalibration). If you want to evaluate the integration before ColorCalibration, apply a STF auto stretch with unlinked RGB channels.

Bernd
Thanks Bernd - I certainly want to use the correct approach. Thanks for the explanation.

Offline SteveJP

  • Newcomer
  • Posts: 10
Re: Flat frame colour influencing colour of calibrated frames
« Reply #5 on: 2018 November 13 01:24:01 »
Hi Steve,

Bernd is absolutely correct when he describes how each individual channel in the CFA (Bayered) OSC image is directly affected by the corresponding channel in the CFA Flat Frame.

I go so fed up having to then tackle the extra Colour Correction caused by the resultant colour cast that I started to look at the issue from a different perspective.  First, we have to remember that Flat Field Correction relies on'division' of the Light by the Flat, and so we need to get the divisor to be as close to maximum as possible. However, we cannot over-expose the Flat frame, otherwise we get a saturate, and therefore non-linear image - which is not what we want.

So, we avoid saturating the image, but that leaves us with a non-maximal divisor. But the non-maximal divisor is a penalty that we have to pay, and which doesn't really affect us because PixInsight is quite capable of retaining numeric accuracy after the division because it natively works with 'real numbers' (Float variables - at up to 64-bit resolution, if you so wish).

And, if our imager had been a monochrome device, our work here would be done!

But, we effectively have a separate imager for each of the colour channels, where we can think of the 'gain' of each of the channels being directly related to both the colour of the incoming light and the sensitivity of a given channel to that colour of light. Even 'perfect white' light is likely to have a different effect on the pixels of each CFA to a different extent.

My first approach to the problem was to split the Master Flat Frame image into images representing each of the individual CFA filters. Using PixelMath, I then subtracted the maximum ADU value of each of the images from 1.0, and then added the result back to all the pixel ADU values for that image. This gave me an 'additively maximised' result for each channel. I then recombined these new images to give me a Master Flat Maximised image for use in subsequent calibration stages.

This worked, with some success, but still left a colour cast that needed to be eliminated during Post Processing - something that became easier to accomplish as PixInsight improved over the years.

This made me realise that the source of the problem was actually the 'colour' of the light used to expose the Flats - often too blue when using sky-flats, or too red when using artificial light. I needed a 'variable-colour' light source, and decided to try the now more readily available RGB leds, where each LED colour, and intensity, could be set, and memorised, by a master controller - on an LED by LED basis, if required.

I could finally now visualise the Histogram peaks more or less in real-time, and could tweak the LED controls to get the channels to line up, and to also give me a non-saturated ADU reading for my chosen exposure time.

Of course, all of this may be un-necessary detail for some, but it was a challenge that I enjoed tackling. Hopefully my ideas may inspire someone.
Thanks Niall for that very interesting narrative of your research. I can't quite see myself going to that extreme in the near future, but I'm greatly impressed with the method you describe.
Rgds
Steve

Offline SteveJP

  • Newcomer
  • Posts: 10
Re: Flat frame colour influencing colour of calibrated frames
« Reply #6 on: 2018 November 13 01:26:19 »
The first step I perform after stacking my DSLR images is to open the master flat and measure the RGB values at the centre.  I then scale RGB channels of stacked image using PixelMath to undo the colour mangling before I apply the camera's daylight white balance.

Definitely don't try to use a debayered master flat for image calibration of all your light frames.

The other thing that sometimes alters the colour balance of a DSLR image is Normalization in ImageIntegration.  I always switch this to "No normalization".   This option is in the "Image Integration" section.   The normalization in the "Pixel Rejection" section should be left switched on.

Mark
Hi Mark. Thanks for the explanation. I'll check my procedure to see if I've got the "normalization" turned on in image integration.
Rgds
Steve