Color Calibration

[dmcclain]

I am perplexed by color calibration, and I'm sure I'm not alone. When I do the canonical calibration using the core of M31 at high elevation to establish my white levels, I still end up with too much red emphasis, according to the writings of Richard Clark. ( www.clarkvision.com )

My peak star levels come out looking great, but the lower level nebulosities tend toward the red, instead of the magenta / orange indicative of H-alpha + H-beta (pink) and dust scattering (orange).

So I'm looking at the process. Background uniformization appears to subtract an amount in R, G, and B channels so that the average across the reference region is low value gray, plus some rescaling to keep from clipping low end data. The color calibration tool looks like it uses that uniform background reference as its estimate of the black level, and applies a multiplicative scaling in each channel when I'm using the manual white balance derived from the core of M31. So far so good.

The STF looks like it tries to make the background levels into a Gaussian with mean 0.25 and sigma about 0.08, so that the 0.25 mean level is 3 sigma above zero. This seems to be a distinctly nonlinear transform of the histogram, because no matter what the original histogram looks like, this makes it appear more or less Gaussian.

But the final result of a STF stretch to nonlinear looks like it does the right thing for bright star cores, and yet low-level nebulosity invariably comes out strongly red. The only way I can get my images to agree more or less with Richard Clark's view of nebulosity is to perform a mild curve boost to the mid-range blues.

Is it possible that STF is hosing up the lower levels and that a blue stretch is in fact a reasonable thing to do? And if so, by how much?

FWIW, I ran several trials where I used different galaxy cores as my white reference level, did the processing in separate CFA planes, SuperPixel deBayering, and VNG deBayering. The difference among these variations was around 3-5% in the manual color balance levels, and made essentially no difference to the reddish outcomes to my low-level nebulosities. So I am convinced that the overall color calibration using galaxy cores is correct. What concerns me is that nonlinear histogram stretch at the lower levels.

[pfile]

isn't RC really focused on DSLRs with stock filters? meaning, an IR cut filter which attenuates a whole lot of Ha signal.

i thought you were using a mono CCD with astronomical filters.

edit: sorry, an OSC CCD, not a mono CCD. the red response of an astronomical OSC setup is going to far exceed a stock DSLR.

rob

[dmcclain]

Yes, RC is primarily focused on DSLR's. But so what? These are Bayered CMOS sensors in most cases, but why would his conclusions for DSLR's be any different in our CCD domain?

What RC states, in fact, is that DSLR's with their red-cut filters are closer to our reality than red-modified cameras - that H-alpha shouldn't be blood-red, but rather more pink to magenta from the combined H-alpha / H-beta, and that the dust should be orange not dark red. I have a hard time disagreeing with his conclusions.

What I'm using at the moment is a Bayered ATIK-490 (Sony 814) sensor. This is normally quite red starved on my Tucson skies, compared to green and blue channel from sky background. I'm using an IDAS LPR filter to cut much of that light pollution, and it gives a greenish cast to raw images. But even my Canon 6D through the IDAS LPR shows strong blue, and little red.

When I calibrate the images from the Canon 6D using M31 core as my white reference, I do appear to be red starved in the nebulae, but still not enough blue. With the ATIK CCD I have much more red, too much, and not enough blue.

I really cannot disagree with the stated principles of color calibration in PI. What I'm beginning to question is the notion that transforming low-end histograms to a Gaussian is the correct thing to do. That seems a disjoint concern to the PI color calibration philosophy.

[pfile]

so what is that i don't think your camera has an IR cut filter like the DSLRs have, so you're starting off at a different baseline from RC. if he's against removing the IR cut filter from a DSLR (because it boosts Ha response) then i don't see why he would approve of an astronomical OSC that lacks such a filter.

LP filters do cut out a lot of light, and at least with the CLS filter i own, imaging galaxies is an exercise in frustration. there's just too many wavelengths missing. i have never imaged through an IDAS but looking at the spectrum on hutech's site, it does seem to pass a little more light than the CLS. at any rate the LP filter could have something to do with the color calibration results you are seeing. you might try without it and see what things look like.

STF is STF; while people do use the STF as a kind of an autohistogram transformation, a calculated STF applied to the Histogram tool is not necessarily part of the PI color flow or PI workflow. regardless, STF is supposed to be a nonlinear transformation (how else can you view linear data with your eye?)... so i'm not sure why what's it's doing is incorrect.

are you running STF with the channels unlocked? after color calibration you should make sure the channels are locked or STF is going to hose up your careful calibration.

rob

[dmcclain]

Yes, I am running the STF with channels locked after color calibration.

I did some tests on dark background regions and found that the normal color calibration really does attempt to produce a low intensity, color-aligned, Gaussian. And that seems totally reasonable. I would expect Gaussian-like behavior there, with no preference to individual color channel.

So what I'm now focusing on is the linear scaling in color calibration. We know that the human eye does not respond linearly. In fact it behaves much more like our hearing, which exhibits cube-root compression with increasing loudness. Our eyes respond similarly. I think it is in the nature of nerve stimulation. And I suspect that is the origin of the notion of Gamma in video processing.

From what I understand of color calibration in PI, it should not matter what biases in offset and slope we start off with in each color channel. The end result should be the same in any event. And that end result is just too red at the lowest levels.

[pfile]

well there's still the issue of the LP filter and the fact that you're trying to make a white reference from something that's definitely missing a lot of wavelengths. the astrodon RGB filters have a pretty big gap between G and R, and color calibration is still possible when using those filters... but i think the CLS just kills too much light. the IDAS could be different, but anyway it seems like eliminating the IDAS could at least be instructive.

anyway yeah gamma is all about human light perception; a CCD responds linearly to photon flux while our eyes have a log response. more than you ever wanted to know here: http://www.telescope-optics.net/eye_intensity_response.htm

rob

[pfile]

oh also here is an old thread that might shed some light (ha) on the subject:

http://pixinsight.com/forum/index.php?topic=2542.0

[dmcclain]

Interesting thread. Thanks. And I'll go back and integrate over the entire M31 image instead of just the core to see if / how much difference that will make.

You also have me curious about the unfiltered response on my camera, without the IDAS LPR filter. That should be interesting.

But just the same, I had understood that calibration resolves differences in the range between the peak intensity levels and a reference background level. So even if you scale a color channel by some wildly incorrect value beforehand, the calibration removes that nonsense and rescales so that the peak white values remain white. Hence it should not matter what base level offsets and sensor sensitivities you have in each color channel, as long as the color components can add up to white upon proper scaling.

[dmcclain]

Hmmm... I just found a potential problem originating in the calibration of raw frames. I did a manual calibration on a single raw frame. During that process I noticed that the raw superpixel debayered frame has a green hue, which arises from the IDAS LPR filter. I have seen that green hue many times when setting up the camera on daylight scenes. That is expected.

Furthermore, when flat fielding with a Bayered image, the color balance is shifted strongly to the red, presumably because the red cells have such little light in them. I am generally red starved on the sky.

But that presents a problem in that the color balance is being shifted during the calibration, and that shift will depend on the flat frames used during the calibration. Hence a color calibration established on an image stack of M31 on one date will likely have no correct relationship to an image stack taken at some later date with different flats.

What I found during my manual calibration was that the R/G/B scaling for color calibration on M31 was 1.0/0.65/0.98, or just about what you'd expect from a 1/2/1 Bayer matrix. (plus a green bias from the IDAS LPR filter)

In contrast to that, what I generally find from normally batch calibrated frames is an R/G/B color balance of 0.47/0.66/1.0.

Seems to me that flat fielding should be color neutral, and it clearly isn't.

[Geoff]

Have you tried using the whole of M31 (and not just the core) as the white reference?

[pfile]

i always had problems with the red channel when using the CLS and a modified 50D. eventually i started using a slightly pink t-shirt so that the flats came out grey. in theory this should not be necessary since flat scaling is done independently on each channel, but of course if the SNR is super low in the red channel of the flats, you're probably really messing up your calibrated data.

so you might try shifting your light source one way or another to see if you can boost the red in the flats.

rob

[dmcclain]

Yes, I just tried using the whole of M31 instead of only its core. It only made < 10% difference to the outcome.

I agree that the SNR in the red channel is low, even in the flats for this particular era. I just tried several alternate ways of calibrating, using deBayered flats, both SuperPixel and VNG, as well as first converting those deBayered flats to gray scale and back to RGB for calibration.

Using these alternate flats does not shift the color balance grossly, although some small shift will be observed as the histograms narrow after flatting. The aim of the flats should be to remove sensor fall-off toward the edges and corners. I do not understand its inverting the relationship of the R, G, and B histograms. That inversion does not happen with my modified flats, and I wouldn't have expected it to.

The degree of histogram rearrangement using Bayered flats will cause problems trying to connect later era frames to the galaxy calibration set. What we need is consistency.

However, that being said, I find the outcome to be remarkably little changed from the default processing.

One very confusing situation arose when processing the CFA planes separately in anticipation of proper MURE denoising. I have Bayer cells that need to be deBayered normally as GRBG. When I split the planes using SplitCFA it ends up numbering them such that CFA0 = G, CFA1 = B, CFA2 = R, and CFA3 = G. Reassembly then requires a GBRG-like sequence in PixelMath.

But even more confusing at first, the results of MURE denoising shows CFA0 = 7, CFA1 = 28, CFA2 = 31, and CFA3 = 42 DN noise levels. From those figures you might assume that CFA0 = R, CFA1 & CFA2 = G, and CFA3 = B. But no. I cannot understand why the G channels (CFA0 & CFA3) have such widely differing noise values of 7 and 42, while the B & R channels have nearly the same noise levels around 30 DN.

[dmcclain]

I just did some trial runs with various Bayered flats. I think I understand why the histogram inversion happens. It appears that the calibration routine is normalizing by the median or mean of the flat, and dividing by the individual pixels in the flat. Hence, when the R channel is starved it gets boosted in the output. The calibration doesn't seem to know or care that we have a CFA sensor.

I guess as long as there is a consistent ratio between the R, G, and B levels in the flats, the outcome will be consistent.

[dmcclain]

I found the problem with MURE denoise. My ATIK camera puts out frames of 3379 x 2703 pix. The SplitCFA cannot deal with odd numbers of rows or columns, so I had been cropping 1 pixel extra outward in both dimensions. The resulting blank rows caused too much variance for MURE denoise.

By cropping inward by 1 pixel in each axis, MURE denoise gave much more consistent results among all channels, and in fact, they nearly all had the same noise level.

But after all this, we are still much too red. I have some spectra taken when the LPR filter was in place, and it shows very clearly ( http://www.astrobin.com/full/236857/0/ ) that H-alpha and H-beta are right in the midst of the pass band. And if anything, the green is somewhat starved. So H-beta ought to be shining through like gangbusters. Yet I am incredibly red no matter what I do unless I artificially boost the mid-range of Blue with a simple curve boost of 0.5 at the midpoint.

[pfile]

well keep in mind that Hb emission is something like 20% of Ha emission, give or take. it's a much less likely transition.

you are probably right about the flat normalization, so that's an argument for trying to make your flats white when using LPR filters.

rob