Integration output levels don't match input files

[mcintyre_sj]

The output levels (mean) of the resultant integrations vary wildly from the average levels of the input files. Most of the time the red channel is close to the average of the input subs. The blue channel is slightly higher and the green channel mean is usually about 4.5x the average mean of the input subs.

I am using ImageIntegration with normalization "additive + scaling" , weights as "Noise evaluation" and Scale Estimator as "Iterative K-sigma" (IKSS).

For a recent attempt at imaging the Horse Head area, i had 16 DSLR subs. I calibrate everything using RAW CFA format, then debayer the calibrated lights and split the channels. From there i align everything in one go and then integrate the r-g-b channels separately. The average of the mean for the subs for each channel is 0.0683, 0.0653, 0.0667. The integrations for each channel have means of 0.0672, 0.3040, 0.2008.

Obviously the colour is going to be a little skewed if just took the integrated channels as is.

So what's going on here? Why are the levels of the integrations so different from the input subs?

[Niall Saunders]

Hi there,

I come across this problem all the time (if I have correctly understood your description***), and I associate it with the use of Flats in the CFA calibration process.

*** EDIT: Obviously, I didn't really understand your issues clearly, however I will leave this post in situ as I still believe it has some merit

The problem is less obvious when dealing with a Mono imager and line-filters (WB or NB) because each 'channel' is calibrated separate from the other channels. However, this is unfrtunately not the case for a CFA imager.

This time, the normal process is to calibrate 'raw' (or CFA-bayered) Lights with similar 'raw' flats - but in a single stage of image 'division' (rescaling back to the normal 0.0 -> 1.0 rabge used internally by PI as required).

However, in the four CFA 'channels' (typically one Red, two Green and one Blue - or one each of Cyan, Magenta, Yellow and Green if you are using a CMYG imager) each channel has its own 'peak' - these can be clearly seen if you look at your MasterFlat on the Histo window. And, when the 'division' process takes place (during the Flat Calibration procedure) the highest peak (closest in value to 1.0) will have less effect on the calibration output image than the peak with the lowest average value which, being the denominator of the division process will cause a significantly higher result than a higher-value denominator would.

In plain terms, those peaks on the left of the Histogram will cause equivalent colour casts after Flat Calibration (once the image is debayered).

EDIT The reason for all of the peaks not being aligned is a comnination of a light source that, perhaps, is not perfectly 'white', and the attenuation of the CFA filters not having been equalised by the manufacturer. What you are performing is a sort of 'white (im-)balance compensation' for your MasterFlat

So, how to solve things?

Fortunately PixInsight coninues to improve, and tools get better. However, there is still not a simple 'one-click' process or script that will solve things - but there are two (new-ish) tools that now make life far easier (at least for me, anyway). These are the SplitCFA (script) and MergeCFA (process). And they are pretty self-explanatory, and relatively simple to use (but are not compatible with CMYG arrays). One splits out all pixels in a RAW image, such that four new images are created, one each for the four coloured filters in the Bayer array. And the second simply recombines each of these four (Grey, or Mono) images back into a RAW image again (ready for subsequent Debayering).

Obviously, we still need to do something 'clever' in between using these two tools. And that will be a simple PixelMath operation. And don't worry, it really is simple! Here goes . . .

In the Expressions pane, the only thing that you need is the following formula entered into the RGB/K field:

Code: [Select]

$T+(1-max($T))
You should have nothing else in this pane, but you should ensure that you tick the [Use single RGB/K expression] option.

In the Destination pane, you should (only) tick the [Generate Output] selection, and you should only select the [Replace Target Image] option.

Then just drag and drop the instance on to each of the four extracted images obtained when you used the SplitCFA script. Once you have done that, you simply use the MergeCFA process to generate a new NormalisedMasterFlat.

You use this image instead of your 'standard' MasterFlat for all subsequent calibration operations, and (hopefully) your colour-cast issues will be significantly reduced.

So, how does it work? Well, remember the expression that you entered:
             $T  +  ( 1  -  max($T) )
What this does is to first extract the maximum value of the 'Target' image (the image you will drop the process instance arrow-head on, referred to as $T in the expression).
Then it works out how much less this value is compared to '1.0' (the maximum value of a pixel in PixInsight).
With this 'free head-room' now known, each and every pixel value in the $T image can be increased by this amount.
The very brightest pixel(s) in the image will now be 1.0 - and in the subsequent division process of flats-calibration, pixels at these locations will now not change in value (because dividing by '1' doesn't change the numerator - the result stays the same).
And, similarly, because the faintest pixels values (at the worst points of vignetting or dust-donutting) are being divided by relatively 'higher' values, their subsequent changes are noticeably less (still enough to help eliminate the shading, but not now quite so 'harsh' - IMHO)
And, because each colour channel has had a totally separate scaling function applied, you no longer have possible huge disparities between the Histo peaks anymore - look at the Histo of the recombined NormalisedMasterFlat to see the evidence of this.

All in all - this works for me. I don't do any calibration without performing this small little step. It doesn't take long to do, and it makes subsequent BackgroundNeutralisation and ColourCalibration far easier. It also seems to have the effect of keeping noise levels down prior to the start of post-processing.

Anyhow - give it a try. Your mileage may vary, but I hope it helps (and, as always, I welcome criticism from those far more knowledgeable than I  )

[mcintyre_sj]

I am familiar with the problem of flat calibration adding colour to the lights, but that's not the problem here.

I think the problem or my misunderstanding here has something to do with how integration is normalizing and scaling the subs in order to create a valid statistical base to do the rejection calculations.

I avoid the problem with flats adding colour by doing all the calibration in raw CFA format. All files -  calibration frames and lights - are essentially monochrome images. I debayer after calibration and then split the rgb (debayered) light subs into separate channels.

When i look at the calibrated version of a debayered light sub, it was the same general colour as the original uncalibrated  light sub.   So calibrations isn't altering the mean levels between channels.

After alignment and integration of the subs for each of the 3 channels, the mean of the integrated images goes screwy. As above, the green channel integration can be at least 4 limes the value of that found in the input subs!

[RickS]

I think the choice of reference image (the first one in the list) may have some influence on this.  You could try some experiments with different reference images for the green integration.

In practice I've never found this a big deal, even with narrowband where the variation can be quite large.  You're probably going to do some form of colour calibration, aren't you?

[mcintyre_sj]

I don't understand what you mean about a reference image. Alignment takes a reference image.

Integration just take a list of files and combines them all using some metric to weight each individual file against the set and a rejection scheme to select pixels.

I always have to do some sort of colour correction as the DSLR has a modified IR filter. But the correction should be a tweat. Not to correct a 4x stretch in one channel. Given the extreme imbalance, i don't know if i can just divide by 4 or should use the historiogram tool.

First i'd like to understand what's happening.

[Niall Saunders]

Hello again,

A quick couple of questions.

First, how are you taking your flats? (Lightbox, EL Panel, Wall Flats / Dome Flats, Sky Flats, etc.), and what is your source of illumination?

Secondly, what does the Histogram look like for your MasterFlat (in Linear, un-modified form)? Can you attach a screenshot, or even post your MasterFlat somewhere and we can have a look at it.

[Juan Conejero]

This is the expected behavior for color images with default integration parameters. I recommend you to pay a read to ImageIntegration's reference documentation, and to the sections on rejection and output normalization in particular.

By convention, the first image in the integration set is used as the reference for image weighting and normalization (both rejection and output normalizations). You can disable output normalization to compute the integrated result as a plain weighted average of non-rejected pixels, but the result won't be optimal in SNR improvement terms. Whether this is relevant or not in your case depends on what you want to achieve and/or your personal preferences. As Rick has pointed out, white balancing naturally solves this issue, and you'll have to apply it anyway. Flats can also be relevant, as Niall says.

[mcintyre_sj]

Thanks Niall and Juan.

I'd like to focus on what's happening during integration and avoid the discussion on balance between colour channels.

Juan, i have read and re-read the ImageIntegration reference document several times and to the extent that i can get a feeling for the math, i understand what the normalization and scaling are required for. I would obviously like to get the best SNR so i think normalization and scaling are required. What i don't have a feeling for is how sensitive scaling is based on the range of stats for input files. 

As a example that uses only one channel (and avoids a discussion about flats and different levels and snr between colour channels):

I took 15 of the red channel monochrome subs and integrated them.
- The average mean of the 15 input subs is 0.0683 +/- 0.0016
- The mean of the integration is 0.0672

I then removed the 8th and 14th sub from the list and did another identical integration of the remaining 13 subs.
- The average mean of the 13 input subs is 0.0683 +/- 0.0016 (same as the 15 sub set)
- The mean of the new integration with 13 subs is 0.2529 (3.8x higher that the 15 sub integration)

- The first image in the list was the same in both integrations as were all the integration settings.
- The subs i removed had similar mean, std and noise characteristics to the image set average.


I am assuming the scaling has something do with the large increase in the output mean. But i don't understand which parameter is causing that large increase.

What stat(s) should i look at in the input set to understand why the mean of the output integration is so much higher than the inputs?

[mcintyre_sj]

I still think there is something wrong with the way i am doing something. I can see colour balancing as tweaking things, but i don't think a 4x difference between the inputs and output is expected behavior.

As above, the example is just the difference between stacking 15 vs 13 monochrome subs from the same set. And the mean of one integration is 3.8x the mean of the other with almost the same input data.

I don't even know the best way to recover the original levels which are 1/4 the mean of the final integration! Do i subtract, divide or use the histogram?

[RickS]

I still think there is something wrong with the way i am doing something. I can see colour balancing as tweaking things, but i don't think a 4x difference between the inputs and output is expected behavior.

As Juan and I both said, the reference image for ImageIntegration does affect the result.  I suggest again that you experiment with that.  Check the stats of the reference image and try some alternative ones.

You could also try disabling normalization, just as an experiment.

I don't even know the best way to recover the original levels which are 1/4 the mean of the final integration! Do i subtract, divide or use the histogram?

The data is linear so you can multiply by a constant.  If you want to match the colour masters to each other try LinearFit.

Cheers,
Rick.

[mcintyre_sj]

I ran a bunch of integrations first using different reference images. And then using a constant reference image and changing the settings.

Different reference images had output integration medians ranging from 0.06162 to 0.06601 from 6 of the 8 tests.
And then two outputs had a median of 0.30171!

The stats for the files input set all seem to be very similar - both in terms of medians, and various noise stats (StdDev, MAD, MRS).
As examples, the input set averages for median and MRS are:
  average Median : 0.06489     (+/- 0.00217)
  average MRS    : 1.57833E-03 (+/- 4.46000E-05)
 

The scale factors (from the console) for ALL the integration test range from about 0.98 to 1.07.

So i would expect the output integration to be in a range similar to the input set average. An output integration that has a median 4.6x higher than the input set average seems extreme.   


I tried different integration settings using a reference image that produced an integration with a very high median.
   Normalization, Weights, Scale estimator           Median of output integration
   ---------------------------------------------     -------
   Additive with Scaling, Noise Evaluation, IKS      0.30171
   Additive with Scaling, Noise Evaluation, MAD      0.30186
   Additive (noScaling),  Noise Evaluation, IKS      0.30218
   No normalization,      Noise Evaluation, IKS      0.06493
   
Note that additive without scaling still produced an integration 4.6x the input set average.
Only the "No normalization" test (with this reference image) produced an output median consistent with the inputs.


Is it reasonable to get an output integration that is 4.6x higher than the inputs? (Considering that all the inputs are similar.)

Any suggestions on how to pick a reference image that will produce a output in the same range as the input? They all look similar to me.
 
How do the "Scaling factors" reported on the console for each input image relate to the scaling of the final output integration?
   

[mcintyre_sj]

I would like to here back from the PixInsight team. There is a real problem with integration and i feel its being dismissed.

While there are several ways to correct the problem post integration, the integration results can be seriously out of whack requiring an extreme correction. This isn't a small correction to compensate for an imbalance between channels of the data set. This is a huge correction required because the integration process messed up.

1) Integration can produce results that are 4 times higher than any of the input files! An averaging process should consistently produce results that are within the range of the inputs. 

2) Just by choosing a different "reference image" the output results can vary by as much as 400% between runs using the same input files. That suggests a serious instability in the normalization algorithm(s).

3) The choice of "reference image" should not affect the output. Integration should be doing weighting, normalization and rejection of pixels based on comparisons to the input "set" as a whole. It should not be doing a statistical comparison of the each image to the arbitrary choice of reference image.

4) There's no identified process for selecting a reference image that will produce the best output. I just use trial and error and usually just try to coerce the integration tool to produce a result within the range of the inputs.

This is the expected behavior for color images with default integration parameters.

The problem is not related to differences between colour channels!

Selecting different reference images for the same data set creates outputs that vary by as much as 400%. The results can vary by 10% between runs that produce outputs within range.  In my current project, it took 5 attempts (for one of the channel's) to find a reference image that allowed integration to produce a result within range of the inputs. 4 of the 5 integrations were 3.8x the input set average.

By convention, the first image in the integration set is used as the reference for image weighting and normalization (both rejection and output normalizations).

I don't agree that the output should vary based on the arbitrary choice of the first image in the list. I can see using the first image as a convenient reference point but it should not affect the rejection and integration.

But i can accept this as it doesn't make that much difference to my process (when output is within the range of the inputs). If i were using tru balance filters then a 10% variance in the outputs would make a difference. And require arbitrary adjustments to correct for the tool. 

[Juan Conejero]

I'll add an option to use a robust average of statistical properties as reference for normalization, instead of the properties of the first image. This will be implemented in the next version of the ImageIntegration tool.

That suggests a serious instability in the normalization algorithm(s).

Not at all. The implemented algorithms are robust and reliable. It's just that the first image in the input list is used as the reference for normalization. That's just a convention, and it does not mean that the results have to be better or worse for that reason. Obviously, if you are integrating a set with substantial differences in location and scale (the relative position and width of the histogram's main peak for each frame), there can be also substantial differences in the result, depending on the image chosen as reference. In all cases, but very especially in cases where there can be large differences among frames, you should perform a statistical analysis of your data set prior to integration, including the possibility to reject a number of frames, based on quality estimates. We have the SubframeSelector script and the Blink tool to perform these tasks efficiently.

[Niall Saunders]

Hi Juan,

As a thought - if the 'first pass' in Image Integration produces a set of noise parameters for each image, and uses either the 'first image' of the data set (as is the current method), or could use a 'nominated image' (as suggested by you), could subsequent runs of ImageIntegration offer the option (via its UI) to use the 'best quality image' of the previous run? I suppose I am thinking of some form of 'iterative' approach.

[sorry to slightly hijack the OP thread]

[mcintyre_sj]

I'll add an option to use a robust average of statistical properties as reference for normalization, instead of the properties of the first image. This will be implemented in the next version of the ImageIntegration tool.

Thanks!
I expect that will address the issue. Certainly it will mean that integration always produces the same result, independent of the first file in the list.

If there is some sensitive in the output normalization process, then perhaps using the input averages will avoid this and always produce reasonable results.

I have been using SubFrameSelector, Blink and BatchStatastics to analyze the input set. I generally run several different integrations with different subsets of input files and then make a decision of which set produces the best results. With my limited unobstructed sky, getting enough subs is an issue. Integration then is often a trade off between reducing background noise or improving detail. So i might include subs with higher than average FWHM and see if the noise is reduced without compromising detail. 

Two of the 15 images have medians which are 5% lower than the average and a different pair have mean deviations 4% higher than the average. Using either of the two images with lower medians, does guarantee a bad result and 400% shift in the output levels. But other reference images also cause a problem. Even though with a casual inspection they all look similar to the image set averages.

I don't have a feel for how sensitive the integration normalization process is to variations in the stats that would cause a 400% shift in the ouput levels. Nothing in the input set varies by more than about 5% for levels and noise stats. The scaling numbers from the console are in line with the 5% variation i see when analyzing the input stats.

Why is the output "normalized"?
What is the output normalized to?

The integration reference document explains thoroughly why having common position and dispersion is required for rejection. But shouldn't the output resemble the inputs in terms of levels? At least resemble the image set averages?