Author Topic: Question on scaling and linear fit  (Read 3399 times)

Offline bitli

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 513
Question on scaling and linear fit
« on: 2014 April 29 10:26:13 »
I tried to mutually Linear Fit two flats, and also to compare the fitting functions with scaling factor calculated by Image Integration (for no other reason than curiosity and urge of understanding).

I was surprised to see that the linear fits are not the inverse of each other. In fact I am also surprised that none is even near the scaling/offset calculated by II (although it is a very different algorithm, I though that it should provide something not too far of a linear fit, because at the end II is doing a scaling+offset to make the images comparable, as it seems).

Could somebody clarify what is wrong with my expectations?

Here are the data:

Copy of image 1 fitted on image 2
y0 = +0.089933 + 0.619350·x0
?0 = +0.001636

Copy of image 2 fitter on image 1
y0 = +0.158579 + 0.952393·x0
?0 = +0.002034

One image being the reference, scaling of other image (Qn)
Scale factors   :   1.23868
Zero offset     :  +1.366877e-001

-- bitli

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Question on scaling and linear fit
« Reply #1 on: 2014 April 30 18:28:37 »
Quote
I was surprised to see that the linear fits are not the inverse of each other.

This is the expected general behavior. To understand why this happens, let's describe the LinearFit task. For a set of N data points {xi,yi} for i = {0,...,N-1}, LinearFit finds the parameters a, b of the straight line function:

y = a*x + b

that minimizes absolute deviation:[1]

Sum( |yi - b - axi| ) for i = {0,...,N-1}

If the data points are exactly related by a linear function, then the resulting absolute deviation will be zero and what you are expecting will happen, that is, if we apply LinearFit to the reverse set {yi,xi} in this unique case, we'll obtain:

x = y/a - b/a

However, if the data are not related by a linear function, then the residual absolute deviation will be greater than zero and the preceding equation will not hold. This is what happens with real world images, where a linear function can only be an approximation to the set of differences between pairs of pixel values at the same coordinates.

Of course, we can carry out a controlled experiment to verify that this works as described. Open an image (any image will work, not necessarily a linear image) and duplicate it. For simplicity, let's rename them as "A" and "B". Now apply the following PixelMath expression to B:

$T*0.75 + 0.0025

Open LinearFit, select B as reference, and apply the tool to A. As expected, you'll get the following console output:

y0 = +0.002500 + 0.750000·x0
s0 = +0.000000

The resulting absolute deviation is zero because we know that both images are strictly related by a linear function. Hence, the third equation above must hold. Undo the LinearFit process on A and Apply LinearFit to B using A as reference:

y0 = -0.003333 + 1.333333·x0
s0 = +0.000000

where, as expected, we have -0.003333... = -0.0025/0.75 and 1.333333... = 1/0.75.

Now undo everything, including the PixelMath instance that you applied to B, and apply the following expression:

$T*$T + $T*0.75 + 0.0025

This time we are forcing a nonlinear function between both images. If you apply LinearFit to A using B as reference, this is the result:

y0 = +0.001956 + 0.803880·x0
s0 = +0.000083

The resulting average absolute deviation is greater than zero, denoting that the linear fit is approximate. The inverse application (undo A, apply LinearFit to B with reference=A) yields:

y0 = -0.002215 + 1.226951·x0
s0 = +0.000102

Note that this time, since we have introduced a term on the square of the image, the fitted function is an approximation and hence not invertible. If it was, we'd get:

-b/a = -0.001956/0.803880 = -0.002433
1/a = 1/0.803880 = 1.243967

instead of -0.002215 and 1.226951, respectively. In the example that you put in your post you get average absolute deviations around 0.002, which is one order of magnitude larger than the deviations we are getting in this synthetic example, so the differences in the fitted function parameters have to be much larger.

Quote
In fact I am also surprised that none is even near the scaling/offset calculated by II

The scale factors and zero offsets computed by ImageIntegration have nothing to do with the LinearFit process that I have described. It is a completely different algorithm, which is described in detail in the reference documentation of ImageIntegration. In this case the normalization parameters are computed from estimates of statistical scale (dispersion, or variability) and location (central value) with respect to the reference image. See Equation 33 in the documentation.

References

[1] W. H. Press et al. (2007), Numerical Recipes: The Art of Scientific Computing, Third Edition, Cambridge University Press, Sect. 15.7.3.
« Last Edit: 2014 April 30 18:34:18 by Juan Conejero »
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline bitli

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 513
Re: Question on scaling and linear fit
« Reply #2 on: 2014 May 01 00:52:11 »
Thanks Juan,  I understand my mistake. I try to rephrase my understanding to check it and hopefully to help others:

Linear fit:

If I take two flats with the same controlled luminosity, with a ratio of exposure of two, and do a 'mutual linear fit' I get a ratio of approximately 2 (resp 0.5) and an offset near 0 (it should be the value of the bias, but there is a large error as we extrapolate too far). For example:

y0 = +0.002060 + 0.474670·x0
?0 = +0.000362

y0 = -0.003743 + 2.057024·x0
?0 = +0.000753


This reflects that statistically a pixel of an image is twice as large (small) than the corresponding pixel of the other image. This is like drawing a line over the main axis of M82, it is pretty easy to find a reasonable one.

If the two flats are taken with the same exposure time (and same luminosity!), there is no relationship between the corresponding pixels. There is no reason that a pixel brighter in one image due to random noise corresponds to a brighter pixel in the same location of the other image (otherwise it would not be random).  The linear fit try to find a relationship between corresponding pixels only.  I was trying to find the best line to draw through a globular cluster to model it. This is not defined (and is not 0 or 1 either!). Not enough sleep, I guess :-)

If I am awake, this shows in the difference in sigma: 0.002034 in the case of the 'random line', 0.000362 in the case of the reasonable fit.  I would like to understand how I evaluate the confidence that the linear fit is meaningful from the sigma, if possible at all.  I need more reading.


Normalization in Image Integration

The documentation of the ImageIntegration is my bed time reading :-) I try to rephrase the ImageIntegration in other terms, less precises and exact, but hopefully providing an hint for the less mathematically minded:

The normalization of ImageIntegration 'aligns' the 'histograms' of the images, a little bit like we do when we align the colors manually with the HT or Curve tools. This is also done by an offset (the distance between the tips of the histograms) and a factor (the scale between the width of the histograms), with many subtleties to quickly find an appropriate measure of the tip and width even when the histogram is not a good looking Gauss curve.
The comparison of the histogram is statistic, it does not care where the pixels are, only the number of pixels that have the same value.  If two images are taken in exactly the same conditions, they should have matching histograms (no offset and scaling of 1).

With two images taken in the same conditions except the exposure time, one being twice longr than the other (resulting in a linear fit with a factor of 2), we should normally expect the location (tip of the histogram) to be twice as high after bias correction and the width of the histogram to be something like Sqrt(2) wider, if I understood correctly. Something I still need to measure to validate (for example this does not work with DSLR dark frames which rebias them to the same pedestal, only the the scale is increased).

I hope that my informal understanding is correct and may help others.

-- bitli


Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Question on scaling and linear fit
« Reply #3 on: 2014 May 01 03:08:30 »
Quote
I would like to understand how I evaluate the confidence that the linear fit is meaningful from the sigma, if possible at all.

The average absolute deviation (sigma) after the linear fit tells you how well the data adapts to a linear model. Sigma is zero only if all the data points lie exactly on a straight line, like the controlled experiment that we did with PixelMath on a duplicate image. The smaller the sigma, the better a linear model can explain the distribution of data points, which are pairs of pixel values at the same coordinates in the case of LinearFit.

However, don't take this value as a precise indicator of the true relationship between the pixels in both images. To put this in perspective I've just made a quick test. Open an image and duplicate it, call them A and B. Now use the Crop tool to shift B by a good distance in both axes; for example I have defined the following margins: top=100, left=100, right=-100, bottom=-100. This effectively shifts the image 100 pixels toward the bottom right corner in the X and Y directions. After this displacement, I have applied LinearFit to B with A as reference, with the following result:

y0 = +0.013356 + 0.009539·x0
s0 = +0.001150

From the relatively small residual average absolute deviation, one could be tempted to draw the conclusion that both images are well fitted by the line with parameters a=0.009539 and b=+0.013356. Although these parameters correspond to the best possible line fit in the least absolute deviation sense, we obviously know that the fit is completely meaningless because the images are unaligned. In this case the sigma tells nothing, but the low value of the slope is a good indicator to know that things didn't go well: the fitted line practically annuls the reference image.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline bitli

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 513
Re: Question on scaling and linear fit
« Reply #4 on: 2014 May 01 03:38:27 »
Thanks, exactly what I wanted to know.
-- bitli