Pixel rejection

[Harry page]

Hi

While having my customary PI discussion on another forum a fellow member posed the following


Harry, here is the bit that concerned me;

http://pixinsight.com/doc/tools/ImageIn ... ation.html, para 14

Averaged Sigma Clipping

Our implementation of averaged sigma clipping is a variant of the similar algorithm (AVSIGCLIP) from the imcombine task of IRAF. This algorithm works in two phases. In the first phase, the gain of an ideal detector with zero readout noise is estimated for each pixel stack. The second phase is an iterative sigma clipping procedure, where the estimated gains are used to compute the dispersion (sigma) of each pixel stack around the median.

Dispersion is calculated based on Poisson statistics, under the assumption that the noise in the images is proportional to the square root of the mean pixel values:
 (my quotes).

We are dealing here with sets of finite data as in combining frames pixel by pixel. Apart from the fact that the author mixes terms (calling standard deviation 'dispersion'). It has long been understood that such numbers are dealt with using Gaussian statistics. To use Poisson statistics (as it says above 'under the assumption') is wrong. There is no random time element involved here, just a set of numbers. It sounds like splitting hairs but either you do your maths right or you don't.

Dennis


Bit past my pay grade , so any thoughts

Regards Harry

[mschuster]

Apart from the fact that the author mixes terms (calling standard deviation 'dispersion').

I think it is good to try to be as precise as possible with terms. But there does seem to be a difference of opinion.

According to Wikipedia on Statistical dispersion:

Common examples of measures of statistical dispersion are the variance, standard deviation and interquartile range.

Mike

[Juan Conejero]

Along with Mike's comment, just a few more:

* The standard deviation is a well-known estimator of dispersion. From the Wikipedia:

http://en.wikipedia.org/wiki/Standard_deviation
"In statistics and probability theory, standard deviation (represented by the symbol sigma, ?) shows how much variation or "dispersion" exists from the average (mean, or expected value)."

Other estimators of dispersion that we use frequently in PixInsight are the average absolute deviation and the median absolute deviation.

* The concept of standard deviation can be used for any distribution with mean and variance. From MathWorld:

http://mathworld.wolfram.com/StandardDeviation.html
"Standard deviation can be defined for any distribution with finite first two moments, but it is most common to assume that the underlying distribution is normal."

* Using Poisson statistics to model the dispersion of values in a pixel stack is not wrong. Noise in CCD images is known to be well-modeled by a Poisson distribution. Hence, using Poisson statistics to scale the variance of a stack of pixels is not wrong, either. From the IRAF documentation of the AVSIGCLIP rejection algorithm:

http://www.eso.org/sci/software/esomidas/doc/user/98NOV/volb/node44.html
"The sigmas at each point in the line are scaled by the square root of the mean--i.e., a Poisson scaling of the noise is assumed."

So I think we are doing our maths quite right here. And we'll try to do them better in future versions.

[Harry page]

Hi

Thanks for the speedy reply 

Regards Harry