# Questions about Generalized Extreme Studentized Deviate Test rejection

#### johnpane

##### Well-known member
I have three questions about the ESD rejection algorithm.

1) Assuming ESD low relaxation is set to 1 (presumably no effect), if outliers is set to 0.3, does that mean:
a) that 30% of the total pixel stack can be rejected by the combined effects of high clipping and low clipping?
b) that 30% of the total pixel stack can be rejected by high clipping and another 30% by low clipping?
c) that 30% of the above-median pixels can be rejected by high clipping, and 30% of the below-median pixels can be rejected by low clipping?
d) that 15% of the above-median pixels can be rejected by high clipping, and 15% of the below-median pixels can be rejected by low clipping?

Option (a) is implied by the tooltip, "The default value is 0.3, which allows the algorithm to detect up to a 30% of outlier pixels in each pixel stack." But this would mean that high clipping would be affected by turning on or off low clipping, which would certainly be an undesirable effect.

2) Is the ESD significance test applied to:
a) each individual potential outlier?
b) the entire pixel stack?

Option (b) is implied by the tooltip, reworded here for clarity "0.01 means that a 1% chance of being wrong is acceptable when rejecting the null hypothesis (that there are no outliers in a given pixel stack)." That is, the tooltip implies that the test is all or nothing, either there are outliers that can be rejected or there are not. This would seem to give the user no control over how many outliers can be rejected. If there is one clear outlier with p<.000001, the null is rejected but what other outlier get rejected in this case?

3) What parameter is relaxation applied to:
a) a divisor for the outliers parameter?
b) a divisor for the significance parameter?
c) something else?

I see no hint in the tooltip. But if the answer is (a), how does that interact with the answer to (1) above?

Thank you,
John

#### johnpane

##### Well-known member
Hello, I am still hoping to learn the answers to these questions if anyone knows. Thank you.

#### Juan Conejero

##### PixInsight Staff
Hi John,

Sorry for the delay in getting back to you.

1) Assuming ESD low relaxation is set to 1 (presumably no effect), if outliers is set to 0.3, does that mean:
a) that 30% of the total pixel stack can be rejected by the combined effects of high clipping and low clipping?
ESD rejection doesn't work like sigma clipping and similar rejection methods, so the concepts of 'low' and 'high' pixels are not directly portable. The answer to this question is approximately a).

Option (a) is implied by the tooltip, "The default value is 0.3, which allows the algorithm to detect up to a 30% of outlier pixels in each pixel stack." But this would mean that high clipping would be affected by turning on or off low clipping, which would certainly be an undesirable effect.
Actually, the ESD algorithm does not 'know' whether it is rejecting low or high pixels. It works on absolute values with respect to a central value (which is not the median except for very small stacks) without taking signs into account. We introduce a relaxation factor for pixels below the central value in each iteration, which conditions the algorithm's behavior as a whole, but the decision to reject or not high/low pixels as a function of process parameters is taken at the very end of the process, when all outliers have already been detected.

This can be seen more clearly in our implementation. This is the main rejection loop:

and here is when low and high pixels are actually rejected depending on process parameters, that is, on whether low and/or high clipping is enabled:

As you can see, both sections of the algorithm are separate and independent each other.

2) Is the ESD significance test applied to:
a) each individual potential outlier?
b) the entire pixel stack?
The generalized ESD test algorithm is quite well described here:

If I've understood it well, the answer to your question would be a mix of a) and b). The test is repeated for the whole stack, but removing potential outliers at each successive iteration to compute the next R_i test statistic.

Note that our implementation introduces two important changes to the original algorithm that make the ESD rejection method more robust and versatile. For robustness, we don't use the mean and the standard deviation of the sample, but a trimmed mean and the standard deviation of the trimmed sample at each iteration. In all of our tests the algorithm behaves much more consistently with this variation. For versatility, we introduce a relaxation factor that multiplies the standard deviation (the s variable in the algorithm description above) for pixels with values smaller than the trimmed mean. This allows us to apply a more tolerant rejection for dark pixels.

That is, the tooltip implies that the test is all or nothing, either there are outliers that can be rejected or there are not.
This is not what this tooltip tries to communicate. The ESD significance parameter does not define a Boolean condition (reject or don't reject) over the whole stack. It works as a limit to compute critical values (lambda_i variables), which are compared to test statistics (R_i) in order to find the number of outliers. By increasing the ESD significance parameter more pixels will be rejected because the algorithm will allow more mistakes made by rejecting the null hypothesis (no outliers in the stack). It's a bit convoluted but that's how it works.

The ESD outliers parameter defines an upper bound for the number of outliers that can be rejected in the sample. For example, for a stack of 10 pixels, if ESD outliers is equal to 0.3 this means that 0, 1, 2 or 3 outliers can be detected. Of course, this also means that in case there were an additional fourth outlier, it would pass unnoticed.

3) What parameter is relaxation applied to:
a) a divisor for the outliers parameter?
b) a divisor for the significance parameter?
c) something else?
The answer is c). As I've noted above, the relaxation parameter multiplies the standard deviation of the (trimmed) sample at each iteration to compute test statistics for pixels with values smaller than the trimmed mean. Since the low relaxation parameter is >= 1, it causes the algorithm to be more tolerant for rejection of low pixels, since the algorithm 'sees' a higher dispersion for these low pixels. In a sense, what we are doing here is telling lies to the ESD test for a subset of pixels where we want less rejection.

#### johnpane

##### Well-known member
Thank you very much for taking the time to reply, Juan. I understand much better now, though perhaps not perfect yet. I apologize if there are misunderstandings / errors in the following comments.

1. It seems important for users to realize that adjusting the low relaxation can change the number of high pixels rejected as outliers. One reason for this may be readily apparent; if more low outliers are tolerated more high pixels can be rejected before the allowed fraction of outliers is exhausted. But the second reason is less obvious; the relaxation factor is applied not only to the standard deviation for calculating low outliers, but also causes fewer low pixels to be trimmed in the trimmed mean calculation (line 925). This shifts the central value lower, having the effect of making high pixels more likely to appear as outliers, as well as further decreasing the number of low outliers (on top of the parameter's effect of as a multiplication factor on sd). Finally, there is a third effect, also less obvious; if less low pixels are trimmed, the standard deviation for high rejection will increase, making fewer pixels qualify as outliers and somewhat counteracting the second effect. The combined effect on high rejection is unclear. I wonder if both the second and third effects should be avoided by not considering low relaxation in determining trimming?

For the tool tip for low relaxation, it would be helpful for users to know that it is a multiplier for the sd so that they know what scale they are operating on. And if this parameter's effect on trimming is retained, I suggest that should also be made evident. The tooltip should also perhaps warn users that adjusting this parameter will affect not just low rejection but also high rejection, Finally, somewhere it should be made evident that turning on/off low (or high) rejection has no effect on determining which pixels are outliers and just on whether those outliers are rejected or retained. This has the surprising implication that, if low rejection is turned off, low relaxation still will affect high rejection.

2. In your example stack of 10 pixels, with ESD outliers equal to 0.3, my understanding is that three statistical tests are performed. The null hypothesis for the first test is that there are no outliers. If that null is rejected, one pixel is deemed an outlier. The null hypothesis for the second test is that there is no more than 1 outlier. If that null is rejected, a second pixel is deemed an outlier. Finally, the third null hypothesis is that there are no more than 2 outliers and if that null is rejected a third pixel is deemed an outlier. As such, I think the simplest way to explain alpha (ESD significance) in the tooltip is the probability that too many pixels are deemed outliers.

Last edited:

#### Juan Conejero

##### PixInsight Staff
the relaxation factor is applied not only to the standard deviation for calculating low outliers, but also causes fewer low pixels to be trimmed in the trimmed mean calculation (line 925).
Yes, but this is intentional, not a mistake IMO, despite the side effects that you have described. What we are doing with the ESD low relaxation parameter is to introduce a priori knowledge in the pixel rejection problem: that the probability of low outliers is systematically lower than the probability of high outliers for all pixel stacks. Hence we are modifying the expected statistical model that we are using for each sample of pixels.

The trimmed mean that we are using in line 926 provides a robust estimate of location for the pixel stack (side note: we use a trimmed mean instead of the median because we want to achieve both robustness to outliers and minimal standard error here). To be coherent with the modified model I've described above, we have to apply it in all relevant stages of the process: not just when we apply it to scale deviations from the central value (line 935), but also when we compute the central value estimate (line 926).

It is true that we are altering rejection of high pixels when we do this, but this side effect has little practical relevance, especially considering that optimal rejection parameters, when necessary, have to be found by manual trial-error work. Despite this, our tests have shown that our implementation of the ESD rejection algorithm can work remarkably well with its default parameter values in most cases.

It seems important for users to realize that adjusting the low relaxation can change the number of high pixels rejected as outliers.
IMO this is the type of information to be provided in the technical documentation of the ImageIntegration tool (which I have to write ASAP, since the current one is severely outdated), but I doubt this can be of special interest in a tooltip text. Anyway, saying this in the tooltip isn't difficult, so we can add it.

This has the surprising implication that, if low rejection is turned off, low relaxation still will affect high rejection.
I agree that this can indeed be surprising, but as I've said above, the ESD low relaxation parameter introduces a priori knowledge in the whole rejection problem. In other words, these side effects are consequences of the modified model used to describe how outliers are distributed in each pixel stack. Of course we could disable ESD low relaxation when low rejection is disabled, but I'm not sure if introducing a 'special case' like this would really be a good idea. Something to think on, anyway.

As such, I think the simplest way to explain alpha (ESD significance) in the tooltip is the probability that too many pixels are deemed outliers.
Agreed. I'll try to make these tooltips easier to understand, thank you for these insights and suggestions.

Thank you, Juan!

#### johnpane

##### Well-known member
One other thought..

At the default settings the effect of the ESD outliers parameter seems likely to be far greater through its effect on the trimmed mean function than throught limiting the number of rejected pixels. Essentially, at 0.3 it is trimming 30% of the high pixels and 20% of the low to compute the mean of 50% of the pixels. However, it seems very unlikely that so many pixels would pass the significance test that the 30% cap on rejection would come into play.

As such, if a user wants to reject fewer pixels, the alpha parameter has much greater leverage but the tooltips as written might lead them to focus on reducing the outliers parameter. They may would have to reduce it very substantially, perhaps down to 5 or 10% before they see any effect, and such a reduction would of course vastly reduce the amount of trimming. Again, it makes me wonder why those two functions (amount of trimming to find the central value, and upper bound on the percentage of pixels rejected) are tied to the same parameter.

#### johnpane

##### Well-known member
Sorry to keep adding more thoughts.

Users not well-versed in statistics may not realize just how much leverage the relaxation parameter has. With the default settings, alpha=.05 and that means that high pixels are rejected if they are about 2 sd or more from the central value assuming a stack of 60 pixels. With relaxation of 1.5, low pixels have to be about 3 sd or more from the central value, making alpha≈.002 for low rejection. If users bump relaxation up to 2, they are effectively setting alpha for low rejection to ≈.00001, if they set it to 3 effective alpha≈0.00000001 (six sigma, but using a t-distribution with 60df), and if they set it to 5 they are essentially setting alpha to zero for low rejection.

I wonder if the interface could display some calculations? Once the input frames have been loaded, n is known. The tool could calculate alpha_high and alpha_low (the sum of which will be lower than ESD significance unless relaxation = 1), the range of pixel rankings that will used for the trimmed mean calculation, and maybe more.

For example, assuming n=60, outliers=0.3, and significance=.05:

relaxation 1 -> alpha_high=0.025, alpha_low=0.025, trimmed mean calculated on pixels 19..42 (sorted low to high)
relaxation 1.5 -> alpha_high=0.025, alpha_low=0.002, trimmed mean calculated on pixels 13..42
relaxation 2 -> alpha_high=0.025, alpha_low=00001, trimmed mean calculated on pixels 10..42
relaxation 3 -> alpha_high=0.025, alpha_low=0.00000001, trimmed mean calculated on pixels 7..42
relaxation 5 -> alpha_high=0.025, alpha_low≈0, trimmed mean calculated on pixels 4..42

If I think 3 pixels in my 60-pixel stack seem reasonable to reject, I can get the trimmed mean to exclude them by setting relaxation to 5 but by doing that I virtually ensure they will NOT be rejected because of the effective alpha.