Statistics process bug

[mschuster]

Hi Juan,

Incorrect Statistics after Pixelmath, see below.

PI 1.8.6.1457
macOS 10.14.3

Thanks,
Mike

Please download this frame.

Launch PI.
Open Statistics process, select 16-bit, normalized on, open Options, check all available metrics, Save, OK.
Open light_01_c, select light_01_c view in Statistics, copy Statistics results, values listed below.
Apply Pixelmath "$T + median($T)" to frame with "Rescale results" option off.
Note that this is a location change and not a scale change.
Copy Statistics results, values listed below.
Note that values "median" did not change, and values "avgDev", "MAD", "sqrt(BWMV)", and "sqrt(PBMV)" did change.
These changes are suspect given a strict change in location.
Save the resulting frame as light_01_c_location and open this frame in a new window.
Select light_01_c_location view in Statistics.
Copy Statistics results, values listed below.
Note that light_01_c_location values differ from the light_01_c post Pixelmath values, and appear to be equal to expected values (ie median is doubled and scale metrics are invariant (up to some apparent roundoff error)).
Also that Sn and Qn did not suffer from this problem.

Although I have not tested this, and I could be mistaken, I am worried that ImageStatistics Core JavaScript object might suffer a similar bug across an image change within a script.

light_01_c pre Pixelmath:
               K
count (%)      99.99933
count (px)     2082062
mean           481.672
modulus        1002871438.753
norm           1002871438.753
sumOfSquares   41379490.289
meanOfSquares  19.874
median         336.139
variance       1070453.545
stdDev         1034.627
avgDev         297.668
MAD            176.344
sqrt(BWMV)     208.955
sqrt(PBMV)     209.696
Sn             175.286
Qn             168.396
minimum        3.050
maximum        56862.308
minimumPos       1341,  1057
maximumPos       1651,   142

light_01_c post Pixelmath:
               K
count (%)      100.00000
count (px)     2082076
mean           817.808
modulus        1702738966.855
norm           1702738966.855
sumOfSquares   55256990.093
meanOfSquares  26.539
median         336.139
variance       1070447.914
stdDev         1034.625
avgDev         603.676
MAD            498.358
sqrt(BWMV)     527.862
sqrt(PBMV)     510.238
Sn             175.288
Qn             168.398
minimum        336.139
maximum        57198.448
minimumPos        237,   131
maximumPos       1651,   142

light_01_c_location:
               K
count (%)      100.00000
count (px)     2082076
mean           817.808
modulus        1702738966.855
norm           1702738966.855
sumOfSquares   55256990.093
meanOfSquares  26.539
median         672.277
variance       1070447.914
stdDev         1034.625
avgDev         297.669
MAD            176.345
sqrt(BWMV)     208.957
sqrt(PBMV)     209.700
Sn             175.288
Qn             168.398
minimum        336.139
maximum        57198.448
minimumPos        237,   131
maximumPos       1651,   142

[Juan Conejero]

Hi Mike,

The bad news is that there is a bug. The good news is that it only happens on the Statistics window, and does not affect the rest of the platform. The bug only happens with clipped statistics, that is, when the Unclipped option is unchecked on the Statistics window. If you check it, all values will be correct and all estimates of scale will remain unchanged with or without the PixelMath instance applied, as expected.

For some reason that I still don't understand, when the Unclipped option is disabled, the median, along with all estimates that depend on it (MAD, avgDev, etc), are retrieved incorrectly from the selected image on Statistics after a process that changes these properties is executed. Apparently, the "Median" view property is being retrieved from values cached in the previous image state, that is, before applying PixelMath in this case. However, the "UCMedian" (unclipped median) view property is always retrieved correctly. A very strange bug, but be sure I'll find and fix it immediately. Thank you for detecting this ugly error.

Just to clarify that this problem does not affect the rest of the platform, please make the following checks:

- Save the image after applying the PixelMath process as a new file, close it, and open it. The median and all estimates depending on it will now be correct on the Statistics tool, with and without the Unclipped option enabled. This shows that the problem is related to accessing cached image properties from the Statistics tool.

- Select the image after applying the PixelMath instance. On the Process Console window, enter this command:

j var S = new ImageStatistics; S.generate( ImageWindow.activeWindow.mainView.image ); S.median*65535

This will show the correct value: 672.277

Now undo the PixelMath process, then enter this command:

S.generate( ImageWindow.activeWindow.mainView.image ); S.median*65535

and again you'll get the correct value: 336.138

- Instead of the ImageStatistics object, you can use methods of the Image object. For example:

j ImageWindow.activeWindow.mainView.image.median()*65535

If you enter this command after PixelMath and after undoing it, you'll get the correct values again.

I'll let you know when this bug is fixed, and why it is happening. Thanks again for discovering it, and sorry for the trouble.

[Juan Conejero]

More on this. The bug is a regression in the core application, not in the Statistics tool. You can check this by inspecting the Median property on the View Explorer window, while undoing/redoing the PixelMath instance. Fortunately, only the Statistics tool is affected, and only when the Unclipped option is disabled.

No problem because I am very close to release a new version, so this bug report comes just in time.

[Juan Conejero]

More information. The problem arises when the median($T) function is invoked from a running PixelMath instance. If instead of your original expression you execute this one (on a duplicate of the original image, or just after loading it):

$T + 336.139/65535

then Statistics will report correct values for all location and scale estimates, with or without clipping, before and after PixelMath.

I suspect the problem is a complex interaction of PixelMath with the core application, but I'm not sure. "Interesting" bug

[mschuster]

Thank you Juan, I appreciate your checking and am glad it is an isolated case.