SubframeSelector and Weight of "-nan(ind)"???

JST200

Active member
Sep 29, 2018
37
0
Hi,

As of the latest version (v1.8.8-6) I have started getting a weight value of "-nan(ind)" on poor images. Previous versions have never caused this with my settings.

Not surprisingly, the value produces a "ReferenceError: nan is not defined".

I have done a blink on the set of subs and the sub that has this weighting is poor, but stars are visible.

Normally ( have the Weighting and Approval expression set as below,, which has never caused problems before:

((Math.pow(Math.pow(0*Math.pow((FWHM-9.26501)/-4.28259,1)+0.05*Math.pow((Eccentricity-0.53117)/-0.21586,1)+0.1*Math.pow((NoiseRatio-0.87122)/0.01721,1)+0.85*Math.pow((Stars-169)/98,1),0.6),1))-0.24021)/0.71807*0.6+0.4

Weight>=0.5

Even if I remove these, untick the "File cache" option and rerun the process, I still get this value.

I have attached a screen capture of the Process error, the sub with the strange weighting and the settings I have used. Any help gratefully received.

Thanks, Jim
 

Attachments

JST200

Active member
Sep 29, 2018
37
0
Just an update/correction...

If I remove the Weighting and Approval expressions it does complete successfully. All the Weights are set to 0.

However, it doesn't seem correct that PI should put what looks like an "expression" in this field. Rather, I would expect a "0" or something.

Thanks, Jim
 

fredvanner

Well-known member
Apr 17, 2019
412
55
Wells, Somerset, UK
I didn't know you could use PCL functions (e.g. Math.pow(x,y)) in Weighting / Approval expressions ...
... in fact, I've no idea what syntax is allowed in these expressions - is it documented anywhere?
The expression above looks wrong [ ...(0*Math.pow(..., ...) will be 0 whatever in in the Math.pow arguments]; not a problem, but can't find errors unless the expresion is correct.
It would also help if you could post one of the images producing the error.
 

fredvanner

Well-known member
Apr 17, 2019
412
55
Wells, Somerset, UK
I would expect a "0" or something.
I can see a problem with this. If evaluating the weight function produces a NaN it would probaby be sensible to substitue a "bad" weighting - but SubframeSelector is agnosic about whether "higher is good" or "higher is bad". So there really is no sensible numeric substitute for this sort of error.
 

JST200

Active member
Sep 29, 2018
37
0
Hi Fred,

Thanks for your reply. :)

To be honest, I'm not good on the maths involved here, nor what expressions are allowed by SubframeSelector. I use the excellent spreadsheet created by Alex Woronow to create the formula (https://pixinsight.com/forum/index....-for-selecting-and-weighting-subframes.14594/).

I think what has happened is I've saved a process icon with one of the formula from another set of subs, rather than the default formula I used to have:

(15*(1-(FWHM-FWHMMin)/(FWHMMax-FWHMMin)) + 15*(1-(Eccentricity-EccentricityMin)/(EccentricityMax-EccentricityMin)) + 20*(SNRWeight-SNRWeightMin)/(SNRWeightMax-SNRWeightMin))+50 )

Having said that, I made the slip up somewhile ago and been using it for several version of PI without a problem. It's only this last update that has taken exception to it.

What I have now done is removed any formula from my Process Icon, and then enter it once I've run the output csv data through the spreadsheet. This is the formula I have just successfully used (including the sub that was causing the nan before):

((Math.pow(Math.pow(0.1*Math.pow((FWHM-8.86075)/-3.59844,2)+0.1*Math.pow((Eccentricity-0.73187)/-0.28941,2)+0.1*Math.pow((NoiseRatio-0.83069)/-0.28309,2)+0.7*Math.pow((Stars-196)/4392,2),0.6),0.5))-0.36068)/0.62589*0.6+0.4

I have attached a screen capture of the spreadsheet in action and the result of applying the generated formula in Subframe Selector.

Surely Weighting is always 1 = good, 0 = bad. When would it ever be otherwise? Also, if it does get an such as this, I would have expected it to flag the "invalid" sub, and complete the process without it. Bombing out of the process seems a bit extreme.

Cheers, Jim
 

Attachments

fredvanner

Well-known member
Apr 17, 2019
412
55
Wells, Somerset, UK
Trying to make sense of the weighting function (to spot possible errors):
Writing FWHM=V1; Eccentricity=V2; NoiseRatio=V3; Stars=V4, and replacing some constants by symbols, including p=1, q=0.6 and r=1, I get:
((((kV1*((V1-mV1)/(-sV1))^p+kV2*((V2-mV2)/(-sV2))^p+kV3*((V3-mV3)/(+sV3))^p+kV4*((V4-mV4)/(+sV4))^p)^q)^r)-0.24021)/0.71807*0.6+0.4
The first part of this: M= ((kV1*((V1-mV1)/(-sV1))^p+kV2*((V2-mV2)/(-sV2))^p+kV3*((V3-mV3)/(+sV3))^p+kV4*((V4-mV4)/(+sV4))^p)^q)
looks like a weighted metric sum, with kVi as the weight, mVi as the mean (or equivalent measure) and sVi as the standard deviation (or equivalent measure). If p was 2 and q was 1/2 this would be an L2, or Euclidian metric. The weighting function is then:
((M^r)-0.24021)/0.71807*0.6+0.4
I've no idea what this last bit is doing - adjusting the result to some preferred range of values I would guess.
I think there is a problem with the "-" signs on sV1 and sV2. I think these signs indicate the fact that FWHM and Eccentricity are better if they are smaller, while NoiseRatio and Stars are better if they are larger. However, this allows the sum S= Sum((kVi*(Vi-mVi)/±sVi)^p) to be negative, in which case S^q will produce an error since q is not an integer. If the more conventional L2 metric is selected, these signs are lost (they are squared by p=2); the obvious alternative is to attach the signs to the weights kV1 and kV2, where they will have exactly the same effect in the current expression (and will produce the same errors).
To fix this sign issue the factor M= S^q should be replaced by M= max (S,0)^q.
 

fredvanner

Well-known member
Apr 17, 2019
412
55
Wells, Somerset, UK
Surely Weighting is always 1 = good, 0 = bad.
FWHM: smaller is better (0 is good);
SNR: bigger is better (1 is good);
I think we just "crossed in the post" on your weighting formula. Your revised version is a "standard" L2 metric with positive weights, so should not have the problem I identified ...
... however, I think the "small is better" problem is not handled correctly by this formula. For example, the expression:
Math.pow((FWHM-8.86075)/-3.59844,2) has a negative sign on the divisor, but this is immediately squared, so has no effect. The result is that FWHM is treated exactly the same as NoiseRatio and Stars ("bigger is better").
Note that your original default formula does not have any of these problems.
 

Juan Conejero

PTeam Member
Sep 2, 2004
7,648
119
56
Valencia, Spain
pixinsight.com
I didn't know you could use PCL functions (e.g. Math.pow(x,y)) in Weighting / Approval expressions ...
... in fact, I've no idea what syntax is allowed in these expressions - is it documented anywhere?
The expression above looks wrong [ ...(0*Math.pow(..., ...) will be 0 whatever in in the Math.pow arguments]; not a problem, but can't find errors unless the expresion is correct.
It would also help if you could post one of the images producing the error.
SubframeSelector expressions are standard JavaScript source code and can use most of the PixInsight JavaScript Runtime. Of particular interest is the fact that you can use the standard Math JS object with all PJSR extensions. This is well described in SS's official documentation, which you can read on the Process Explorer window.

If an expression evaluates to a non-numeric entity (such as NaN, +Inf or -Inf), this is because the expression is trying to perform an illegal operation, such as the logarithm of a negative number, raising (Math.pow()) a negative number to a non-integer exponent, a division by zero or by an insignificant value, etc. So the culprit here is the expression, not SubframeSelector.
 

fredvanner

Well-known member
Apr 17, 2019
412
55
Wells, Somerset, UK
This is well described in SS's official documentation, which you can read on the Process Explorer window.
I had looked for the SS documentation here, but the documentation browser window was empty. Following your note above, I extracted the file path from the document browser toolbar, and found the documentation on my disk. Is there any obvious reason why the documentation browser in the process explorer might fail to open a documentation file (for which it is displaying the correct path)?
 

bulrichl

PTeam Member
Nov 2, 2016
890
67
La Palma, Canary Islands
I confirm that although the correct file path is displayed in the document browser toolbar, no documentation is displayed in the documentation browser (Windows 10 Pro, PI v1.8.8-6). o_O

Bernd
 

JST200

Active member
Sep 29, 2018
37
0
Hi Fred,

Thank you for your analysis. :)

Although Alex kindly coauthored me on the spreadsheet, it was more for some work I did on the excel side rather than the maths. I'll ping Alex with a pointer to this thread to see if he has any comment to make on the maths side if things.

Hi Juan,

I wasn't so much pointing the finger at a problem in the process. It was just a surprise that what used to work, suddenly stopped working. Since the only thing that I could remember changing was the new version of PI, I wanted to check whether a PI change could have had an effect. I guess something else must have coincidentally changed at the same time. Still it's all good now.

So, thanks very much, both, for your inputs. :)

Cheers, Jim
 

fredvanner

Well-known member
Apr 17, 2019
412
55
Wells, Somerset, UK
Having examined Alex Woronow's spreadsheet, I now understand the weighting expression, and it does indeed work OK (since each of the weighting terms is constructed to always be >=0). I would advise setting the "Gamma stretch" to 1 (I think it should be applied outside the norm calculation, not half way through, but 1 is safe). However, the generated expression only applies to the data set for which it was calculated - it can't be applied as a general expression.
The basic weighted norm (M in my post above) can be written in generalised SS expression form, using SS variables, as can the final pedestal calculation (... *0.6+0.4 ), but the global normalisation (... -0.36068)/0.62589 ... in your example) requires an additional loop through the frame data, which can't be done in SS (as far as I know). The following gives the general idea:
JavaScript:
w1=0.1;w2=0.1;w3=0.1;w4=1-w1-w2-w3;
v1=w1*(FWHM-FWHMMax)/(FWHMMin-FWHMMax);
v2=w2*(Eccentricity-EccentricityMax)/(EccentricityMin-EccentricityMax);
v3=w3*(NoiseRatio-NoiseRatioMin)/(NoiseRatioMax-NoiseRatioMin);
v4=w4*(Stars-StarsMin)/(StarsMax-StarsMin);
L2=Math.sqrt(v1*v1+v2*v2+v3*v3+v4*v4);
pedestal=0.2;
L2*(1-pedestal)+pedestal
 

JST200

Active member
Sep 29, 2018
37
0
Hi Fred,

As promised, I've pinged Alex and hopefully he will reply.

Thanks again for your very helpful input. I said before I'm not good on the maths. That's not strictly true. I understand the maths, its the effect it has that's less clear. Fortunately Alex does.

There are so many areas to learn about in this hobby, it makes flying model helicopters a breeze in comparison. Lol!

Stay safe.

Cheers, Jim
 

fredvanner

Well-known member
Apr 17, 2019
412
55
Wells, Somerset, UK
... setting the "Gamma stretch" to 1 (I think it should be applied outside the norm calculation, not half way through ...
[Probably for Alex...] The gamma stretch is done (correctly, I believe) outside the norm in the post-normalisation column (column S), so I suspect it is a typo in the weighting expression (in cell C13):
... ,"&$X$4&"),"&$L$6&"),"&1/$X$4&"))-" ... should be:
... ,"&$X$4&"),"&1/$X$4&"),"&$L$6&"))-"

... but I don't think it will make much difference!
[end of geeky digression]
 

JST200

Active member
Sep 29, 2018
37
0
Hi Fred,

Your forgiven. Lol

If you are interested in the spreadsheet (rather than just the calls therein), there is an updated version available. Let me know and I'll aid a pointer or else get Alex to update his download site.

Cheers, Jim
 

aworonow

Well-known member
Feb 19, 2013
279
0
faintlightphotography.com
The correction indicated by Fred Vanner is correct. Attached is a corrected version. Thanks Jim and Fred!!!
If you have any difficulties, requests, ideas...don't hesitatie to contact me: alex(at)FaintLightPhotography(dot)com
 

Attachments

aworonow

Well-known member
Feb 19, 2013
279
0
faintlightphotography.com
OK, after making all corrections, I still got the NAN error and the SubframeSelector would no longer plot output until PI was shutdown and restarted. I think the NAN arose because the equation from one data set (e.g., green) was left in the equation box and a new data set (say blue), using the old equation, caused the NAN. Solution: delete the equation before processing new data.