SubFrameSelector question

[EorEquis]

I've recently begun exploring the SFS script, and have been extremely pleased with the results.  The improvement in my initial integrations is remarkable.

My current approval expression considers the Subframe Median and Subframe MeanDeviation for FWHM and SNRWeight.  Specifically, I seek to keep frames that are within two deviations of the Median value for those two properties.  Currently, I calculate that value manually.  For any given set of frames, I observe the Median, and add or subtract (as appropriate) 2x the value of the MeanDeviation, and then plug that value into my expression.

My question :  Is there a way for me to write the expression to do that calculation for me on each set of data?  I'm not seeing any property I can include in the expression that says "Subframe Median of Property X" for example, and while I admit my grasp of statistics is tenuous at best, I don't believe any of the -Sigma properties do the job for me.

If it helps clarify, here is an example :

FWHM < {X} &&  SNRWeight > {Y}

Where:

X = FWHM Subframe Median + (2 * FWHM Subframe MeanDeviation)
Y = SNRWeight Subframe Median - (2 * SNRWeight Subframe MeanDeviation)

Thanks in advance...

[mschuster]

You should be able to use the Sigma properties to do this.

Sigma normalizes the property with respect to the median and mean absolute deviation of the underlying property, measured across the entire frame set.

To help understand this, as a quick test, please set the Weighting Expression equal to FWHMSigma and the Plot Ordinate equal to Weight.

You should see that a frame whose FWHM equals the median FWHM of the whole set of frames has a FWHMSigma value equal to 0.

A frame whose FWHM equals one mean absolute deviation larger than the median FWHM has a FWHMSigma value equal to 1.

And a frame whose FWHM equals one mean absolute deviation smaller than the median FWHM has a FWHMSigma value equal to -1.

Likewise for SNRWeightSigma.

So your approval expression may be expressed as FWHMSigma < 2 && SNRWeightSigma > -2, given that your thresholds are 2 mean absolute deviation units for both properties.

Best regards,
Mike Schuster

[EorEquis]

Ah, that explanation makes sense...the language was simply over my head.

Thank you much for the clarification!

[Eddy Timmermans]

Is it more efficient to select, for example, only the best 100 frames that have, for example, a FWHM value less than 3, than to stack all 300 frames that all have a FWHM value less than 4.5 ?
To put it another way, will the added signal, produce a better image even if the quality is less ?

Are ther some objective values that I need to take into consideration ?
For example FWHM better than 3, or only the lowest SNR values ?

Eddy

[mschuster]

Eddy,

Try integrating both ways and see what looks best.

My setup is undersampled, not seeing sensitive, FWHM does not vary much, and I discard only the real outliers.

Previously, FWHM did vary due to a focusing problem, but now that is solved.

SNR measurements sometimes don't deal well with thin high clouds and wild gradients, so I like to check for those by blinking. Again, try several integrations.

Mike

[jerryyyyy]

Thanks for all tehse explanations.  The documentation on this scipt exists, thank-god, but it is dense. 

[EorEquis]

My hope, with my approval expression, is to keep all of the data that approximates the best I got that evening.

Conditions change, equipment changes, and so on.  There is no hard number for FWHM, or SNRWeight, or anything else imo, that's going to always represent the "right" data to keep.  So, asking to keep "All data with an FWHM of 4 or less" is going to reject some of the better frames some nights, but keep some of the lesser ones on others.  So too will some values vary from one object to the next, or one piece of equipment to the next.

Instead, then, I sought a way to reject outliers, and keep a significant sample.  Thus, my expression to keep things that fall within 2 deviations of the median.  In that way, if I had a significant number of bad frames (perhaps seeing went south, or focus slipped) I don't include poor frames in order to "force" some percentage, but if I have a particularly good night, I don't toss otherwise acceptable frames for the same reason.  By the same thinking, I don't keep outliers that happen to just sneak in under some arbitrary value, but are well out of line with other frames, and don't reject good frames that are just barely above/below that value on nights when conditions may not have been perfect.

IMO...and it's likely not worth much, newbie that I am...the two properties that seem to be most determinant for MY tastes and processing skills are FWHM and SNRWeight.  So, my expression has become:

FWHMSigma < 2 -- FWHM Less than 2 deviations above the median
&&
SNRWeightSigma > -2  -- SNRWeight less than 2 deviations below the median
&&
WeightSigma > -2 -- Result of Weight expression less than 2 deviations below the median

My Weight expression (called MYWEIGHT) is :

(2 * SNRWeight) - FWHM

Which, when combined with the evaluation in the expression above, allows me to kick out frames that just barely squeaked by the first two test, but are near the "worst" end of both.  In other words, I'll keep a particularly low FWHM even if it's a bit noisy, or a particularly high SNR even if the FWHM is a bit high...but I don't want to keep one that just snuck under the limit for both.  SNRWeight-FWHM was tried, but resulted in negative values every so often, which then tripped up Image Integration later on.

Finally, MYWEIGHT is written to the headers, and used in Image Integration for "Weights".

=========

Is any of this "right" or "good"?  I have no idea.  As I made clear by asking the question in the first place, I obviously know just enough statistics to be dangerous to myself and others.  But, for me at least, the results have been really rather dramatic.  Sessions that previously produced integrations that proved too noisy to process easily, and seemed to lack sharpness, now produce integrations that are a joy to work with.

[mschuster]

Very good, thank you for sharing!
Mike

[Eddy Timmermans]

Mike,

I used the expressions suggested by EorEquis and they performed very well.
My result was clearly superior to the image produced by DSS, even though I had no flats.
When I used DBE on the result it almost entirely cleared up the vignetting in the image.

I used only 126 frames out of 140. I had a total of 280 images made over 2 nights.
I will surely try to stack them using the frames tha pass the check used in SubFrameSelector.

Less appears to be more ! ;-)

Eddy

[EorEquis]

Mike,

I used the expressions suggested by EorEquis and they performed very well.
My result was clearly superior to the image produced by DSS, even though I had no flats.
When I used DBE on the result it almost entirely cleared up the vignetting in the image.

I used only 126 frames out of 140. I had a total of 280 images made over 2 nights.
I will surely try to stack them using the frames tha pass the check used in SubFrameSelector.

Less appears to be more ! ;-)

Eddy

Glad they worked for you, Eddy!