Winsorised sigma clipping rejection values

[blinky]

guys,

How do I work out what pixel rejection settings to use for Winsorised sigma clipping? If I leave it at default and look at the high pixel rejection image I can see most of the galaxy I am stacking, the main image still looks good but I'm worried I'm rejecting good data. I tried ramping the value Al the way up to 10 but still see loads of pixels rejected in what I would have thought was a high signal area. I just wondered if this is normal and if anybody has any tips for determining optimal values ?

[RickS]

Hi Blinky (Bill?)

First do an integration with no rejection.  The median noise reduction printed on the process console is your target.  That's the best you can do.  Now adjust the rejection parameters while checking the total rejection percentages in the process console output and also the rejection maps and the integration result.  You want to choose parameters that get you close to the target noise reduction without allowing any artifacts like hot/cold pixels, cosmic ray hits and satellite trails to appear in the integration.  I usually stop once I get within a couple of percent of the target.  If you don't have a lot of data or the quality is poor you might have to be satisfied with a bigger delta from the target.

BTW, the noise estimation is imprecise, so you might get the occasional surprising (probably incorrect) median noise reduction value.

If you have very few subs or very many you might also want to consider trying other rejection algorithms, e.g. percentile clipping is good for very small data sets and linear fit is good for very large ones.

Cheers,
Rick.

[blinky]

Thanks ric,

Will have a look at his tonight, it's craig by the way, really should add it to my footer on here.....

[blinky]

OK, Im still confused...... What noise figure am I looking for/at?

[RickS]

Sorry, Craig, it was a little joke.  Blinky Bill is a koala in a book for children but probably not well known outside Australia and New Zealand 

The number you are looking for is at the end of the output from ImageIntegration.

Cheers,
Rick.

[blinky]

OK, I tried what you said, with no rejection the median noise was around 2, with the default rejection params it was 1.8 - is this correct?  does that mean the defaults are pretty good for this image set?

[blinky]

and i just discovered i never ticked they were to be debayered

[blinky]

here is the output after i selected to debayer them!

[RickS]

here is the output after i selected to debayer them!

You'll need to also do it again with debayer enabled and no rejection to get
an apples to apples comparison.

If possible, it would be good to see an extra couple of lines back in the
output of the run with rejection.  The line starting with "Total" will tell
us what percentage of pixels were rejected at the lower and upper limits.

Cheers,
Rick.

[blinky]

Total :    637631   0.520% (    86160 +    551471 =   0.070% +   0.450%)

MRS noise evaluation: done
Computing noise scaling factors: done

Gaussian noise estimates  : 1.2648e-004 1.0951e-004 1.1244e-004
Scale estimates           : 1.6308e-003 1.3722e-003 5.8446e-004
Location estimates        : 2.4922e-002 2.2765e-002 1.7651e-002
SNR estimates             : 4.3076e+004 5.0812e+004 3.0078e+004
Reference noise reduction : 3.7772 4.7205 3.2321
Median noise reduction    : 4.3417 4.7151 3.3013

[RickS]

Hi Craig,

I'm a bit confused about the relationship between the different snapshots you posted so I'll just give you an example from some of my data.  I only do mono, but it shouldn't be difficult to generalize to OSC.

Here's an integration with no rejection of 48 Luminance frames:

Total :         0   0.000% (        0 +         0 =   0.000% +   0.000%)

MRS noise evaluation: done
Computing noise scaling factors: done

Gaussian noise estimates  : 1.3917e-004
Scale estimates           : 2.3246e-004
Location estimates        : 2.9326e-002
SNR estimates             : 6.2051e+004
Reference noise reduction : 1.7233
Median noise reduction    : 1.7216

The interesting thing here is the median noise reduction value of 1.7216.  This is my target for further integrations with rejection.  I want to get close to this target without making the rejection too weak and allowing hot or cold pixels, cosmic ray hits, satellite trails, etc. to appear in my integrated master.

Here's an integration with rejection parameters that are too strong:

Total :  16746392   4.189% (  1516681 +  15229711 =   0.379% +   3.810%)

MRS noise evaluation: done
Computing noise scaling factors: done

Gaussian noise estimates  : 1.5448e-004
Scale estimates           : 2.4126e-004
Location estimates        : 2.8720e-002
SNR estimates             : 4.8401e+004
Reference noise reduction : 1.6114
Median noise reduction    : 1.6098

You can see that the median noise reduction is well below the target.  It's also obvious that I'm rejecting a lot of data, especially on the high side.  The values of 0.379% and 3.810% show the percentage of data values rejected by the low and high parameters respectively.

Here's an integration that's pretty good:

Total :    587071   0.147% (    88557 +    498514 =   0.022% +   0.125%)

MRS noise evaluation: done
Computing noise scaling factors: done

Gaussian noise estimates  : 1.3778e-004
Scale estimates           : 2.2871e-004
Location estimates        : 2.9152e-002
SNR estimates             : 6.2719e+004
Reference noise reduction : 1.7128
Median noise reduction    : 1.7111

The median noise reduction is only about half a percent below the target.  The amount of data being rejected is pretty small.  I've also inspected the result carefully and can't find any artifacts that shouldn't be there.  Voila!

Note that sometimes you won't be able to get this good a result.  If you have limited data or poor quality data you might only be able to get within a few percent of the target.  Also, the noise estimation isn't infallible and occasionally you'll get unexpected results (like a median noise reduction that is better than the target.)  In these cases I typically go by the rejection percentages since I have enough experience with my gear that I know roughly what to expect.

Cheers,
Rick.

[blinky]

Thanks Ric - that makes things a little clearer, although I would have though that in the example where you say the median noise reduction is well below target - I would have thought that the difference between the original with no rejection of 1.7216 would have been close enough to the 1.6098 value, I need to have a think about percentages, math never was my strong point!

[RickS]

Thanks Ric - that makes things a little clearer, although I would have though that in the example where you say the median noise reduction is well below target - I would have thought that the difference between the original with no rejection of 1.7216 would have been close enough to the 1.6098 value, I need to have a think about percentages, math never was my strong point!

The difference between the initial integration with rejection and the better one is about 6%.  That might not seem like a lot, but it wasn't hard to do and could easily be equivalent to the addition of hours of data.  In some situations the gain is greater than this example.  Processing, IMHO, is all about making many small improvements which add up to a big difference in the final image.

Cheers,
Rick.

[cfranks]

Rick,

Thanks for your explanation, I have always had problems in understanding in this area. 
Do I attempt to get as close to that no rejection Median NR as I can, without being too precise?  Above or below?  My usual start point for Sigma High, 3.5, gives me (on my current project) a figure above that reference value and I mentally fall in a heap since I assumed the reference number was the max achievable.
I'm using Average combination with the standard defaults, Winsorized SC, Sigma Low 4.0, Sigma High 3.5 and 20 subs.

Cheers
Charles

[RickS]

Rick,

Thanks for your explanation, I have always had problems in understanding in this area. 
Do I attempt to get as close to that no rejection Median NR as I can, without being too precise?  Above or below?  My usual start point for Sigma High, 3.5, gives me (on my current project) a figure above that reference value and I mentally fall in a heap since I assumed the reference number was the max achievable.
I'm using Average combination with the standard defaults, Winsorized SC, Sigma Low 4.0, Sigma High 3.5 and 20 subs.

Cheers
Charles

Hi Charles,

In theory the no rejection number is the best you can do but sometimes the noise estimates do bounce around unexpectedly.  In a case like that I'd just see how low I can get the rejection percentages and still end up with a clean integration.

Cheers,
Rick.