Author Topic: K Sigma Noise Thresholding In Wavelets  (Read 9284 times)

Offline Jack Harvey

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 975
    • PegasusAstronomy.com & Starshadows.com
K Sigma Noise Thresholding In Wavelets
« on: 2010 February 07 07:09:09 »
Snowy weekend with no new data so have been experimenting with Atrous  Wavelets.  Need some idea of what the K SIgma Noise Thresholding does?
Jack Harvey, PTeam Member
Team Leader, SSRO/PROMPT Imaging Team, CTIO

Offline Jack Harvey

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 975
    • PegasusAstronomy.com & Starshadows.com
Re: K Sigma Noise Thresholding In Wavelets
« Reply #1 on: 2010 February 07 07:17:12 »
Seemed to have answered my own query.  If you set the Atrous Wavelets to the defaults and only check the K Sigma you  get a nice noise reduction effect.  SO now all i have to do is either use this as an additional noise reduction tool and or figure out how to integrate it with the Noise Reduction settings when using wavelets.
Jack Harvey, PTeam Member
Team Leader, SSRO/PROMPT Imaging Team, CTIO

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: K Sigma Noise Thresholding In Wavelets
« Reply #2 on: 2010 February 07 19:40:41 »
Jack
Good trick..!
What is happening when we do this..??
More info please..!!

Dave
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: K Sigma Noise Thresholding In Wavelets
« Reply #3 on: 2010 February 07 20:32:04 »
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
Re: K Sigma Noise Thresholding In Wavelets
« Reply #4 on: 2010 February 08 05:47:49 »
Hi all,

Juan and I exchanged some general comments about Winsorized Sigma Clipping - which may help you understand K-Sigma clipping in general.

Have a look at
http://pixinsight.com/forum/index.php?topic=1611.0

When I get more time, I'll try and explain Sigma Clipping again (but, you may have to remind me, possible several times ::)).

Cheers,
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline Jack Harvey

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 975
    • PegasusAstronomy.com & Starshadows.com
Re: K Sigma Noise Thresholding In Wavelets
« Reply #5 on: 2010 February 08 06:00:27 »
Thanks.  I will have to read that discussion a couple of times but I think I get it and understand that the tool just replaces outliers at using data at the clipping point, which in this case has sliders you can adjust<G>.  So it works a a form of sigma clipping, and as I have noted does deal with noise effectively. 

Glad you are back from France and on the boards again<G>
Jack Harvey, PTeam Member
Team Leader, SSRO/PROMPT Imaging Team, CTIO

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
Re: K Sigma Noise Thresholding In Wavelets
« Reply #6 on: 2010 February 08 06:14:04 »
Hi Jack,

The fundamental premise behind all 'K-Sigma' sliders is that the software looks at the image data and establishes the 'Mean' (or Median) and the Standard Deviation (StDev).

The 'StDev' represents the amount of 'spread' of the data, and I can't remember the exact maths here as I am not near my reference books, but a 'single' StDev (or Sigma = 1) should contain (and this is purely an 'example figure' here), say, 95% of the data.

If you apply a 'multiplier' (Sigma = 2.00) to the StDev figure, then you are telling the software that you want to include MORE of the OUTLYING data - so a Sigma=2.00 applied to the above thought experiment might yield 96.3% of included data.

However, if you have some data that IS 'outlying' at the very first inspection, then this data will have 'skewed' the original Mean or Median, around which the (Sigma x StDev) capture window will be centred. This is where Winsorized Sigma Clipping comes in to play - as it can start to 'intelligently' exclude these outlying data values, providing a more and more 'robust' estimate for the true 'Mean' of the data set as the algorithm 'loops' (or iterates).

And many of the PI algorithms that include 'Sigma' also allow Sigma to be independently specified for the 'lower' and 'higher' ends of the distribution curve - very important as our astronomical data tends NOT to be equally distributed on each side of a mean value.

All that said, I am basing THIS response on my more detailed understanding of the ImageIntegration process, NOT the ATWT process that you initially asked about. However, if you just bear in mind the fundamentals, it may help:

1.) Analyse the data set
2.) From the statistical analysis, determine the Mean and the StandardDeviation of the data
3.) Expand the 1-Sigma deviation (StDev) by a factor, K, such that the included data is represented by (K x StDev) - often just abbreviated to 'Sigma'

Cheers,
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: K Sigma Noise Thresholding In Wavelets
« Reply #7 on: 2010 February 08 12:48:06 »
Hey folks...
Its me... >:D

So is all this something like the "kappa sigma median" setting,(as opposed to "kappa sigma clip") that DSS uses..?
Because I always found I got better stacks with a "kappa" (or was that a "sigma"...) of around 1.3...

Now I am confused with the "image integration" (stacking) feature in Pix...and the recommended setting of 4-5...
What am I missing,or on to here..?
Is there a correlation..?
Maybe someone who knows DSS can chime in here...(Luc,where are you...?)

Wavelets stuff continues to scare me ???
Niall;it should right up your alley...!!

Dave
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
Re: K Sigma Noise Thresholding In Wavelets
« Reply #8 on: 2010 February 09 05:19:44 »
Quote
Wavelets stuff continues to scare me

Don't worry - I am pretty afraid of them as well ;D

I mean, I have read, and re-read all about them, and I 'think' I know how to use the ATRW process to my advantage - but I don't really think I use more than about 5 or 10% of the power of the tool. So, I doubt whether I am going to be of much help - not yet, anyway ::)

Cheers,
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: K Sigma Noise Thresholding In Wavelets
« Reply #9 on: 2010 February 10 13:34:00 »
I am not 100% sure that this "k sigma" wavelet noise rduction,in linear data is so great...
It DOES give an "odd" background...certainly under the STF it looks odd...
Greycstoration,in low doses also seems to help.
Can Juan,Jack or anyone comment more about it...?
What about the PARAMETERS...
And what about the "noise reduction" option...??
Surprised I am not hearing more...

CLEAR sky tonite,and I have swapped to an f/5 20cm Newtonian...
No more Mr Nice Guy... >:D

Dave
PS my latest work on the M1 data;
http://www.flickr.com/photos/daveh56/4344696876/sizes/l/
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline Jack Harvey

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 975
    • PegasusAstronomy.com & Starshadows.com
Re: K Sigma Noise Thresholding In Wavelets
« Reply #10 on: 2010 February 11 05:07:03 »
No experience with linear data.  As I mentioned before I use any noise reduction at the very end so not as to reduce signal early in the processing.  Any of the noise programs i use (GreyC, ACDNR or the SIgma) are done darn near as a last part of the processing.  Also if I have good signal and little noise in the object I will mask it and only work on the background or periphery of the object.  Finally i only apply noise reduction to the combined "final" RGB image not the individual master frames.
Jack Harvey, PTeam Member
Team Leader, SSRO/PROMPT Imaging Team, CTIO

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: K Sigma Noise Thresholding In Wavelets
« Reply #11 on: 2010 February 11 16:49:57 »
Jack
My error,,,I was applying it at linear,under SCT...
Works GREAT AFTER (!!) histo,...
So what combo of GreyC,ACDNR,and tha ATrous voodoo do we use,and in what order...????

Dave
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline Jack Harvey

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 975
    • PegasusAstronomy.com & Starshadows.com
Re: K Sigma Noise Thresholding In Wavelets
« Reply #12 on: 2010 February 11 17:44:03 »
usually you should only use or need one once you have it adjusted for an image.  I like GreyC and use it often.  The previous arguments for ACDNR are certainly compelling and of course I stumbled on K which I will probably not use as much.  My favorite is to make a preview of the periphery of the object and some background and then star applying decreasing amounts of Grey C till I am just correcting and then apply to image


Jack Harvey, PTeam Member
Team Leader, SSRO/PROMPT Imaging Team, CTIO

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: K Sigma Noise Thresholding In Wavelets
« Reply #13 on: 2010 February 12 04:14:29 »
So now you are cooling on K...?
It seems so handy and quick...I have been making it a "first stop" after a 3/4 histo stretch...
Then SOME GreyC..and then ACDNR to the background...
But so many setting options...
Today I JUST discovered that "Multiscale recursive" will "depixilate" a upsampled image,for example.
God knows what else is in ACDNR... >:D
 
In ACDNR...any way to get a better Lum mask..?
Make your own ?? (probably)...the sliders seem "clunky"...I seem to get a lot of "moss" hanging off my stars (? bad guiding)...
I often "hand make" my own mask...then do NR on it,etc...


Dave
« Last Edit: 2010 February 12 05:34:24 by dhalliday »
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: K Sigma Noise Thresholding In Wavelets
« Reply #14 on: 2010 February 14 17:28:23 »
I have abandoned "k" NR..
You lose object detail,ie high SNR areas do not do well
Here is another go at M1 data WITHOUT "k" NR
Morphology was used to "clean up" everything,and a great tool it is !!

http://www.flickr.com/photos/daveh56/4358080412/sizes/l/

Dave >:D
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME