Author Topic: Image Integration, Winsorized Clipping, Default  (Read 21150 times)

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Image Integration, Winsorized Clipping, Default
« on: 2010 February 03 00:39:14 »
Hi,

a couple of questions about ImageIntegration:
- The purpose of Winsorized Sigma Clipping is to remove outliers from the data. Causes for outliers can be hot pixels, airplanes passing through the image, ... . But will it also reduce noise?
- What would be typical pixel rejections rates? Applying the default sigma of 2.0 on my Canon EOS 40D images yield something like 10-15% rejection. Using a sigma of 3 is around 1%, which seems to be more reasonable.
- Considering that the noise characteristics of Canon EOS 40D images are different for the 3 channels (red has the most noise), would it be necessary to have different sigmas for the 3 channels?
- From reading other posts, I have the impression that "Weights=Noise Evaluation"+"Winsorized Sigma Clipping" are the most powerful options. Why aren't they the defaults that are selected when starting the ImageIntegration dialog?

Cheers,
Georg
 
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: Image Integration, Winsorized Clipping, Default
« Reply #1 on: 2010 February 03 04:03:47 »
Georg
This is new to me..
Are you talking about stacking subs/calibrating them..?
My understanding was that Pix did not even do this..
I need to hear a bit more about your question...
excuse my ignorance here :-[
I know when I stack subs in DSS these sorts of features are available (ie averaging around a "sigma" value..)
What Pix trick/tool am I missing out on here..?

Dave
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline Jack Harvey

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 975
    • PegasusAstronomy.com & Starshadows.com
Re: Image Integration, Winsorized Clipping, Default
« Reply #2 on: 2010 February 03 06:00:37 »
Dave the Image Integration tool is a real gem.  It is used to stack subs.  After you have calibrated your images and registered them you then open Image integration and load a stack of images (usually from a specific filter, i.e. Red filter).  Then set the parameters.  I am not going to go throuhg all but will tell you I use Average, Normailzation is additive  and Scaling and weighting by Noise.  For Pixel Rejection I use WIndsorized Clipping and Scale and Zero Normalization.

Hit the button and you get three frames.  THe rejected high and low pixel frames and the Integration Frame which is you image with lots of the noise now removed.
Jack Harvey, PTeam Member
Team Leader, SSRO/PROMPT Imaging Team, CTIO

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1458
    • http://www.harrysastroshed.com
Re: Image Integration, Winsorized Clipping, Default
« Reply #3 on: 2010 February 03 09:39:46 »
Hi George

From my understanding the sigma clipping only removes noise as  " unwanted pixels from things like hot pixels plane trails etc " the noise reduction comes from std average stacking of you images ( This part is no better than other programs as its simple maths )
I personally find no benefit from separating my rgb and use a sigma setting from 4 to 8  and a average of 5 and usually only reject probably less than o.25 % of pixels
If you reject to many good pixels you can end up with a noisier image  ???

I agree generally about default settings and in general it would be nice if we could save new default settings on all tools  ;D  ( Please Mr Juan )

Dave

Now you have not been watching these video's again    >:D    , we will have to send you to stand in the corner  :'(

Pixinsight has excellent stacking properties , its only calibration that can not be done easily  ( yes I know you can do it in pixel maths  :footinmouth:)


Go and have a look and be impressed


Harry
Harry Page

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
Re: Image Integration, Winsorized Clipping, Default
« Reply #4 on: 2010 February 03 09:41:36 »
Hi all,

I have spent a LOT of time messing around with ImageIntegration, and have had quite a few exchanges with Juan on the whole issue of 'robust statistical analysis' - which is at the heart of II.

I would love to be able to give you a detailed insight, but I am 'lost in France' at the moment, sruck in some crummy hotel who seem to think that InternetAcees was a rock'n'roll band from the late '60s  :yell:

Maybe I can get something written up 'offline' - but I wouldn't then have access to a lot of my notes and scribbles. So, unless someone else wants to step up to the mark, my response might take a week or so to put together.

Cheers,
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline Niall Saunders

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1456
  • We have cookies? Where ?
Re: Image Integration, Winsorized Clipping, Default
« Reply #5 on: 2010 February 03 09:45:52 »
Quote
yes I know you can do it in pixel maths

Unfortunately, no - you can't  :'(

Not if your 'source' images (Bias, Flats, FlatDarks) are initially ('natively') FITS 32-bit Float images, with data in the 00000 to 65535 range.

I have yet to see my reported 'bug' problem actually get resolved. There has been NOTHING that I have been able to do to get the image data correctly scaled, and that currently means that all of my attempts at calibration within PI will fail.

The latest suggestion that I have seen is that I need to change from a wireless keyboard to a wired keyboard - but I think that is a suggestion better reserved for April 1st !!!

Cheers,
Cheers,
Niall Saunders
Clinterty Observatories
Aberdeen, UK

Altair Astro GSO 10" f/8 Ritchey Chrétien CF OTA on EQ8 mount with homebrew 3D Balance and Pier
Moonfish ED80 APO & Celestron Omni XLT 120
QHY10 CCD & QHY5L-II Colour
9mm TS-OAG and Meade DSI-IIC

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
Re: Image Integration, Winsorized Clipping, Default
« Reply #6 on: 2010 February 03 10:54:15 »
Hi George

From my understanding the sigma clipping only removes noise as  " unwanted pixels from things like hot pixels plane trails etc " the noise reduction comes from std average stacking of you images ( This part is no better than other programs as its simple maths )
I
Harry

I find the rejection integration and integration is much more powerful the most if not all programs I have tried.

It does more than removing airplane trials. It will improve you image's S/N if done right. 

You should dither your sub images to receive the Maxium benifit. 

Generally 1-1.5 or less percent are need to be rejected.

This varies a lot sky conditions, dither, camera etc. 25% rejection is way to much.
If you have CCD with fewer hot pixels you will see/ need less % rejected.

My CCD is fairly easy to see if the rejection is good. The sensor is big enough there are plenty of cosmic ray hits and often a sat. trial to reject. I move the slider up until they are pretty much gone. I see the result on the high Map after STF.

I watch the S/N outputs on consul if you doing better then numbers decrease. I wish knew more about the consul output. Ie S/N verses S/N increament. Someday they will let me in on the secret.  ???

Max




Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1458
    • http://www.harrysastroshed.com
Re: Image Integration, Winsorized Clipping, Default
« Reply #7 on: 2010 February 03 11:11:18 »
Hi

Just to repeat I did say    Zero . 25 %   or 1/4 of a percent not 25 %   :P

Yes I get the best results from using the tools in pixinsight     :D    if you want the best S/N do not reject any pixels at all  ,  Pixel rejection does not increase your signal to noise   :)
Using the noise evaluation  ( weight ) make the most of your images and this is where the improvement come from !

I will also say again 1 to 1.5 percent rejection on average is to high ( This is based on a 4 meg image ) ? I examine my rejection maps and change the settings till only the hot pixels / planes / sats etc are shown , there should not generally be a background showing in these maps !

As you know you can see on the council the amount of rejected pixels per image

All this is of course my opinion , which is often wrong  :yell:

Harry

« Last Edit: 2010 February 03 11:20:56 by Harry page »
Harry Page

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
Re: Image Integration, Winsorized Clipping, Default
« Reply #8 on: 2010 February 03 12:11:23 »
Hi

Just to repeat I did say    Zero . 25 %   or 1/4 of a percent not 25 %   :P

I will also say again 1 to 1.5 percent rejection on average is to high
As you know you can see on the council the amount of rejected pixels per image




I gave 1 to 1.5 percent as upper limit.
If you have more than this normalization if often not working well.
This can happen if add in some images with bad gradients weak clouds etc.
I  rejection will improve you S/N if you dither them.
Then non performing pixels are rejected out of the set of subs.

Max

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: Image Integration, Winsorized Clipping, Default
« Reply #9 on: 2010 February 03 12:42:10 »
Yikes !!
More gadgets...
Isn't this (sort of) what DSS does for me...???
I see it has the usual 8-28 "parameters"...
Still another weapon in my quivering belt... >:D
Perhaps one of you would be good enough to say a little more about this tool...and how it differs from DSS...
I have to say I like it !!
A "baseline" cookbook style set of settings would be a good starting point...What "percentile" rejection is a good one for example,etc.

Harry,I must have missed all this,AND the video...
Boy....my Pix powers will be up their with Dr Evil soon.. >:D

Dave
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1458
    • http://www.harrysastroshed.com
Re: Image Integration, Winsorized Clipping, Default
« Reply #10 on: 2010 February 03 12:48:12 »
Hi Dave

go watch http://www.harrysastroshed.com/Stack.html

As I have not used Dss I can not tell you how iy differs  ???


Harry
Harry Page

Offline dhalliday

  • PixInsight Old Hand
  • ****
  • Posts: 307
    • on Flickr
Re: Image Integration, Winsorized Clipping, Default
« Reply #11 on: 2010 February 03 13:14:41 »
Harry
Just did my homework...can I have a cookie now..?

Well obviously I will have a "throwdown" between DSS and Pixinsight...winner takes my business.. >:D
But I will still use DSS to "calibrate"...
I understand Luc,the developer of DSS is/was a Pix user.
Maybe he does not image these days...just wondered why I never see him around (here)...
? Software politics ??

Thanks again Harry!!
What NUMERICAL value is "sigma"...and why choose 5..??

Dave
Dave Halliday
8" Newtonian/Vixen VC200L/ TV 101,etc etc
SSAG/EQ6
CGE Pro
SBIG ST2K,ST10XME

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1458
    • http://www.harrysastroshed.com
Re: Image Integration, Winsorized Clipping, Default
« Reply #12 on: 2010 February 03 13:23:17 »
HI

I do not know the tech answer for sigma,   But I know that the higher the number the less the number of pixels rejected  8)

This number is not set and varies due to things like signal , but mainly the number of images you are stacking  with more images Pi is able to decide with more
certainty which are the outliers  :)  so you have to experiment to get the right number   not to high and not to low  :o

Harry

Go on then only One cookie  ;D
Harry Page

Offline Simon Hicks

  • PixInsight Old Hand
  • ****
  • Posts: 333
Re: Image Integration, Winsorized Clipping, Default
« Reply #13 on: 2010 February 03 13:29:58 »
DSS is so simple and does all the other stages of alignment as well. Its a joy to use (well done Luc). However, PI is pushing the limits and I suspect its somehow going to be better.  So how do I convince myself to learn how to do everything in PI?

Can anyone else tell us how the PI stacking differs from DSS? Is the PI version better? If so, why? Are there any comparitive results?

And assuming I start using the PI route for stacking....should I still use DSS to do the initial calibration (darks/flats/bias) and alignment....and just output 32bit calibrated/aligned images to read into PI?

Offline Jack Harvey

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 975
    • PegasusAstronomy.com & Starshadows.com
Re: Image Integration, Winsorized Clipping, Default
« Reply #14 on: 2010 February 03 13:44:38 »
Harry  You can have your defaults saved.  In tips and tricks see Basic Processing Icons.  If Image Integration is in your set with the proper defaults then you are good to go by starting a processing session with loading your Basic Processing Icon set
Jack Harvey, PTeam Member
Team Leader, SSRO/PROMPT Imaging Team, CTIO