Image Integration, Winsorized Clipping, Default

[Niall Saunders]

A pixel in a rejection map image has a real value directly proportional to the number of rejected (=clipped) source pixels in the integrated set at the corresponding map coordinates

That's what I had hoped to be the case (and it was what my experiments were tending to suggest) - so, by careful analysis of the 'cliiping images' it becomes possible to determine whether a pixel site is/was 'bad' over the entire range of source data. I can, once more, see another use for the 3-D viewer script (a script that I feel could now be considered for implementation into the 'core' of PI)

Cheers,

[Niall Saunders]

Correct me if I haven't understood what you want to do, but I detect some statistical misconception here. The fact is that you really don't want to know what the sigma value has to be in order to reject a given fraction of pixels in the integrated stack. You want to find an optimal sigma value that achieves a good compromise between efficient rejection and minimal SNR degradation. Usually the process to find the correct sigma is iterative: you implement a sort of binary search where the correct value is within the limits of a narrower boundary at each iteration.

If you want to know that sigma value in advance because you already know how many pixels you want to get rejected, then you don't need sigma clipping at all, but something without any statistical basis, such as min/max clipping.

In cases where you don't have enough images to substantiate a sigma clipping procedure (say less than 8 or 10 images), you can use either percentile clipping or averaged sigma clipping. Both algorithms utilize robust statistics and provide excellent results with reduced data sets.

OK - life is easier on the 'high end' clipping - I can iteratively adjust the High Sigma value until I eliminate (for example) ONLY those pixels hammered by incoming cosmic rays, or aircraft trails, etc. That is (relatively) easy. And, once I have tweaked the High Sigma to achieve 'just' that level of elimination, I can even then tweak the value down, just a little, to give a 'safety margin', if you will (tweaking DOWN to clip a few EXTRA pixels, just in case).

But, at the bottom end, I have less to go on. Sure, I can move the Low Sigma slider about, and can see my ClipLow image showing more, or fewer, clipped pixels as I make the adjustments. But, what do I use to determine WHERE to leave the slider setting? This was where I felt that some sort of 'empirical' decision might need to be made - for example, I might decide to be willing to 'clip' a total of 0.1% of all 'low end outliers'. I can then iteratively adjust the Low Sigma slider until the "Nr/N" calculation tells me that the total number of pixels that have been 'Low Clipped' (i.e. the cumulative ADU sum of all pixel values in the low-clipped image DIVIDED BY the simple number of pixels in the integrated image) reaches the target percentage of 0.1%.

My desire was to 'automate' this iterative procedure, such that I could 'request' a desired percentage of pixels to be clipped (whilst still relying on Winsorization) and the algorithm would set the Sigma slider accordingly. I don't actually end up 'caring' what the Sigma value becomes - but I see this as being 'different' to the algorithm that simply clips the requsite percentage of pixels WITHOUT the 'robustness' of Winsorized analysis.

Am I just trying to be 'too clever' here

Is there a 'better' way of trying to decide where to set the Sigma slider (especially for the more difficult ClipLow case)? In other words, is there a way to aim for 'maximum SNR' in the background (which is where the ClipLow will be affecting)?

Is there even an argument for using one variation of pixel clipping for the low-end data, and another for the high-end data?

My head hurts

Cheers,

[Harry page]

Hi

As I said before , why this thing for a fixed percentage ,  you only want to exclude the outliers 

I can manage to do a good job with the high setting and also the low setting , forgive me but we seem to be making a simple job hard 

Thats my 2p worth ignor me if you like 

Harry

[georg.viehoever]

Niall,

...
My desire was to 'automate' this iterative procedure, such that I could 'request' a desired percentage of pixels to be clipped (whilst still relying on Winsorization) and the algorithm would set the Sigma slider accordingly. I don't actually end up 'caring' what the Sigma value becomes - but I see this as being 'different' to the algorithm that simply clips the requsite percentage of pixels WITHOUT the 'robustness' of Winsorized analysis.
..

I am not sure, but percentile clipping may just be what both of use want.  There is no real need to have estimates of the mean or sigma in this case, so also no need for winsorization http://en.wikipedia.org/wiki/Winsorising. 

Georg

[Harry page]

Hi

Yes this is what I think , just use the percent clip if you want to

I only use the win / sig clip as I think this gives the best results you can achieve  and makes the most of my data 


Harry

[Niall Saunders]

forgive me but we seem to be making a simple job hard

No - that's what I am trying to 'avoid'. What I am trying to do is to establish a set of guidelines that allow you to set the LowClip Sigma slider to a 'level most appropriate' for your source data.

Now, we all have different data sets - so what works for one set may, or will, not work for the next. What, therefore, are the criteria that 'you' use to set the LowClip Sigma slider? When do 'you' decide 'yep, that's about right'? What makes 'you' decide 'nope, that's too much' or 'nope, not enough'?

Right now, I am of the (personal) opinion that I don't want to clip 'too much' of the bottom end data. "How much is too much?" is not a question I have an answer for - hence the decision to sacrifice some random, empirical percentage (either by asking PI to sacrifice 'exactly' that specific percentage, or by just tweaking the LowClip Sigma slider until I feel that the target percentage has been clipped.

However, what I really want to do is to maximise the background SNR - the aim we all have after all. And, as I see it, once we have acquired our basic data, and calibrated and aligned it to the best of our abilities, the 'only tool' we have left is "Image Integration" - either within PI (naturally ) or by 'other means'

So, unless someone can give me (show me, remind me of) a worked example of how to determine the 'ideal' position for the LowClip Sigma slider when using Winsorized Clipping, I don't actually know of a better way than the two (near identical) methods that I describe above.

And, setting a 'Percentile Clipping' point is NOT the same as setting a 'Winsorized LowSigma Clip' point in order to include a certain number of clipped pixels - at least that is not how 'I' understand things. Perhaps I need to run a set of data using both methods - aiming for the same number of clipped pixels in each case (i.e. the same 'percentage' of clipped data in each case) - and then compare the resulting two LowClip images.

By my deductions these will NOT be 'the same'. It will be interesting to see if they are. And, if they ARE different, which set of 'clipped data' represents a 'more robust' data extraction?

Sorry if this all sounds confusing - perhaps it is because IT IS 

Cheers,

[Harry page]

Hi


Ok we differ on what we exspect 

Harry

[Niall Saunders]

It's not that we 'differ' as such Harry - what I am trying to do is figure out some sort of 'rule' for establishing the position of the LowClip Sigma slider.

How is anyone supposed to know whether it is too high, or too low? Apart from just picking a number, at random, and then trying to rectify any errors that are then introduced by relying on subsequent processing.

That just goes against the grain. The whole point about PixInsight is a 'scientific' approach to data manipulation. If we are just going to 'chuck a number at it' then we might as well use PS - after all, the airbrush works wonders (I even came across an article where the recommended approach to eliminating background noise was to determine the 'average colour' for the background, select and delete the original background, and then replace it with a blended in layer of synthesised colour sprinkled with some Gausian noise - what the hell is THAT all about?).

So, I am relying on "minds immeasurably superior to mine" to come up with some sort of rational for setting the LowClip Sigma slider - so that Winsorized Clipping can apply its FULL power to my data

In the meantime, I just have to figure out how I am supposed to open my Calibration images without PI corrupting them beyond all hope of salvation

Cheers,

[Juan Conejero]

In the meantime, I just have to figure out how I am supposed to open my Calibration images without PI corrupting them beyond all hope of salvation

I have good news: I am adding built-in support for the FITS, TIFF and JPEG formats in PJSR (by mirroring the corresponding PCL classes). In this way you'll be able to manipulate FITS files directly from JavaScript code, circumventing the standard FITS support module, and having access to the actual data stored in the files (without rescaling to [0,1]). This means that writing a little script to convert your odd floating point FITS images into regular 16-bit integer FITS images will be extremely easy (right now this can only be done with PCL in C++). Once converted, you'll be able to open your images without problems in PI.

This will be available in the next version, which I'm working on now. Patience...  8)

[Niall Saunders]

Thanks Juan,

I appreciate the effort - but I still don't understand why I cannot enter '65535' in the FITS Explorer window - as discussed in the Bug Reports section.

Surely this is a 'bug'?

I even tried to do this via a TeamViewer 'remote control' session today - with exactly the same results.

There is nothing 'odd' about my FITS data format - it passes FitsViewer inspection with no problems - and I have never failed to be able to open the images in any other software. I just happen to be using 32-bit floating point data storage - and PI happily opens that data format ALL THE TIME.

The problem ONLY exists when PI rescales to [0,1] - and even then it is ONLY a problem if the image does not have a MAX ADU value, and yet the data must be scaled to fit the [0,1] range as if it DID have such a MAX ADU value (of 65535, in this case). In other words, if the ACTUAL maximum ADU value in an image being opened was, say, 32767.00000000 - then it should be rescaled to 0.500000, not (as is the current case) to 1.00000

Honestly, right now, this bug is not just a 'gripe' for me - the inability to be able to rescale correctly is a PixInsight 'showstopper' for me.

I am trying to analyse my calibration images in PI - and NONE of these will EVER have a maximum ADU value anywhere near 65535, and so they will ALL be incorrectly scaled if I cannot set the "Default Floating Point Input Range" parameters to 0.000 and 65535.000.

Perhaps tomorrow I will try installing PI on a second PC - just to see if it is 'this' (Vista 64-bit) installation that doesn't work.

Cheers,

[NKV]

I am trying to analyse my calibration images in PI - and NONE of these will EVER have a maximum ADU value anywhere near 65535, and so they will ALL be incorrectly scaled if I cannot set the "Default Floating Point Input Range" parameters to 0.000 and 65535.000.

Perhaps tomorrow I will try installing PI on a second PC - just to see if it is 'this' (Vista 64-bit) installation that doesn't work.

If the installation succeeds, try to see on your screen the fractional part. For example, try to see a pixel with a value of 32768.5
I see instead 32768.5 only 32768 or 0.500015258789063

[Juan Conejero]

I still don't understand why I cannot enter '65535' in the FITS Explorer window - as discussed in the Bug Reports section.
Surely this is a 'bug'?
I even tried to do this via a TeamViewer 'remote control' session today - with exactly the same results.

No other user has reported a similar problem, and I have no way to reproduce it. If this is a bug, it is certainly an obscure, hardware-related bug. Let's see if the new version, which uses a new version of the Qt library (4.6.1), solves this issue.

In the meanwhile, if you have a wired keyboard at hand (from other computer for example), you could try replacing your current wireless one.

There is nothing 'odd' about my FITS data format - it passes FitsViewer inspection with no problems - and I have never failed to be able to open the images in any other software. I just happen to be using 32-bit floating point data storage - and PI happily opens that data format ALL THE TIME.

I've said it's odd, not incorrect as per the FITS standard

The oddity is in the fact that your camera control software is storing raw CCD data as floating point numbers. This is a conceptual error because raw CCD observations are discrete by nature, and hence they must be stored as integer numbers, which is what they are. Not to mention the fact that 16-bit integer data are being stored as 32-bit numbers, which wastes a 50% of storage space unnecessarily and tends to provide a false impression of increased accuracy.

Floating point pixel storage must only be used when the data are being represented as real or complex numbers for a plausible reason. This only happens when the data don't correspond to direct, physical, raw observations. Real or complex floating point representations are often plausible when the data are being post-processed.

The fact that a given file is correct as per applicable standards does not imply that it is valid from a computational point of view.

All of the above is just my opinion, which is, as every opinions, open to discussion.

The problem ONLY exists when PI rescales to [0,1] - and even then it is ONLY a problem if the image does not have a MAX ADU value, and yet the data must be scaled to fit the [0,1] range as if it DID have such a MAX ADU value (of 65535, in this case). In other words, if the ACTUAL maximum ADU value in an image being opened was, say, 32767.00000000 - then it should be rescaled to 0.500000, not (as is the current case) to 1.00000

Agreed. But as you know, this happens because you cannot specify a custom FITS input range for floating point data, due to a problem with your keyboard, as we've discussed above.

Honestly, right now, this bug is not just a 'gripe' for me - the inability to be able to rescale correctly is a PixInsight 'showstopper' for me.

Okay. Let's solve the problem in a truly dirty way (which, as all dirty solutions, works. Only who has programmed in assembler knows the true meaning of this sentence, so I'm sure you know what I mean here).

Do the following:

1. Define any FITS floating point input range using the FITS Format Preferences dialog. That is, define that incredibly odd range that you get with your keyboard. The actual values don't matter; we only want to be sure that the corresponding configuration data are being generated.

2. Exit PixInsight.

3. Open Windows Explorer and locate PixInsight's configuration file. It is the following file on Windows Vista:

%APPDATA%\Pleiades\PixInsight.ini

where %APPDATA% is usually:

C:\Users\<user-name>\AppData\Roaming

on Windows Vista and Windows 7.

4. The configuration file is a plain text file. Open it with a good code editor (PixInsight's Script Editor can be used, but then you'll need to make a temporary copy of the file, as PixInsight is using it).

5. Search for the following two lines of text (they should be together in the file, and in this order):

ModuleData\FITS\FITSLowerRange=<whatever1>
ModuleData\FITS\FITSUpperRange=<whatever2>

where <whatever1> and <whatever2> are the strange values that you entered on the FITS Format Preferences dialog. Change these values so that the two lines are exactly as follows:

ModuleData\FITS\FITSLowerRange=0
ModuleData\FITS\FITSUpperRange=65535

6. Save the file.

7. Launch PixInsight. Now your FITS files will load OK.


let me know how it goes.

[Juan Conejero]

Hi Nikolay,

If the installation succeeds, try to see on your screen the fractional part. For example, try to see a pixel with a value of 32768.5
I see instead 32768.5 only 32768 or 0.500015258789063

In PixInsight, floating point real-valued pixels are internally represented in the [0,1] range, where 0 means "no signal" (usually black) and 1 means "full signal" (usually white). There are several reasons, ranging from performance reasons to elegance and simplicity reasons.

The [0,1] range is what we call normalized real range. Strict normalization of real and complex pixels is a strong design principle in PixInsight that pervades it at all levels; it must not be seen as a problem or a deficiency, but as an identity sign and a powerful feature. Working with normalized real pixel readouts has several important advantages:

- Normalization leads to an abstract representation of pixels. Real and complex pixel data are treated in purely mathematical terms, not tied to the particular range of any physical device (as the 16-bit range of a CCD camera). Any image processing algorithm can be described and implemented in a much more natural and accurate way thanks to this abstract representation of the data.

- The [0,1] range is much easier to understand and manipulate than other ranges associated to physical devices. A normalized value of 0.5 is clearly in the middle of the available dynamic range in any context. A value of 32767 is in the middle of the numerical range in the context of a 16-bit CCD camera, but not at all, for example, if we are working with a 32-bit integer image.

- Normalization allows us to work independently of the actual numerical range of the data. In PixInsight, you get normalized readouts for 8-bit, 16-bit, 32-bit integer pixels, as well as for 32-bit and 64-bit real and complex pixels. For example, a PixelMath expression and a JavaScript script work exactly in the same way, irrespective of their target image's data type.

- Normalization provides us with a unique reference framework to express pixel sample values beyond images. For example, in the HistogramTransformation interface you work in the [0,1] range to define histogram clipping points and midtones balance values. This is an abstract representation that works equally for all data types supported (and for all data types that will be supported in the future).

So you can't get 32767.5 as a pixel readout because real pixel readouts are normalized to [0,1] in PixInsight. You can get integer pixel readouts in the 16-bit range, where 32767 does make sense, but not 32767.5 because it is not an integer.

[Niall Saunders]

Let's solve the problem in a truly dirty way

Now THAT is what I wanted to hear - if I can get this to work, I think that my problem should disappear.

However, this means that I will have to press the PixInsight 'Exit' button - and I haven't had to do that since early January

I might even have difficulty FINDING the button

And, I am going to lose huge numbers of WorkSpace Layouts (seven, if I remember correctly - each with many images and many ProcessIcons). I will have to take a decision now as to whether I will need beer, whisky, wine or coffee to keep me calm whilst PI is not running. In fact, my wife actually believes that I have changed operating systems from Windows Vista to PixInsight, as that is all she ever sees on my monitor 

I will try this at lunchtime, remotely - using TeamViewer.

Thanks again for your patience and understanding Juan.

Cheers,

[Juan Conejero]

However, this means that I will have to press the PixInsight 'Exit' button - and I haven't had to do that since early January

Quite a stability test, indeed.  8)

You actually don't need to exit PixInsight Open a second instance of the PixInsight Core application and define your FITS input range. Exit the second instance and follow steps 3 to 6, but make the changes only for the second PixInsight instance.

All PixInsight instances share a unique PixInsight.ini configuration file. Within PixInsight.ini, each instance has its own block of configuration settings, starting with [000], [001], [002], and so on. So you just have to search for "[001]" to locate the settings block corresponding to the second instance, then proceed as explained. As you are not modifying any settings for the first (and running) instance, there will be no problems when you save the modified PixInsight.ini. Your second instance of PixInsight will load your FITS files correctly, then you can save them as 16-bit integer FITS files and open them in the first instance. Nice, isn't it?