Winsorized Clipping for Dark Frames

[Farzad_k]

Juan,

After examining my camera with different exposure settings, but all at 300 seconds, I am getting the same low bit rate results. What should that mean?

[ngc1535]

In both cases, the original, and now moodified, images are re-evaluated - repeating the calculation of mean and SD, and the production of a further set of data. The process iterates until no further improvement is obtained for the mean value, thus giving the final image.

However, crucial to the success of these clipping methods is that the original data set (hence images) contains data with a distribution curve that allows the analysis to take place (i.e. there has to be 'some' SNR to work with).

If the data has very poor SNR, the clipping processes have little or nothing to work with. This is the case with Biases and Darks - in these cases, a simple arithmetic mean is all that is required when creating Master frames.

Hi Niall,

Hmmm... I don't agree with that statement concerning darks in particular. Darks suffer from cosmic rays (as do biases when you take enough of them)- Min/Max clipping with an average of the remainder is a very good way (in my opinion) of dealing with darks and biases in the creation of master frames. Unfortunately there is a small bug concerning Min/Max clipping (I have reported it before, but I have not checked recently)- so I simply allow for a sigma clipping in BPP.  I don't see how any of the statistical clipping methods would be a bad idea in terms of rejection methods for outlier values (or in the case of guassian noise... large "rare" fluctuations).

-adam 

[Juan Conejero]

Hi Adam,

Unfortunately there is a small bug concerning Min/Max clipping

Can you please describe this bug? Sorry but I can't remember if you have reported it before; in such case, I apologize for having overlooked it.

Min/Max clipping with an average of the remainder is a very good way (in my opinion) of dealing with darks and biases in the creation of master frames.

Min/max, indeed, cannot fail to reject cosmics, since it performs an unconditional rejection. However, with min/max you get a constant SNR loss proportional to the square root of the number of clipped pixels. With constant I mean that this happens for every pixel, irrespective of if a particular pixel requires rejection or not. IMO, this is too high of a price to pay. Any statistics based rejection method is preferable to min/max, in general, but especially for rejection of cosmic rays, since they are very easily detected as outliers without uncertainty. I would use min/max just as a counter-test for evaluation of results achieved with other methods.

[Farzad_k]

...I have heard "cosmic ray" in various communities and documentations but I cannot see how dark frames and bias frames can suffer from them considering the scope is covered. Can someone please enlighten...

Farzad

[msmythers]

Farzad

Check this link.
http://www.darkerview.com/CCDProblems/particlehit.php


Mike

[dld]

Cosmic rays are high-energy particles. They leave traces even in my relative short (< 1 min) dark frames, taken inside my house.

[ngc1535]

Juan,

I agree that Min/Max is a bit draconian... but in many cases darks are cheap (easy to make) and statistically significant measurements can be obtained from a large sample- and if large enough... clipping isn't *too* bad. But your point is well taken.

I am sorry Juan... my memory of the error was not good. The error I was thinking of was here:
https://pixinsight.com/forum/index.php?topic=7258.msg48847#msg48847
I haven't checked in a while to see if it still occurs. Min/Max is OK.

...
Concerning the cosmic rays- just consider when you see them in the darks.... they are doing the same thing to your body! (And you have a MUCH larger surface area..)

-adam

[Farzad_k]

...sorry, but I can only ID noise and hot/cold pixels - not cosmic rays. I would love to see a picture of how it looks like.

[Farzad_k]

Juan,

After changing the camera settings from HDR which is associated with a gain of zero to a Unity Gain with a 300 gain, I am able to use WSC to integrate the 300 second darks.

Farzad

[pfile]

...sorry, but I can only ID noise and hot/cold pixels - not cosmic rays. I would love to see a picture of how it looks like.

[pfile]

continued... all these screencaps are from 30 min narrowband subs...

[Farzad_k]

I can imagine cosmic rays when I look at the night sky and in images of the night sky but I can’t tell what part of a dark sub is cosmic ray. All that I can see in dark sub is noise and bad pixels.

[pfile]

at the center of every one of those clippings i posted is the track of a cosmic ray. they are not bad pixels, they are pixels that were excited by a cosmic ray passing through. that's why they look like random lines and squiggles. there's a statistical element here - the larger the detector and the longer the sub, the more likely you are to capture one.

if the cosmic rays have enough energy they can permanently damage the photosite they hit, but at least for me here at sea level i've never had permanent damage.

rob

[Farzad_k]

Rob,

I do understand that. When I asked someone to please show me how the cosmic rays look like in dark frames I was hoping that someone would show me a dark frame ans point out to an element that is not noise, hot pixel, and it is cosmic ray so that I could hopefully learn how to recognize that.

So the reason we believe cosmic rays affect even the dark images is because we know cosmic rays exist and we know they go through everything and we know that a camera is no exception to that. Is this correct?

Thanks

[ngc1535]

Looking at a single frame you could not, in the general sense, identify for certain all of the elements you want to have pointed out. For example a cosmic ray that hits the chip and activates a single pixel- well it will look like a hot pixel.

BUT... if you BLINK many images (or take the difference of images, or see the results of the rejection between images) you will be able to see many different kinds of things. Have you ever blinked your dark frames (you will need to zoom in)?

1. Most pixels of the detector will not change much in brightness. There will be random fluctuations due to readnoise and other sources. They will be small. If this wasn't the case...the detector wouldn't be useful. You will see most things look pretty "constant."

2. This includes hot pixels- on short time scales they will not change much. But blink dark frames taken a year ago with the same detector and you will certainly see changes in hot pixels, perhaps bias level...etc.

3. All of the things that DO change- that is the stuff Rob is trying to indicate. Those transient events will only exist in a single frame. Cosmic rays in particular are very easy to see... just blink your images.

4. If there are other changes with your chip that occur on short time scales (electrical/frequency banding, pattern shifts)- then the detector itself is the issue. Generally this would be a less than desirable thing to see from one dark frame (or bias) to the next. A fast fourier analysis of your calibration data can help identify all kinds of goofy things.

A bias frame is just a short dark frame- so when you look at a bias frame you are looking at an ocean of electron readings that are bobbing up and down about some average sea level. The level is the pedestal bias and the fluctuations from pixel to pixel (if they were all the same) is noise. There isn't a way to show you an element that is "not noise...etc."   A dark frame is the same...but there will be an additional measurable increase of electrons through time due to dark current.

-adam