Local (small scale) Flat Error Rejection (repost)

[ngc1535]

Hi,

I am reposting this question from a while ago. I think it is a good one because of its usefulness. If the answer is "no" then the question is could it be considered as a feature request.

Hi,

After calibrating light frames I occasionally find that some flat errors remain in a particular night's data. A typical example is that flats are generated at the end of the night. At some point in the middle of the night a dust mote or other thing falls on an optical surface in front of the camera. Although the newly generated flats will correct images taken after the dust mote fell- they will not help for images at the beginning of the night.

So as part of the image integration and rejection process is there a method to locally select these areas in the problem images to have them be rejected pixels (in addition to normal statistical rejection)?

Thanks,
-Adam

[aworonow]

Maybe I'm missing something, but could you just cloneStamp the spurious dust motes away and apply that result selectively to the appropriate lights?
AlexW

[pfile]

there is no method to manually identify areas for rejection in an integration.

having said that, the large scale rejection algorithm in the upcoming version of PI may be able to help with this.

rob

[ngc1535]

AlexW: Yes, there is a more nuanced way of dealing with the issue I describe that does not require manufacturing (cloning) data. If you have a significant number of images- it is possible to reject (no include) the values at particular pixels. This means a pixel stack of values can locally vary in number (if you have 10 images perhaps one pixel stack averages only 6 or 7 values because you have rejected the pixels in three frames). This effects the S/N of the computed mean- but with a large enough set of images/values/measurements it isn't a big issue. Also, the method I imagine is possible would actually be easier than cloning things.

Rob: Please pass on the word to the proper person... this would be a really great feature to bring to PI.

[mschuster]

Will this work?

In ImageIntegration, enable "Clip low range" and set "Range low" to 0. (These are defaults).

Then zero out the bad pixels in each of the problem frames. The zeros will be rejected.

There are several ways to zero pixels. Eg box the problem: eg (x, y) of (200, 600) to (300, 700) and apply pixelmath "iif(x() > 200 && x() < 300 && y() > 600 && y() < 700, 0, $target)"

If loss of SNR is a problem, selectively denoise using a mask during post processing.

[ngc1535]

Hi Mcschuster,

A couple of points:

1. The images will generally be dithered. Since rejection occurs after registration- imagine trying to specify x,y coordinates for each of 7-10 frames (or something) where there is an issue like a dust donut, glint, faint satellite trail etc etc. The method of identifying areas for rejection needs to be on an image by image basis using the eye-brain pattern recognition skills. This means something that permits drawing on the image.  Keep in mind- this is NOT manufacturing data- so it is not a drawing tool- but a kind of selection tool.

2. I did attempt your suggestion- and discovered something not well known. I asked Carlos about it in fact. When you make shapes using PixelMath- there is a small interpolation of values at the boundaries of the shapes! This results in a dirty rejection at the edges and it is easily seen. I spent a day, once, doing this for many frames just to see how it would work. So in addition to the above the implementation of the feature will have to select real pixels without interpolation.

[Juan Conejero]

When you make shapes using PixelMath- there is a small interpolation of values at the boundaries of the shapes!

PixelMath does not apply any interpolation if the images involved in the running expressions have consistent geometries. A PixelMath expression can implement some kind of boundary smoothing, such as antialiasing, either explicitly or implicitly (for example, by mixing proportionally pixel values around the limits of the regions being defined as a function of distance). I'd have to see the expressions being executed to judge in such case.

Can you put an example of the small interpolation you are describing?

[Juan Conejero]

As Rob has pointed out, the new large-scale pixel rejection algorithm in version 1.8.5 has been conceived to help solve these issues, among many others.

[ngc1535]

Juan,

Yes, anti-aliasing is the right term. There was smoothing at the boundary. I was of course using circles to try to cover the dust donuts.
I will look back to see if I saved the files- I think I did (because I spent so much time trying to make it work!).

It is great news concerning the improvements to the rejection algorithm.

-adam