Patterns in Pixel Rejection during Image Integration ?

[sdh]

Hi, has anyone ever seen odd patterns in the rejection_high and rejection_low images when doing Image Integration? Am I doing something wrong?

Here's some details:

156 subexposures of M31 with a 200mm lens at f2.8, on 6.8 micron pixels (significantly undersampled).
These subexposures were taken thru LRGB and neodymium filters on several nights between 2010 and 2011.

When I do an Auto Stretch with STF on my rejection_low image, the lower left diagonal half of the image has a diagonal cross-hatch pattern of bright areas, about 200 pixels wide. A kind of waffle pattern.

On my rejection_high image, I get a matching pattern in the upper-right diagonal half of the image.

While this patterning does not extend into M31, I wonder, do I have something wrong in my pixel rejection parameters?

Thanks,
Steve

[pfile]

were the subs calibrated and by what method?

[sdh]

Sure, darks and flats, using MaxIM. Hmm, 16bit calibration ... urrr ... But then again, a more interesting possibility, a vanishingly faint molding pattern, or perhaps a strain pattern, in my diffuser. 

Let me say I began with Winsor sigma defaults, then went to Winsor with 4 std devs high/low. Then I noticed the patterns and first posted.

Since then, I have tried a few different variations in my rejection criteria. The patterning decreases a lot when I go from 4 std devs high/low to 5 std devs high/low. 5 std devs, with Winsor sigma, is per Harry's tutorial 

The pattern also decreases a lot when I go to Linear Fit at 4/4. When I go to Linear Fit with defaults, i.e. 5/2.5, the patterning essentially disappears. I also read Juan's past comments suggesting Linear Fit instead of Winsor sigma for large data sets. My data set seems to agree 

When I get more time, I would like to learn what the "slope file" is when using Linear Fit rejection, but for now, I feel confident I have a way forward. And I any additional comments/questions are welcome.

Cheers,
Steve

[pfile]

i should have also asked, what kind of camera is it?

it could be banding noise that's being rejected, especially if the subs were dithered. or, it could be amp glow left over from imperfect calibration, since you say it is in the lower corner of the chip. you say you used a lens, but that of course does not preclude the use of a dedicated astro camera. if it is a dslr, some of the older canons do exhibit amp glow. perhaps it's a combination - i'm not sure you'd see amp glow in the rejection_low map since it's usually kind of bright, and the banding noise would probably show up in the rejection_low map.

i guess it's not surprising that as you increase the sigma values the pattern disappears from the rejection maps. as you make the bell curve wider those pixels are no longer rejected but are included in the average pixel data for that location.

i believe the slope image is meant to represent the slope of the line that was fit for each pixel. that is, the intensity of the pixel at a given location is related to the slope of the best-fit line for that pixel. however, i don't know how to make intuitive sense of what is shown in the slope image.

anyway, it could be the case that you actually do want to reject these pixels instead of including them in the stack. it really depends on whether or not those pixels represent unwanted signal or wanted signal. given the waffle patterns, it sounds like the signal does not originate in space but rather in your camera...

[georg.viehoever]

Steve,
how did you calbrate your images?
Georg

[sdh]

Just to clarify, the camera is a cooled monochrome, not a DSLR. And there is no deliberate dithering. However, each night's image set has a slightly different rotation, and download time for each image impedes guiding and causes a slight shift.

I do flat-fielding at twilight using an oversized flat plastic diffuser. Each flat image is 30s long, and I use a sigma-median of 15 flats computed in MaxIM.

I have used this camera and flat-technique for several years on other OTAs, but 7+ hours of data at f2.8 is new for me.

One thing that occurs to me is the lens hood. This is a stock Canon 4-lobed hood, with flat-black paper card around the front edge to block light from the gaps between lobes. The combined edge looks pretty square, but the diffuser just sits on top. On other OTAs, my diffusers have a foam strip to make a tight seal.

Something I find puzzling is the orientation of this pattern. In the image, M31 is fairly well centered and lies close to one of the diagonal axis of the image. The pattern is extracted from above and below M31, but not from the sides, i.e. this pattern is oriented on the same diagonal axis as M31.

If the pattern came from the lens hood, I would expect either 2-fold lines of symmetry, or no line of symmetry.

Comments? Questions?

[pfile]

well i guess you can recalibrate without the flats and see if the pattern persists. or maybe take a look at your flats by stretching them and see if you can see the pattern there...?

[Lex]

Hi,

Have you got street lights on your observation sight?
This seems like there could be some kind of light pollution invasion to your optics or filterwheel?
Lightpollution noise (Moon, Streelights) changes with different RA/DEC positions and will not be taken into account by calibration measures; on the other side the negative artifact will perhaps not be directly be visible until visualisation of the low/high rejection frames...

If the main actor/object of the image isn't affected there is no need to worry, or do i misunderstand something? 

Regards


Lex

[zvrastil]

Hi Steve,

maybe, if you attach a screenshot of the rejection maps, we'll be able to make better guesses. Now, it's kind of crystal ball fortune-telling.
You can attach images up to 512kB to your posts under Additional Options link on forum post page.

best regards, Zbynek

[sdh]

Here's what the patterning looks like. Sorry about the quality, these are scaled 0.5, then saved at 50% quality.

One thing I noticed in making these, this patterning is more obvious with STM, than with the STM instance dragged to Histogram Transformation; of course the latter is what is attached.

I tried a couple of the suggestions people have offered. First I tried integrating my flats (in PI, not MaxIM) with various rejection parameters. This gave some interesting results, but nothing similar to the attached. Next I aligned and integrated the full set of *uncalibrated* M31 images, again with various rejection parameters. Again, no pattern.

So right now, my best guess is the pattern is an artifact of calibrating in MaxIM; or to be more fair, calibrating 7+ hours of f2.8 data in 16bits, and then switching to 32bits and PixInsight. Tomorrow I want to run my darks thru PI, and if they look ok, then I'll try calibrating in PI.

[georg.viehoever]

My guess regarding the waffle like patterns would be that it is some stray light entering your telescope/camera. But that's difficult to say. Do you see any of those patterns in the source images?

Georg

[sdh]

After stacking uncalibrated images, and stacking uncalibrated/unaligned images, I found this pattern in the image data. Its very  faint, I had to tweak my monitor, and I can only see it in the farthest corners, but its there.

I haven't accumulated definitive proof (I'd prefer to get a finished image, some day  ), but I believe the pattern comes from this:

a) Faint circular patterns in the 16bit-processed flats, caused by the many optical elements in this lens.
b) Faint vertical bar patterns in the readout noise of my camera.
c) Image rotation necessary to align several data sets.

Combine all 3 factors, and I get a waffle pattern in my background.

I have moved beyond this by calibrating in PI, and in particular, including Bias frames in my calibration. I thought I had this covered in MaxIM calibrations because I had separate darks for my flats. But because my flats are only 30s long, those separate darks may have made things worse, not better.

All this gets me into some new problems with my procedure for selecting flats, but that will have to wait until I get some more free time.

Meanwhile, thanks for the suggestions 

[NKV]

My guess regarding the waffle like patterns would be that it is some stray light entering your telescope/camera.

Georg, you don't right.
The pattern produced by ImageRegistration during interpolation... I don't remember which one interpolation method to it, but I am sure.

Steve, try other interpolation method in ImageRegistration.

[Philippe B.]

Here's what the patterning looks like. Sorry about the quality, these are scaled 0.5, then saved at 50% quality.

Hi
I already saw this pattern on some of my rejection images. Mostly with narrowband images.
But I don't care as the rejection process does the job, even on higher rejection high & low values to preserve maximum of signal.
What is sure is the final image has no pattern on it.
Don't worry 

[georg.viehoever]

Hmm, if this is really an artefact of the interpolation in ImageRegistration, I think Juan should have a look. I dont think that interpolation artefacts should so strong that they show in the resulting image or the rejection image.

Georg