Cross hatch pattern after alignment

[SeaLandSkyPhoto]

Hi,

I sometimes get a cross hatch pattern after using StarAlignment.  It doesn't happen on all images and it seems to be more prominent with larger values for dx and dy.  rotation values seem to have no relation to the issue (all my images have < 0.1 for rotation values - usually around 0.01 or 0.02).  It seems to mostly occur with images with low SNR backgrounds.

I've uploaded some files that illustrate this to https://drive.google.com/drive/folders/0BzcGWf3G2o6gcHpWREtGeGJyWTQ.  I've included two samples - the Oiii data is more subtle while the Sii data is more prominent.  I've included the calibrated frames before alignment (-Calibrated.xisf), the aligned frames (-Aligned.xisf), the frame used as the alignment reference (Alignment.xisf), the process icon used for alignment (alignmentProcess.xpsm), and jpg screenshots of the frames (alignment-).  I applied an auto histogram to each image in blink to get the screenshots.  I also have available the process console output from the alignment.

These were done with Lanczos-5 with clamping at 0.15.  I get pretty much the same results with the interpolation set to Auto and using Lanczos-5 with clamping at 0.3.  The cross hatch seem to be imperceptible (if not non-existent) using Mitchell-Netravali Filter.

Any thoughts on what might be causing it and ways to avoid it (other than just using the Mitchell-Netravali Filter)?

Thanx,

Paul Howard

[pfile]

i have seen this kind of problem in narrowband images where there is essentially no background signal. in your calibrated OIII image the background reads out at 1-3 ADU which is vanishingly small. are you sure the darks match? what does the background look like in an uncalibrated image?

rob

[SeaLandSkyPhoto]

Hi Rob,

Thanx for the response.  I believe my dark properly matches the raw frame.  The dark is an integration of 50 dark frames taken at -30c with a readout mode of 1MHz Flood (RBI mitigation).  The Oiii frame was taken at -30c with a readout mode of 1MHz Flood (RBI mitigation).  The dark frames were taken about 30 days after the Oiii image (I take darks every 90 days and they are mostly consistent between sets).

The image was taken from extremely dark skies with excellent seeing.  Thus the extremely low background level with narrowband images from this location is not all that unusual.

I created a preview in the background of the Oiii image of about 100,000 pixels.  Then dragged the preview over to the dark frame.   The statistics of both are:

 
                  Dark                    Oiii
mean            2085.8                  2088.9
median          2083.6                  2087.0
stdDev           177.4                   182.2
avgDev             5.6                    15.5
MAD                2.7                    13.3
minimum         2069.3                  2036.0
maximum        45613.0                 47813.0

What do you think?

Paul

[pfile]

yes - i should have added that the context in which i saw this problem was with "short" narrowband exposures under extremely dark skies, also using RBI preflash (with a 16803).

i think the problem is that the lack of sky signal means that the noise in the dark (and the light) is a significant percentage of the background signal. as a result there are many pixels where the value in the dark exceeds the value in the light, and so in the calibrated frames there are many, many pixels that have been clamped to 0 (since negative results are not allowed during dark subtraction). these 0 value pixels cause issues with certain interpolation algorithms as you've found. the clamping may only help with small-scale bright ringing such as around stars, but does not help in this situation.

obviously one solution is to expose longer; we were using 1800s exposures for narrowband. that might get the background signal up to the point where most of the background pixels in the light exceed the dark.

another solution would be to add an output pedestal during calibration. this should prevent the 0 values, which i believe are what is contributing to the cross hatch pattern. although in discussing this problem with another user some time back, he stated that this threw off the flat scaling and caused overcorrection of the lights.

rob

[avastro]

Hi, I've seen also this kind of problem with narrowband as with RGB, mainly the blue channel. Using  interpolation Lanczos-5 with different Clamping threshold doesn't help, I solved the problem using  in Pixel interpolation BiCubic B-Spline with very good results.
Antoine

[SeaLandSkyPhoto]

Thanx for the responses.  This is also a 16803 using RBI mitigation.  I've been using 15m exposures so I get better results during the integration (more images in the stack for a given integration time).  I did play some with 30m exposures, but I still ran into this problem occasionally, but not as often....it's very dark out there ;-).

I agree that adding a small pedestal value during calibration would mess up the flatting...unless the pedestal could be added AFTER flatting.

I'm wondering if there would be a way to correct this using CosmeticCorrection after calibration and before alignment?  Perhaps by configuring it to treat the 0 pixels as cold pixels?  Not sure if this is doable or if it is how to do it.   Or maybe a PixelMath expression applied after calibration?

I haven't tried BiCubic B-Spline yet.  I stopped looking for a workaround when I got good results with the Mitchell-Netravali Filter.  I'll give the BiCubic B-Spline a try.

Any thoughts on a way to automate the detection of problem frames so they can be processed aligned differently?

Paul

[pfile]

dunno about automatically detecting bad frames but i suppose you could replace any zero valued pixels with a small constant in an image with iif($T==0,0.01,$T). but your calibrated image posted has something like 38% of the total pixels clamped at 0, so the images would look pretty weird after the fix. however, i assume that DBE would be able to remove this offset in the integrated images, or you could just subtract it from all pixels afterwards.

rob

[SeaLandSkyPhoto]

Thanx for the response.  I don't understand where you saw 38% of the pixels clamped at 0.  I looked at the stats for the Oiii calibrated image and saw a minimum value of 3.5547e-09. 

Anyway, I played with a simple PixelMath expression to add a slight pedestal to all pixels - $T+1e-08.  Then I aligned the resulting image and inspected it.  Using the Oiii image, I used successive values of 1e-08 thru 1e-04 and saw incremental improvements with each with the final image using 1e-04 having almost no visible cross hatching.

So then I went to the Sii image (which has a much more visible cross hatch - bigger dx, dy).  Adding 1e-04 to this image improved it, but did not eliminate it.  So I tried 1e-03 and that eliminated the visible cross hatching.

However, when blinking the Sii+1e-04 aligned frame versus the Sii+1e-03 aligned frame, I noticed that not only did the visible cross hatching disappear, but the star alignment shifted.  I've uploaded these two files as sii_p1e-04.xisf and sii_p1e-03.xisf to the same folder.  Look at the upper right corner while blinking.   I then went back to the Alignment frame (Alignment.xisf) and blinked it with these two frames.  It seems that when there is visible cross hatching (sii_p1e-04), the star alignment is also slightly in error.   The sii_p1e-03 image appears to be correctly aligned (as well as having the visible cross hatch eliminated).

So it would seem to be not just the visible cross hatching that is at issue (which can be eliminated in the integration with a sufficient number of images in the stack), but mis-alignment as well (thus valid signal is getting rejected during the integration due to the mis-alignment).

Paul

[pfile]

open your initial calibrated image and use the pixelmath expression iif($T==0,1,0) to get a new image with all the 0 pixels set to 1 and all else set to 0. then look at the statistics of that image; the mean is 0.38 which must mean that 38% of the original pixels were 0.

when running SA are you using distortion correction?

rob

[SeaLandSkyPhoto]

So I don't understand how that correlates with the stat process showing a non-zero minimum?

No, I'm, not using distortion correction in SA.

Paul

[pfile]

i think the stats process excludes 0 and 1 pixels. there might be a checkbox to turn that off.

rob

[SeaLandSkyPhoto]

Yep, I found it - the "unclipped" check box which excludes 0 and 1.  I also found the disconnect between our numbers - you were looking at stats for the Oiii frame (38.7% zero) while I was looking at the stats for the Sii frame (4.8% zero).

I looked again at calibration - the Output Pedestal field.  The mouseover popup seems to indicate that this is applied after calibration (so post flatting).   So I ran a series of re-calibrations of the Sii frame with pedestals from 10 to 100 in increments of 10.  A pedestal of around 50 seemed to do well with eliminating the visible signs of the cross hatching.  There was also no apparent impact on the flatting of the image so the pedestal does appear to be added after flatting the image.  Curiously the images with pedestals of 10 and 80 showed mis-alignment errors - the 10 with visible signs of cross hatch and the 80 without any visible signs.  This would seem to indicate that the presence of the cross hatch and the mis-alignment are not linked.

I looked at all of the alignment interpolation algorithms using the Sii frame (without adding a pedestal in calibration) with all values except the interpolation set to defaults (did a reset of the process).  Some of them showed a visible cross hatch and some did not.  The interesting things is NONE of them showed a mis-alignment.  So it would seem that something in the other star alignment parameters that I had changed was resulting in the mis-alignment issue when aligning a frame with a clipped background.  The lanczos-3, lanczos-4, and lanczos-5 interpolations where showing the most visible cross hatching, but much more subtle than using my normal alignment process (lanczos-5 with as yet to be researched parameter changes).  The others showed no visible cross hatching.  Interestingly the BiCubic B-Spline and Cubic B-Spline showed increased contrast over the other interpolations.

So it would seem, at this point, that adding a pedestal of 50 in the output from calibration addresses the cross hatch (at least for the Sii frame).  Without the pedestal, 4.8% of the pixels are clipped.  Adding the pedestal of 50 leaves only 0.0045% of the pixels clipped.  At a pedestal of 40 (with slight cross hatch showing), there were 0.0064% clipped.  Sounds like a target of no more than say 0.005% clipped would keep the cross hatch mostly under control.  I'll need to establish what it takes with the Oiii frame to control the cross hatch since this frame has a weaker background than the Sii frame.

Since there are still clipped pixels after adding the pedestal, it would seem that the calibration process applies the pedestal and then clips the remaining negative values to 0. 

This just leaves, I believe, figuring out the cause of the mis-alignment.  I need to examine my alignment process settings versus the defaults and run some tests to see which changes are resulting in mis-alignment with a weak background.

Paul

[SeaLandSkyPhoto]

I found the cause of the mis-alignment.  Somehow or another, the StarAlignment process icon that I was using had this value

P.matcherTolerance = 0.0030;

which I didn't see anywhere in the gui.  The default StarAlignment process has a value of 0.05 for this parameter.

And for the Oiii frame, a pedestal of 60 seemed to do the trick for removing the visible signs of cross hatching (leaving < 0.001% clipped)

Thanx for all the help on this.

Paul

[pfile]

interesting about the matcherTolerance value. this is different than the RANSAC tolerance setting? 0.05 seems quite different from the default RANSAC tolerance so i guess it is something different.

sorry about not mentioning it was the OIII frame i was looking at. that would have eliminated some confusion. at any rate, when i had this problem it was subtle enough that the crosshatching was only visible in the integration rejection maps. i was not sure about the output pedestal/flattening thing since it was another user's data that i never actually looked at; i had just suggested the output pedestal and he reported that there was some issue with flattening (bright corners) but he was only interested in the center of the frame so it all worked out.

anyway - glad it's resolved.  i never realized that this problem partially tied to the interpolation algorithm.

rob