Diagonal Streaks?

[dmcclain]

Has anyone ever seen this kind of thing before? Diagonal streaks running across the entire image. Only shows up at 1:1 magnification and higher. I looked at the raw integration and it is in there. But when I examine the individual frames of the stack I cannot detect it visually.

FFT shows very clear difference along the two 45-deg diagonals.

None of the registered subs have FFT's with any hint of this behavior. A few do have elongated regions contrary to the length of the streaks, but all exhibit generally very broad ellipsoidal spectral magnitudes. The result appears to be some kind of unlucky phase cancellation during ImageIntegration.

[dmcclain]

I went back and discarded the images with elongated FFT cores -- appears to be due to poorer tracking during the exposure.
But that did not solve the situation in the final integration. It appears that there is some kind of long-term correlation, row to row in these images, each row marching toward higher X position. They were from an ATIK OSC, and used SuperPixel deBayering.

[SteveP]

Hi - I am experiencing a similar issue!

I've just posted a reply in a separate thread to someone having issues with BPP.  I'm having issues with both BPP and Image Calibration and am wondering if it is something to do with the application of the flats (or maybe my problem in how I create the flats).

The root of my issue is that I am taking subs with a QSI690 (ICX814 Sony) - and at certain wavelengths (H alpha and SII - but not for example OIII or wideband filters) the flats exhibit diagonal streaks.  I have read that this is something like a reflection or interference pattern related to the gold connections on the chip but I've seen no objective proof of this so far.  However it does seem to be a feature on the chip that causes the diagonal streak.

If you can establish that this is what is happening in your case - the next thing to establish is, are the flats being applied properly?  I am suspicious that something might have gone slightly awry in the algorithm here - but perhaps PI developers could help us here.

Steve

[Alejandro Tombolini]

Hi, this processing example may help.

Saludos, Alejandro.

[pfile]

yeah generally speaking this is the result of less-than-optimal calibration and differential flexure (assuming you are autoguiding). if not autoguiding, then it's a result of bad polar alignment. if you blink your frames, you'll see the stars marching along a diagonal line as the movie progresses. thus the hot pixels are doing the same thing, and since they were not removed completely in calibration, they are difficult to reject during integration.

rob

[IanL]

You need to dither between frames, 12 to 15 pixels ideally. Even with good calibration this kind of pattern will appear to some extent if you have condtant drift beteeen frames. Dithering randomises the position of hot/cold/warm pixels and PRNU more generally, and thus any residual artefacts post-calibration will be randomly distributed, more easily rejected and even if not rejected won't form a pattern that your eye readily detects.

[dmcclain]

Interesting comments, seemingly plausible...

This was in fact from a Sony 814 OSC.

I blinked the images in the order acquired and see two parts of the acquisition session. The first part with 5x120 frames, followed by an axis flip, and the second part with 6 x300 frames. I do see a linear progression in the second part, but not so much in the first.

So if I integrate the first 5 frames they seem to produce a clean result. And if I integrate only the second 6 frames they show the streaks.

And so you may be correct about the need for better and wider dither. But if the hot pixels were the cause of this there would have to be a ton of hot pixels. And I do not see that. I have my share of hot pixels but they are distributed quite widely over the image and nowhere near the areal density indicated by these streaks of high density and uniformity.

I did carefully examine the old flats, and found no artifacts that could explain this. But when I blink the second set of longer exposures and magnify the blink screen, I do see what appears to be clumps of warm pixels remaining stationary as the stars drift past, and those clumps do have a relatively high areal density, compared to truly hot pixels.

So this hints to me that perhaps I should remove the 1-pixel wavelet layer from the flattened calibrated images, before registration and integration. My star images all have 2-4 pixels across them, not 1 pixel.

[dmcclain]

Those clouds of warm pixels are just barely visible, and can really only be seen in a blink progression as the stationary part of the pattern. I tried to produce a snippet to show everyone, but there is almost nothing to see in a single frame. They have about the same amplitude as the general background. But the clouds of warm pixels are stationary while the sky and stars move about from dither.

I did the removal of the 1-pixel wavelet layer, then registered and integrated. And that did help a lot, but there is still a trace of the diagonal streaks in the result. Very faint now, and takes much higher magnification to see them.

So it puts me on notice for two things: (1) need to dither more widely and randomly, and (2) be careful about long integration times on the ATIK-490.

I should also state that these images and all others I have done with the ATIK-490 camera, did not incorporate any dark frames. The wisdom of the experts was that such things were unnecessary in a camera with such low dark current. But I now see that as mistaken. While it may be true that the dark current is quite low, there is a subtle accumulation of warmer than average pixels over long exposure times. And I think some dark frames would have gone a long way to ameliorate these artifacts.

(Ahh.. but that's a pickle... the camera is a HyperStar assembly, and it is not possible to cover the camera to make darks, and there is no internal shutter to help with such things. So probably best to avoid long integration times. )

[dmcclain]

I found some old 150s darks and redid the entire calibration anew, using those darks as well as the previously used bias and flat frames. I did both VNG and SuperPixel deBayering.

Despite the protestations of the calibration pipeline about the darks not being correlated with the light frames, the results of both forms of deBayering show marked improvement over the original images created without the benefit of dark frames.

No trace of the streaks in either result.

[pfile]

the underlying cause here is dark scaling. the dark current in a well-behaved pixel scales linearly with exposure time, which is the theoretical basis for why scaling a dark even works. a hot or warm pixel will not obey this rule, and since it gets scaled as well, you end up under-subtracting the hot/warm pixel. you're then left with what you are seeing - kinda warm pixels all over the place.

when you used the mismatched dark, PI could not find a correlation between the dark signal in the light and the dark signal in the dark. PI does this because rather than scaling based on exposure time, it simply scales the dark until the noise is minimized in the calibrated output. the message means that it scaled the dark all over the place and never saw a correlated change in the output frame's noise. if i remember correctly, it simply does not scale the dark in this situation... and so your hot pixels were properly subtracted. so your lights still had residual dark current (assuming the 150s darks are too short) but the hot pixels were gone.

so the solution here is to take darks which exactly match the duration of your lights, and then untick "optimize" in the master dark tab. that should fix both problems.

rob

[pfile]

oh, i just noticed that this thread is a sticky, currently right above this post:

http://pixinsight.com/forum/index.php?topic=8839.0

rob