Streaky images with registration

[astropixel]

Starting a new thread for this one.

The issue with streaky images seems to be related to registration. I've tried various interpolations and only managed to change the direction of the streaks from oblique to vertical.

If I align the images with dynamic alignment I get no streaks - it took and hour, but I was determined to follow the hunch (DA doesn't increase SNR from what I can see. So its not a solution).

I've pretty much exhausted the registration interpolation combinations without removing the streaks. Dithering is probably the answer. But the difference between dynamic alignment and staralignment is striking.

[georg.viehoever]

Look similar to what I sometimes get with my Canon EOS40D, especially during the hot summer months. Are you using a Canon? If so: rotate the images to make the streaks horizontal, then use the CanonBandingReduction script.

Georg

[astropixel]

Hi Georg - no change unfortunately. I would like to eliminate it at registration if I can.

[zvrastil]

Hi,

according to my experience, this pattern is result of incompletely removed dark and/or bias frame. After registration, these residuals create streaks in the direction of the drift between exposures. The more dark frames match actual time and temperature, the less prominent this effect is. Also, high temperature (and thus high dark current) makes it more prominent.
Ultimate solution to this problem is "exposure dithering" - small random shift between exposures, added by acquisition software. This random shift keeps the noise level, but removes patterns easily detected by human brain. However, dithering is hard to achieve if your autoguider and exposure control device are independent.
I believe that is the reason why you do not see streaks when using DynamicAlignment - as manual procedure, it is does not have high sub-pixel precision of StarAlignment and you unintentionally introduced this random shifts with small errors in the alignment.

It is hard to remove it once introduced. I would try to get better set of dark frames, especially if you used dark frames with very different time or exposure to make the effect less prominent. Or use DynamicAlignment as you described. It may also be possible to remove the pattern from final image using Fourier Transformation, as it is highly periodical and it would exhibit peaks in power spectrum (but this is just quick idea, I never tried it).

regards, Zbynek

[astropixel]

Hi Zbynek.

Thank you. I have been relying on the dark scaling in PI and not producing sufficient dark frames, I suspect. As I sat pondering this issue last night I came back to calibration as a root cause. My experiment with DA confused the issue a little. Not being a technician the complexity of PI can be a little daunting, so I really appreciate the explanation. It makes complete sense. Others have mentioned that dithering is a solution, but the other part of the problem is adequate calibration frames, particularly for this sensor it seems?

Took another set last night, and its cold again tonight, so I'll cold soak the camera same as last night and takes a lot more darks.

Thanks again,

Rowland

[Alejandro Tombolini]

Hola, coincido con Zbynek en que es el resultado de una incompleta remosión de hotpixeles y un mejor MasterDark puede ser de ayuda.

No se exactamente la causa, (posiblemente temperatura), pero siempre tengo algún grupo de imágenes que en algún canal quedan con hotpixeles y que al registrar se transforman en líneas en el sentido de la deriva.

Te recomiendo usar el fantástico Animation (en un crop si es necesario) para detectar aquellas imágenes que permanecen con hotpixeles luego de la calibración.
Si te sucede como a mí podrás encontrar que hay imágenes donde los hotpixeles quedaron solo en algún canal o en todos. La solución es calibrarla manualmente restandole a la foto el MasterDark (BiasSusbtracted) multiplicado por el dark scaling factor (k) que calculó Pix para esa foto o podrías probar manualmente para hallar el valor que hace que queden bien.

Te dejo un link al respecto: http://proxima-sur-fotos.blogspot.com/2011/04/ngc-3324-y-ngc-3293-detalles-de.html

BTW, Sería fantástico si Animation tuviera un botón para visualizar sólo de a un canal por vez en las imágenes RGB.


Saludos.
Alejandro. 


Hi, I agree with Zbynek that it is the result of not completely removed hotpixeles and a better MasterDark could help.

I do not know exactly the cause, (possibly temperature), but I always have some group of images that in some channels stay with hotpixeles and after registration become in lines.

I recommend you to use the fantastic Animation (in a crop if it is necessary) to detect those images that remain with hotpixeles after calibration.
If it happens as to me, you will be able to find that there are images where the hotpixeles remain maybe in a channel or in all of them.

The solution is to calibrate it manually substracting from light the MasterDark (BiasSusbtracted) multiplied by the dark sacling factors (k) that Pix calculate for that image or you might try manually to find the value that fit better.

Here is a link about that: http://proxima-sur-fotos.blogspot.com/2011/04/ngc-3324-and-ngc-3293-details-of.html

BTW, It would be fantastic if Animation had a button to visualize separete channels in RGB images.

[astropixel]

Thanks Alejandro. Taking more darks as we speak. Fortunately winter night time temperatures are quite stable at this time of the year - little variation. Acquired another 2 hours of lights last night (dithered), so about 5.5 hours in total.

[edd]

Apologies for the thread necromancy, but I think it's acceptable in this case. I've been plagued by similar streaking, and while severely clipping high pixels in integration helps, it has left a noisier image from eliminating much of the signal along the way.

I've fudged some acceptable results in the process outlined below - I say fudged as I'm aware it is not really an ideal or appropriate calibration process but it improves the results a lot, so I'm going to keep using it and others might like to try it too. The best situation would be to get dark calibration working better of course.

I suspect it is happening in my case as the dark frames rarely remove all the dark current acceptably. I'm suspicious this might be because my Nikon's NEF raw format isn't quite as linear as one might hope so dark scaling doesn't always work well, and temperature control isn't good enough to just stick with appropriate length darks and apply those with or without scaling.

Here's an example of a calibrated (bias, dark, flat), aligned and stacked, ABE'd and basic histogram-stretched image.


The process I've used to correct this is not dissimilar to reconstructing flats from large numbers of observation frames. I've taken calibrated but unaligned and undebayered images and stacked them, no weighting, additive, percentile-clipped to ~0.1 top and bottom to generate an image that has little to no sources in, but sky background and the residual dark current that is causing the streaking. I've rerun image calibration on the already calibrated frames used in the stack, using the result as a dark (with scaling enabled) but using no flat or bias, added a pedestal in the output (as subtracting dark and sky background like this will leave very little left, possibly negative values in the areas where there's no significant source) and then debayered, aligned, and stacked the result.



The downside is that you can see oversubtraction around bright sources - visible as mild banding, but that might be fixed by more carefully making the correction frame. Overall the result is a lot better I think.

[astropixel]

This is an old one. See Zbynek's comments.
 
There are processing solutions to this problem, however, the easiest is a hardware approach during image acquisition. Dithering eliminates and softens a number of DSLR sensor influences. Perhaps we need a separate DSLR dithering tutorial?

Berry and Burnell suggest dithering DSLR images by at least 12 pixels. Personal experience suggests more is OK.

As can be seen from your image, processing can be destructive. Calibrating DSLR images does not always produce exact results. Basically, dithering overlaps images such that artifacts get covered up, overlayed by healthy pixels.

From what I have gleaned about dithering, DSLRs require a coarse dither because consumer CMOS sensors are usually active devices with unique noise characteristics, compared to their passive CCD cousins. Dithering by half a pixel or 2 or 3 pixels may not be effective.

Dithered results
http://www.astrobin.com/full/9821/0/
http://www.astrobin.com/full/88324/B/

[edd]

These are unguided, so overly high pixels don't stay in one place anyway. To deal with this by randomly spotting those pixels around over potentially hundreds of exposures on a system that is hand controlled is an extreme investment of effort. It might be a better idea in principle, but in practice it just isn't possible. This was in a light polluted area imaging at F/2 so exposure times were necessarily short, and dithering by hand every 15 seconds over hours would not only be laborious but would deeply eat into exposure time as well. The sort of dithering you suggest is therefore just not a convenient option without significant investment in hardware that we can't all afford.

This is a relatively low cost solution to the problem even if as I say, it isn't ideal.

Better methods are better, but this is low cost and works on unguided imaging.

[astropixel]

edd. My system is also unguided and manual, under light polluted skies. My set up is DIY, in many respects, modified with low cost solutions to address the issues under discussion here. I am conscious of the cost factors.

Dithering with a hand controller is laborious, and now that I understand the context, yes, impractical. Automating the process would be an advantage, if possible.

I pressed a low cost Arduino microprocessor into service, to interface the camera and hand controller to sequence image capture and dithering. It is very convenient and I am happy to share the details, if it may be of assistance.

Are you using an AltAz mount - 15 second exposures?