Ideas on DSLR Noise Removal (D800)?

[jerryyyyy]

I have finally got my D800 cooking with my Tak 180ED and pretty happy with the images.  But as we all know the reason CCDs are cooled is to reduce noise and there in noise in spades in the D800.  I have tried Luminosity masks and standard noise reduction techniques such as TGV and ATM and they work pretty well, but knowing there are many magicians out here, what are other suggestions?

This is a parent image that if you look in the high resolution you will see the issue:



I also uploaded a cropped section of the image (after processing) so you can see the noise in a bit more detail.  I think it is a combination of the camera plus the sky glow from the SF Bay Area and I get this swirly mostly redish stuff. 

[Zocky]

How the noise looks at the single frame? Is the pattern the same?
To me, it looks like there was some wrong steps during calibration and integration process.

[pfile]

that kind of "rain" pattern usually indicates 2 things:

1) there is differential flexure between the guider and the imager. a really long subexposure would show star trails in that direction even though the guider thinks everything is working great
2) the darks don't exactly match the lights and hot pixels were undercorrected.

the foolproof cure to #1 is to use an OAG, but this is really difficult with a DSLR. they do exist, but perhaps only for canon bayonets. still, a super pain in the butt. alternate cures are to stiffen up everything in the imaging and guiding train as much as possible, and make sure neither camera is sagging at all.

#2 is kind of a problem in general with DSLRs. there's no temperature control, so it's hard to get darks that match your lights exactly. these marching pixels are probably hot pixels that were not fully calibrated away.

there are a couple threads here where people were discussing alternate techniques for integrating DSLR darks and how to apply them in ImageCalibration. i can't remember the upshot now, but the solutions were somewhat unorthodox (something like turning on normalization for the darks during dark integration, but don't quote me on that ).

in the short term you might experiment with turning dark optimization on and off, or perhaps using cosmeticCorrection to try to get rid of any residual hot pixels in the calibrated results. or, its possible that by tweaking your ImageIntegration rejection parameters during the light integration that you can get these hot pixels rejected.

rob


rob

[jerryyyyy]

How the noise looks at the single frame? Is the pattern the same?
To me, it looks like there was some wrong steps during calibration and integration process.

Here is a cropped part of the original nef stretched in PI.  Not sure what you think, but I still see the noise.  One thing I might do is to look at the flats and have in the past applied TGV to the flat. 

Will check what Pfile has to say also.  Wai has made some notes on the Nikon darks and bias files.  I had a couple lights in this that I should have rejected due to the guiding. 

[IanL]

The rain pattern is most often a consequence of not dithering adequately (or at all) between frames. For DSLR images a dither of about 15 pixels should work, much less than this is often ineffective.

[jerryyyyy]

The rain pattern is most often a consequence of not dithering adequately (or at all) between frames. For DSLR images a dither of about 15 pixels should work, much less than this is often ineffective.

I think you nailed it here.  Normally I use Maxim for image acquisition attached to my EQ-G and that runs the SBIG with OAG... and dithers away. 

I have had trouble running Maxim with the D800 and so have been using CameraRC to collect the images.  Well this collects the images alright but has not dithering as it is just a general camera application.  I am still running Maxim for the guiding... they are very small changes so the dithering is a requirement. 

Have to think this through. 

[Don]

I agree with IanL, I think dithering, or dithering more, will help with this.  It doesn't look like residual hot pixels to me, but the dreaded "red blotchiness" that often invades DSLR images, complicated by inadequate dithering.  I also think longer exposures will help with this, even if it means dropping the ISO.

That said, I found a method that deals with the red blotchiness in DSLR images pretty well.  The technique was presented by Alejandro Tombolini in this forum, and I found it in this tutorial: http://lightvortexastronomy.blogspot.com/2013/12/tutorial-post-processing-pixinsight.html.

The technique involves creating a mask using the RangeSelection tool to protect the stars and high signal areas, then using AtrousWaveletTransform with the target set to Chrominance to remove detail from layers four through seven.  In the case of your cropped sample, this reduced the red blotchiness but left some of the "rain" pattern as light streaks. 

A subsequent pass of ATWT with target set to RGB/K and noise reduction enabled for layers one through 4 helped with the light streaks.

You can find the details in the above link. 

Cheers,

Don

[pfile]

you have to ask yourself why there is a strong linear component to this noise. at the root of this is DF; every time i've seen this "rain" pattern that's what it's been.

in some sense differential flexure is already a form of dithering. the same rejection principles apply, except that eventually with such extreme "linear" dithering, you start getting 'collisions' between neighboring hot pixels which make some of them harder to reject.

it's easy to prove though, while StarAlignment is registering the images, if you see each sub's X/Y offset incrementing roughly linearly, that's what's happening. or, just blink your unregistered subs and see if there is an X/Y motion in the resultant movie.

rob

[IanL]

you have to ask yourself why there is a strong linear component to this noise. at the root of this is DF; every time i've seen this "rain" pattern that's what it's been.

in some sense differential flexure is already a form of dithering. the same rejection principles apply, except that eventually with such extreme "linear" dithering, you start getting 'collisions' between neighboring hot pixels which make some of them harder to reject.

it's easy to prove though, while StarAlignment is registering the images, if you see each sub's X/Y offset incrementing roughly linearly, that's what's happening. or, just blink your unregistered subs and see if there is an X/Y motion in the resultant movie.

rob

Well yes, if each sub was perfectly guided and there was no shift between them then no linear pattern would appear. It doesn't have to be DF though. I am sure my setup has little to none, but when dithering at about 5 pixels it would appear, but not at 10 to 15 pixels as advised by Berry and Burnell. The streaks wander from side to side a bit more but are still visible at 5 pixels. Sadly the PHD guiding default is  5 pixels so this crops up a lot in forum posts.

Field rotation is another cause, as subs may be short enough to produce good star shapes, but when registering later curved streaks will appear in the background.

[pfile]

fair enough. it may be that because it's an OSC then there's a correlation between pixels (after debayering) that otherwise would not be there, and so a larger dither does help.

rob

[edd]

I've found with my cheaper Nikon that dark frames just don't correlate well at high ISO. Reducing ISO makes the application of dark frames massively more reliable and reduced this problem a lot (but not entirely).

[andreasmax]

i also shoot with the d800 unmodified.
do you have the firmware hack already?
i do not have seen these kind of noise at my stackings ever!
please post some informations about your camera settings: ISO, shutterspeed, time, etc...

[jerryyyyy]

I agree with IanL, I think dithering, or dithering more, will help with this.  It doesn't look like residual hot pixels to me, but the dreaded "red blotchiness" that often invades DSLR images, complicated by inadequate dithering.  I also think longer exposures will help with this, even if it means dropping the ISO.

That said, I found a method that deals with the red blotchiness in DSLR images pretty well.  The technique was presented by Alejandro Tombolini in this forum, and I found it in this tutorial: http://lightvortexastronomy.blogspot.com/2013/12/tutorial-post-processing-pixinsight.html.

The technique involves creating a mask using the RangeSelection tool to protect the stars and high signal areas, then using AtrousWaveletTransform with the target set to Chrominance to remove detail from layers four through seven.  In the case of your cropped sample, this reduced the red blotchiness but left some of the "rain" pattern as light streaks. 

A subsequent pass of ATWT with target set to RGB/K and noise reduction enabled for layers one through 4 helped with the light streaks.

You can find the details in the above link. 

Cheers,

Don

very helpful link.  Was hard for me to do exactly what they said, but will try on a long up coming trip.  The revision applies their two most useful denoising ATW techiques... I used the same but setting were much better. 

They applied these to linear images and I had trouble with that and subsequently stretching. 



You can probably look at the large image and see the improvement. 

[jerryyyyy]

i also shoot with the d800 unmodified.
do you have the firmware hack already?
i do not have seen these kind of noise at my stackings ever!
please post some informations about your camera settings: ISO, shutterspeed, time, etc...

Thanks.  Another user Wai has recommended this but I do not have the link?  There is a lot of real estate on this chip. 

[jerryyyyy]

fair enough. it may be that because it's an OSC then there's a correlation between pixels (after debayering) that otherwise would not be there, and so a larger dither does help.

rob

Pfile also gets credit for suggesting to blink the raw files and the distortion follows the direction of the stars. 

His comments:  "so anyway probably you have warm pixels in the calibrated frames. consistent DSLR calibration is next to impossible due to lack of temperature control. probably if you try CosmeticCorrection you can get rid of those hot/warm pixels. worth a try anyway."

So, in summery what have I learned:

1.  Lower the ISO and lengthen the Subs
2.  Get the Nikon D800 Hack
3.  Use the noise reduction techniques
4.  Substantially dither
5.  Conside cosmetic correction (does not allow this in BPP).

If I missed anything, will edit the list.