Background noise reduction

Harry page

PixInsight Ambassador
Hi

I seem to be able to handle small scale noise very well in Pixinsight , but do not have a satisfactory way of doing the background without creating the dreaded lumps . :surprised:

I would be interested in other peoples approach to this  O:)

Harry
 
Hi Harry:

Could you post an image example?

This is an example of how well ACDNR perform with the 2 step ACDNR workflow.

One for Small Scale noise, and another for Large Scale noise.

Image is a integration of 1 subexposure each channel.

5666375203_274f935dd4_b.jpg


Reading this tutorial is a must if you want to do ACDNR properly.

http://www.pixinsight.com/tutorials/ACDNR/en.html
 
Hi

Yes thanks for that , I was aware of that and do know in practice what to do , but Often seem to end up with the lumpy background perhapes I am stretching my images too much  :-[  something I often do ( trying to get every last bit out of the data )

Thanks for your input


Harry
 
I'm facing the very same problem - I am not able to remove large-scale noise with satisfactory result.
With ACDNR, the result usually feels like noise was just shifted from high frequencies to lower ones.
On contrary, programs like NeatImage or NoiseNinja seems to really "remove" noise, without "washing" it from small grains to large blobs.
I even tried to implement some noise reduction algorithms in PixInsight, but up to now, without satisfactory result.

Zbynek
 
On contrary, programs like NeatImage or NoiseNinja seems to really "remove" noise, without "washing" it from small grains to large blobs.

Are you aware of PixInsight's GREYCstoration implementation? I honestly think GREYCstoration is fairly superior to those plugins,

By the way, I cannot stress enough the fact that David Schumperl?, the author of the GREYCstoration, kindly given me his permission to implement his algorithms as an open-source PixInsight module.

As Silvercup has shown with his nice example, ACDNR can be used iteratively to effectively remove the noise at growing scales. I agree however that the procedures involved usually require experience and significant trial-error work. But the results normally are well worth the effort.

On the other hand, we have a powerful wavelet-based noise reduction routine implemented in the ATrousWaveletTransform tool. Unlike ACDNR and GREYCstoration though, ATWT noise reduction can be applied to linear and nonlinear images. This noise reduction is very efficient for astronomical images, although it also requires a lot of fine tuning. If you upload an image, I'll be glad to make a brief example.

One of the advantages of ACDNR and ATWT noise reduction for astronomical images is that they are purely isotropic algorithms. GREYCstoration and the plugins you have cited are anisotropic regularization algorithms. This poses the risk to introduce some alterations to the morphology and distribution of image structures that, while admissible (and even sometimes desirable) in daylight images, should in general be avoided in astronomical images. Personally I only apply slight amounts of GREYCstoration in some cases at the final stages of processing, as a final refining step.
 
i frequently have this problem of excessive noise in the sky background of my image. i suppose this must be due to the light pollution at my home. i cool my camera, but nowhere near the amount that would be needed to significantly reduce dark current, so that could be a source of background noise as well.

i struggle with background noise reduction on every single image, despite help from silvercup on the iterative process. at this point i am left wondering if i am stuck with this noise, or if i just don't know how to use ACDNR.

the file pointed to in the link below is a stack of 40 4-minute exposures from a modified canon 50d running at iso 1600. the OTA is a 35+ year old celestron 8 (which needs collimation) and a 0.63x focal reducer. it would be interesting to me, and i would be grateful, if someone could take a look at this file and see what they can do with the background noise.

the stack has had the following processing done: 1) integration (with crop to area of interest), 2) dynamic background extraction, 3) canon banding reduction on the red channel, and 4) deconvolution through a mask which exposed only the brightest parts of the galaxy. the supernova and other bright stars in front of the galaxy were masked during deconvolution due to problems with ringing.

thanks for any advice...

http://dl.dropbox.com/u/8884840/Image35_DBE_decon.fit.zip
 
Hi,

I was trying a little and it is not easy, but perhaps it could help.

Attach the process.


Hola,

Estuve probando un poco y no resulta f?cil, pero quiz?s pueda ayudar.

Adjunto los procesos

Saludos.
Alejandro.
 

Attachments

  • Canon50Dpfile.xpsm
    23.7 KB · Views: 265
no se como applicar los procesos. en orden num?rico o en orden de positi?n? es la ATWavelets proceso (process27) para la construcci?n de una m?scara?

es posible crear un proyecto de pixi 1.7? si tengo la historia, va a ser m?s f?cil de comprendar.

lo siento para mi malo espa?ol.


--

i don't know how to apply the processes. in numerical order or in positional order? is the ATWavelets process (process27) for constructing a mask?

is it possible for you to make a pixinsight 1.7 project? if i have the history it will be easier to understand.

sorry for my bad spanish.

 
Hi:

It seems you deconvoluted the whole image without a mask protecting the background, so it is difficult to reduce background noise.

Could you upload a non-deconvoluted versi?n? I think ceconvolution can be improved with a non-lineal image.

Silvercup
 
Hi pfile,

I'm sorry for the short explanation!

The process are in order and now with new name:

Process01 Apply to the image
Extract Luminance and aplly Process02, Process03 and Process04 in order
Masking the galaxy apply Process05, Process06, Process07, Process08, Process09
Invert the mask to protect background and apply Process10

Silvercup, the deconvolution is done masking the background.

Regards, Alejandro.


Hola pfile,

Perd?n por la corta explicaci?n!

Los procesos est?n en orden y ahora con nuevo nombre:

Process01 aplicar a la imagen.
Extraer la luminancia y aplicar Process02, Process03 y Process04 en orden correlativo
Enmascarar la galaxia y aplicar Process05, Process06, Process07, Process08, Process09
Invertir la mascara para proteger el fondo y aplicar Process10

Silvercup, la deconvoluci?n est? hecha con m?scara en el fondo.

Saludos, Alejandro
 

Attachments

  • Canon50Dpfile.xpsm
    23.7 KB · Views: 150
Silvercup said:
Hi:

It seems you deconvoluted the whole image without a mask protecting the background, so it is difficult to reduce background noise.

Could you upload a non-deconvoluted versi?n? I think ceconvolution can be improved with a non-lineal image.

Silvercup

no, it was completely masked off. believe me the background of the mask is 0, i worked hard to make sure i was not even deconvolving the dimmer parts of the galaxy. i can post the raw stack or just the DBE'd stack, though.

Alejandro's processes did work well, though. i figured out how to apply them.


 
Hi folks,

Harry has been very kind to upload one of his images as the subject for a noise reduction tutorial. The file he has uploaded is a remarkably good image of the M81/M82 pair. In Harry's words "this image has a bit of flux in it", and it has been indeed a real pleasure for me to play with it. Take the following as a good example of how easy things can go when there are plenty of well acquired data.

As there are many examples on noise reduction with the ACDNR tool out there, I'll put an example of wavelet-based noise reduction with the ATrousWaveletTransform tool. I'll show you how this tool can be applied to implement an efficient noise reduction procedure in both the linear and nonlinear stages. This is a unique characteristic of ATWT; no other noise reduction tool can be applied to linear images.

So we begin loading the image and applying an automatic STF, as usual. See it below.


An interesting background for sure. The next step is building a mask to protect high SNR areas during noise reduction. Good masking is particularly important with a linear image. The mask is actually as simple as it can be: I have duplicated the image and applied a histogram transform, which is just the automatic STF transferred to the HistogramTransformation tool.


This mask must be activated on the original (linear) image with mask inversion, since we want to protect bright areas.


Here is the ATrousWaveletTransform tool with the noise reduction parameters I have applied. First the image enlarged 2:1 before noise reduction:


Now the same view after noise reduction:


Let's take a closer look. This is before noise reduction, zoomed 6:1:


And the same view after noise reduction:


I would say the result is excellent. The main reason is, beyond the quality of the applied tools, that with abundant data image processing simply works and yields good results in a natural and smooth way. With insufficient signal, one often feels like trying to substantiate a questionable result from marginal data. This is particularly true of noise reduction.

I want to draw your attention to several important facts about the above noise reduction procedure:

- We are working on the linear image. As I said above, wavelet-based noise reduction works with both linear and nonlinear images. This is a nice feature because noise reduction can be much easier to understand and more controllable with linear data, especially with high SNR linear data.

- A simple lightness mask is being used. The mask has been activated with inversion because we want to protect high SNR regions, that is bright pixels. Recall that a mask protects where it is black, and allows full processing where it is white.

- Wavelet-based noise reduction works on a per-layer basis. By applying noise reduction to the first wavelet layer, we can suppress or reduce high-frequency noise. On subsequent layers we can apply noise reduction to larger structures. In this case we have worked on the first four wavelet layers, that is up to the scale of eight pixels.

- Wavelet noise reduction parameters are very easy to understand. This is one of the reasons why wavelet-based noise reduction can be so powerful. We have the following parameters for each layer:

* Threshold. This parameter is expressed in sigma units. Sigma here refers to the standard deviation of the set of wavelet coefficients in a wavelet layer. As you known the standard deviation is a measurement of dispersion in a data set. Wavelet coefficients have positive and negative values and a mean value of zero in each layer. Coefficients corresponding to significant image structures tend to have larger magnitudes (absolute values), while coefficients corresponding to the noise are smaller and hence closer to the central peak of the distribution. The threshold parameter tells how much of these noisy coefficients will be removed or attenuated. By increasing threshold you can remove more noise, but if you increase threshold too much you'll start removing significant structures.

* Amount. This parameter governs the degree of attenuation applied to noise wavelet coefficients. When amount is one, noise coefficients will be completely removed (well, the actual process is not so simple but you get the idea).

* Iterations. Sometimes noise reduction can be better controlled by applying the same process several times on the same wavelet layer. For example, one can decrease amount and increase iterations as a way to gain more fine control.

Note that a threshold value of 3 sigma means that a 99.7% of the wavelet coefficients will be removed or attenuated. One can only apply such a strong noise reduction to the first wavelet layer, and not in all cases. This is because the first wavelet layer supports most of the high-frequency noise in the image and few significant coefficients, but each image is different. For most deep-sky images, threshold=3 is a good starting point for the first wavelet layer. For subsequent layers, threshold must be reduced drastically to avoid destroying important image features.

Now the image is no nice and seems to respond so well to everything I do, that I can't resist doing a more elaborate example. This is a first nonlinear stretch. As is customary in PixInsight, we have high precision tools so this can be done perfectly in a single step --forget all that ugly multiple stretches so common in other applications. They simply reflect the inability of those applications to handle real image data (they are image retouching toys, not real image processing software).


Now this is a busy background (I read this sentence somewhere and nobody understood him --I did/do). How about a HDR wavelet transform? Here it is.


Easy and effective, doing justice to all the nice data we have. Again, this is real image processing.

Finally, as an option, we can apply a second noise reduction with wavelets. This is the mask, which is a simple duplicate of the image:


And here is the result of wavelet-based noise reduction applied to the nonlinear data:


As I said before, a real pleasure. I hope you find it useful. Thanks Harry and congrats on a very nice image. I do hope you've got good RGB data for this one! :)
 
thanks juan. we've seen several methods for background noise reduction: acdnr, wavelets and greycstoration. is there a rule of thumb for which one is applicable to what situation?
 
Hi Robert,

pfile said:
thanks juan. we've seen several methods for background noise reduction: acdnr, wavelets and greycstoration. is there a rule of thumb for which one is applicable to what situation?

Just as a rule of thumb—which means that you can always find examples where this is not applicable:

GREYCstoration is an extremely powerful noise reduction tool (perhaps the best one IMO) for daylight images. For deep sky astrophotography it works well during the final stages of post processing, as a finishing tool to be applied cautiously, but I don't recommend using it otherwise. When applying it, watch for possible artifacts generated as a result of excessive anisotropic filter response. This algorithm (at least in its present implementation) doesn't work for linear data.

ACDNR has been our noise reduction workhorse for a long time. When properly applied, ACDNR yields excellent results for nonlinear deep sky images. However, there are many cases where applying it correctly isn't an easy or obvious task. This happens when the noise must be reduced at a wide range of dimensional scales. In these cases, several applications are necessary with increasing filter size and mask protection. This can lead to manual procedures requiring a lot of experience and trial error work.

I designed the current version of the ACDNR algorithm in 2005-2006. I think it's time for a serious revamping. New noise reduction tools are now necessary, with more efficient algorithms and implementations. So don't be surprised if ACDNR gets moved to the Obsolete category in a future release, in the short-medium term, and replaced with new, more efficient noise reduction tools.

Finally, we have ATrousWaveletTransform. I've just put a good example of what this tool can do for noise reduction. As usual, the multiscale paradigm provides new and powerful ways of facing old problems. In this case, noise reduction becomes simplified because wavelet layers allow us to isolate the noise at different dimensional scales in a natural way. With just two or three simple and easy-to-understand parameters we can attack the noise very efficiently, with the added bonus that it works equally well on linear and nonlinear data. I recommend trying wavelet-based noise reduction in the first place for deep sky images. When no good results can be achieved with wavelets—no algorithm or tool works in all situations—, ACDNR remains as a solid fallback option.
 
Hi Juan

Thanks for your Demo , I need to take it all in and as my brain is small I will need a few days  ;D

I do have some binned RGB but it seems to be very poor as I am new to this Binned thing , plus I had some hardware issues to resolve  :'(  my efforts so far are not satisfactory and My RGB might even be beyond your repair  :footinmouth:

Regards Harry
 
Hi Juan,

thanks a lot for demo, it is excellent. I wonder how many hidden (= undocumented) pearls like this one are there in the PixInsight  8)). I've noticed this noise reduction parameters in ATWT some time before, but I did not experimented with it not knowing what I could expect. I'll definitely start to now.
One question - would it be possible to compute these parameters from "noise sample" - small preview containing only noise? Maybe some kind of script doing this could highly improve ease-of-use of this method.

regards, Zbynek
 
Back
Top