Author Topic: Background noise reduction  (Read 70845 times)

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1458
    • http://www.harrysastroshed.com
Background noise reduction
« on: 2011 June 25 01:33:37 »
Hi

I seem to be able to handle small scale noise very well in Pixinsight , but do not have a satisfactory way of doing the background without creating the dreaded lumps . :surprised:

I would be interested in other peoples approach to this  O:)

Harry
Harry Page

Offline Silvercup

  • PixInsight Addict
  • ***
  • Posts: 187
Re: Background noise reduction
« Reply #1 on: 2011 June 25 03:08:44 »
Hi Harry:

Could you post an image example?

This is an example of how well ACDNR perform with the 2 step ACDNR workflow.

One for Small Scale noise, and another for Large Scale noise.

Image is a integration of 1 subexposure each channel.



Reading this tutorial is a must if you want to do ACDNR properly.

http://www.pixinsight.com/tutorials/ACDNR/en.html

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1458
    • http://www.harrysastroshed.com
Re: Background noise reduction
« Reply #2 on: 2011 June 25 05:01:11 »
Hi

Yes thanks for that , I was aware of that and do know in practice what to do , but Often seem to end up with the lumpy background perhapes I am stretching my images too much  :-[  something I often do ( trying to get every last bit out of the data )

Thanks for your input


Harry
Harry Page

Offline zvrastil

  • PixInsight Addict
  • ***
  • Posts: 179
    • Astrophotography
Re: Background noise reduction
« Reply #3 on: 2011 June 25 08:10:41 »
I'm facing the very same problem - I am not able to remove large-scale noise with satisfactory result.
With ACDNR, the result usually feels like noise was just shifted from high frequencies to lower ones.
On contrary, programs like NeatImage or NoiseNinja seems to really "remove" noise, without "washing" it from small grains to large blobs.
I even tried to implement some noise reduction algorithms in PixInsight, but up to now, without satisfactory result.

Zbynek

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Background noise reduction
« Reply #4 on: 2011 June 25 11:28:55 »
Quote
On contrary, programs like NeatImage or NoiseNinja seems to really "remove" noise, without "washing" it from small grains to large blobs.

Are you aware of PixInsight's GREYCstoration implementation? I honestly think GREYCstoration is fairly superior to those plugins,

By the way, I cannot stress enough the fact that David Schumperlé, the author of the GREYCstoration, kindly given me his permission to implement his algorithms as an open-source PixInsight module.

As Silvercup has shown with his nice example, ACDNR can be used iteratively to effectively remove the noise at growing scales. I agree however that the procedures involved usually require experience and significant trial-error work. But the results normally are well worth the effort.

On the other hand, we have a powerful wavelet-based noise reduction routine implemented in the ATrousWaveletTransform tool. Unlike ACDNR and GREYCstoration though, ATWT noise reduction can be applied to linear and nonlinear images. This noise reduction is very efficient for astronomical images, although it also requires a lot of fine tuning. If you upload an image, I'll be glad to make a brief example.

One of the advantages of ACDNR and ATWT noise reduction for astronomical images is that they are purely isotropic algorithms. GREYCstoration and the plugins you have cited are anisotropic regularization algorithms. This poses the risk to introduce some alterations to the morphology and distribution of image structures that, while admissible (and even sometimes desirable) in daylight images, should in general be avoided in astronomical images. Personally I only apply slight amounts of GREYCstoration in some cases at the final stages of processing, as a final refining step.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: Background noise reduction
« Reply #5 on: 2011 June 25 17:59:25 »
i frequently have this problem of excessive noise in the sky background of my image. i suppose this must be due to the light pollution at my home. i cool my camera, but nowhere near the amount that would be needed to significantly reduce dark current, so that could be a source of background noise as well.

i struggle with background noise reduction on every single image, despite help from silvercup on the iterative process. at this point i am left wondering if i am stuck with this noise, or if i just don't know how to use ACDNR.

the file pointed to in the link below is a stack of 40 4-minute exposures from a modified canon 50d running at iso 1600. the OTA is a 35+ year old celestron 8 (which needs collimation) and a 0.63x focal reducer. it would be interesting to me, and i would be grateful, if someone could take a look at this file and see what they can do with the background noise.

the stack has had the following processing done: 1) integration (with crop to area of interest), 2) dynamic background extraction, 3) canon banding reduction on the red channel, and 4) deconvolution through a mask which exposed only the brightest parts of the galaxy. the supernova and other bright stars in front of the galaxy were masked during deconvolution due to problems with ringing.

thanks for any advice...

http://dl.dropbox.com/u/8884840/Image35_DBE_decon.fit.zip

Offline Alejandro Tombolini

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1267
    • Próxima Sur
Re: Background noise reduction
« Reply #6 on: 2011 June 26 14:53:16 »
Hi,

I was trying a little and it is not easy, but perhaps it could help.

Attach the process.


Hola,

Estuve probando un poco y no resulta fácil, pero quizás pueda ayudar.

Adjunto los procesos

Saludos.
Alejandro.

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: Background noise reduction
« Reply #7 on: 2011 June 26 16:19:20 »
gracias, voy a mirar a este xpsm.

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: Background noise reduction
« Reply #8 on: 2011 June 27 00:00:56 »
no se como applicar los procesos. en orden numérico o en orden de positión? es la ATWavelets proceso (process27) para la construcción de una máscara?

es posible crear un proyecto de pixi 1.7? si tengo la historia, va a ser más fácil de comprendar.

lo siento para mi malo español.


--

i don't know how to apply the processes. in numerical order or in positional order? is the ATWavelets process (process27) for constructing a mask?

is it possible for you to make a pixinsight 1.7 project? if i have the history it will be easier to understand.

sorry for my bad spanish.


Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: Background noise reduction
« Reply #9 on: 2011 June 27 10:38:44 »
lo siento, no vi las notas en los iconos de proceso, voy a mirar una vez mas.

Offline Silvercup

  • PixInsight Addict
  • ***
  • Posts: 187
Re: Background noise reduction
« Reply #10 on: 2011 June 27 14:00:45 »
Hi:

It seems you deconvoluted the whole image without a mask protecting the background, so it is difficult to reduce background noise.

Could you upload a non-deconvoluted versión? I think ceconvolution can be improved with a non-lineal image.

Silvercup
« Last Edit: 2011 June 27 15:32:03 by Silvercup »

Offline Alejandro Tombolini

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1267
    • Próxima Sur
Re: Background noise reduction
« Reply #11 on: 2011 June 27 17:12:54 »
Hi pfile,

I'm sorry for the short explanation!

The process are in order and now with new name:

Process01 Apply to the image
Extract Luminance and aplly Process02, Process03 and Process04 in order
Masking the galaxy apply Process05, Process06, Process07, Process08, Process09
Invert the mask to protect background and apply Process10

Silvercup, the deconvolution is done masking the background.

Regards, Alejandro.
 

Hola pfile,

Perdón por la corta explicación!

Los procesos están en orden y ahora con nuevo nombre:

Process01 aplicar a la imagen.
Extraer la luminancia y aplicar Process02, Process03 y Process04 en orden correlativo
Enmascarar la galaxia y aplicar Process05, Process06, Process07, Process08, Process09
Invertir la mascara para proteger el fondo y aplicar Process10

Silvercup, la deconvolución está hecha con máscara en el fondo.

Saludos, Alejandro

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: Background noise reduction
« Reply #12 on: 2011 June 27 20:26:14 »
Hi:

It seems you deconvoluted the whole image without a mask protecting the background, so it is difficult to reduce background noise.

Could you upload a non-deconvoluted versión? I think ceconvolution can be improved with a non-lineal image.

Silvercup

no, it was completely masked off. believe me the background of the mask is 0, i worked hard to make sure i was not even deconvolving the dimmer parts of the galaxy. i can post the raw stack or just the DBE'd stack, though.

Alejandro's processes did work well, though. i figured out how to apply them.



Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Background noise reduction
« Reply #13 on: 2011 June 29 05:47:28 »
Hi folks,

Harry has been very kind to upload one of his images as the subject for a noise reduction tutorial. The file he has uploaded is a remarkably good image of the M81/M82 pair. In Harry's words "this image has a bit of flux in it", and it has been indeed a real pleasure for me to play with it. Take the following as a good example of how easy things can go when there are plenty of well acquired data.

As there are many examples on noise reduction with the ACDNR tool out there, I'll put an example of wavelet-based noise reduction with the ATrousWaveletTransform tool. I'll show you how this tool can be applied to implement an efficient noise reduction procedure in both the linear and nonlinear stages. This is a unique characteristic of ATWT; no other noise reduction tool can be applied to linear images.

So we begin loading the image and applying an automatic STF, as usual. See it below.


An interesting background for sure. The next step is building a mask to protect high SNR areas during noise reduction. Good masking is particularly important with a linear image. The mask is actually as simple as it can be: I have duplicated the image and applied a histogram transform, which is just the automatic STF transferred to the HistogramTransformation tool.


This mask must be activated on the original (linear) image with mask inversion, since we want to protect bright areas.


Here is the ATrousWaveletTransform tool with the noise reduction parameters I have applied. First the image enlarged 2:1 before noise reduction:


Now the same view after noise reduction:


Let's take a closer look. This is before noise reduction, zoomed 6:1:


And the same view after noise reduction:


I would say the result is excellent. The main reason is, beyond the quality of the applied tools, that with abundant data image processing simply works and yields good results in a natural and smooth way. With insufficient signal, one often feels like trying to substantiate a questionable result from marginal data. This is particularly true of noise reduction.

I want to draw your attention to several important facts about the above noise reduction procedure:

- We are working on the linear image. As I said above, wavelet-based noise reduction works with both linear and nonlinear images. This is a nice feature because noise reduction can be much easier to understand and more controllable with linear data, especially with high SNR linear data.

- A simple lightness mask is being used. The mask has been activated with inversion because we want to protect high SNR regions, that is bright pixels. Recall that a mask protects where it is black, and allows full processing where it is white.

- Wavelet-based noise reduction works on a per-layer basis. By applying noise reduction to the first wavelet layer, we can suppress or reduce high-frequency noise. On subsequent layers we can apply noise reduction to larger structures. In this case we have worked on the first four wavelet layers, that is up to the scale of eight pixels.

- Wavelet noise reduction parameters are very easy to understand. This is one of the reasons why wavelet-based noise reduction can be so powerful. We have the following parameters for each layer:

* Threshold. This parameter is expressed in sigma units. Sigma here refers to the standard deviation of the set of wavelet coefficients in a wavelet layer. As you known the standard deviation is a measurement of dispersion in a data set. Wavelet coefficients have positive and negative values and a mean value of zero in each layer. Coefficients corresponding to significant image structures tend to have larger magnitudes (absolute values), while coefficients corresponding to the noise are smaller and hence closer to the central peak of the distribution. The threshold parameter tells how much of these noisy coefficients will be removed or attenuated. By increasing threshold you can remove more noise, but if you increase threshold too much you'll start removing significant structures.

* Amount. This parameter governs the degree of attenuation applied to noise wavelet coefficients. When amount is one, noise coefficients will be completely removed (well, the actual process is not so simple but you get the idea).

* Iterations. Sometimes noise reduction can be better controlled by applying the same process several times on the same wavelet layer. For example, one can decrease amount and increase iterations as a way to gain more fine control.

Note that a threshold value of 3 sigma means that a 99.7% of the wavelet coefficients will be removed or attenuated. One can only apply such a strong noise reduction to the first wavelet layer, and not in all cases. This is because the first wavelet layer supports most of the high-frequency noise in the image and few significant coefficients, but each image is different. For most deep-sky images, threshold=3 is a good starting point for the first wavelet layer. For subsequent layers, threshold must be reduced drastically to avoid destroying important image features.

Now the image is no nice and seems to respond so well to everything I do, that I can't resist doing a more elaborate example. This is a first nonlinear stretch. As is customary in PixInsight, we have high precision tools so this can be done perfectly in a single step --forget all that ugly multiple stretches so common in other applications. They simply reflect the inability of those applications to handle real image data (they are image retouching toys, not real image processing software).


Now this is a busy background (I read this sentence somewhere and nobody understood him --I did/do). How about a HDR wavelet transform? Here it is.


Easy and effective, doing justice to all the nice data we have. Again, this is real image processing.

Finally, as an option, we can apply a second noise reduction with wavelets. This is the mask, which is a simple duplicate of the image:


And here is the result of wavelet-based noise reduction applied to the nonlinear data:


As I said before, a real pleasure. I hope you find it useful. Thanks Harry and congrats on a very nice image. I do hope you've got good RGB data for this one! :)
« Last Edit: 2011 June 29 08:55:27 by Juan Conejero »
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: Background noise reduction
« Reply #14 on: 2011 June 29 08:00:26 »
thanks juan. we've seen several methods for background noise reduction: acdnr, wavelets and greycstoration. is there a rule of thumb for which one is applicable to what situation?