Author Topic: Noise Reduction Challenge  (Read 20656 times)

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2173
  • Join the dark side... we have cookies
    • View Profile
    • http://www.astrophoto.cl
Noise Reduction Challenge
« on: 2012 June 11 17:58:00 »
As some of you know, we are developing new noise reduction tools. As part of some early experiments, I created a synthetic image, composed by two linear gradients. To this, gaussian noise has being added to simulate a real, noisy image. Attached are the original image, and three noisy images, with different degrees of noise.
So, the challenge is this: Use all the tools you want to process the images (either inside PI, and with other software) and publish your best results. If you upload results done with different tools (for example, comparing your best results with GREYCstoration and ACDNR) that would be greatly appreciated. Please accompany your results with detailed description of the steps (if there are more than one), all the parameters that you adjusted, approximated time of execution, and some comments of yours about the sensibility of the parameters and easy of use.

Results should be evaluated by three criteria:
- Edge preservation
- Presence of artefacts (Gibbs effects, spurious pixels, staircase effects, etc).
- Smoothness of the gradients.

Images may be rescaled for display or comparison, after noise reduction. If you want to compute an error measurement, it should be the mean quadratic difference between the original and the denoised image, or the mean absolute difference.


Have fun! :)
I'll upload the results of the new algorithm in a few days.

Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline Enzo De Bernardini

  • PTeam Member
  • PixInsight Addict
  • ***
  • Posts: 271
  • Resistance is futile.
    • View Profile
    • Astronomí­a Sur
Re: Noise Reduction Challenge
« Reply #1 on: 2012 June 12 14:47:44 »
Here we go! :D

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2173
  • Join the dark side... we have cookies
    • View Profile
    • http://www.astrophoto.cl
Re: Noise Reduction Challenge
« Reply #2 on: 2012 June 12 16:44:29 »
Very nice results, Enzo! I'm specially surprised by the performance of ACDNR in the harder example. Was it difficult to fine tune the parameters?
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline Enzo De Bernardini

  • PTeam Member
  • PixInsight Addict
  • ***
  • Posts: 271
  • Resistance is futile.
    • View Profile
    • Astronomí­a Sur
Re: Noise Reduction Challenge
« Reply #3 on: 2012 June 12 20:50:50 »
Not much (which required more trial and error was GREYCstoration and MMT). Images with stars, color, fine structures and various noise types are more complicated I think. I suspect also that MMT can achieve better results.

Greetings,

Enzo.

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2173
  • Join the dark side... we have cookies
    • View Profile
    • http://www.astrophoto.cl
Re: Noise Reduction Challenge
« Reply #4 on: 2012 June 13 14:03:26 »
Well, since there are no more inputs, I'll show the results I've got with the new algorithm. It is based on Total Generalized Variation. Simple put, is a diffusion problem (i.e. two fluids interacting) that is set to evolve until a steady state is reached. This physical evolution has some restrictions to preserve edges, and to generate smooth surfaces when the gradient involves are low, thus, avoiding staircase artefacts.

The nice thing about this algorithm is that it is also useful to regularize deconvolutions, and other inverse problems. So, I may include it in other processes as well.
The PixInsight implementation is going to wait for a while... I'm still experimenting with the algorithm, and designing new ways to improve it. Specially, I'm looking for an statistical method that spatially modifies the strength of the algorithm (this may result in the deprecation of the use of masks).


I'm going to post another challenge, with real data, to compare algorithms. Also, it would be great if someone attacks the previous challenge with another approach.
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline Alejandro Tombolini

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1267
    • View Profile
    • Próxima Sur
Re: Noise Reduction Challenge
« Reply #5 on: 2012 June 13 14:16:25 »
Well, since there are no more inputs,
Time, time, time!!!. I don't know where to find it!

Here is my approach always with more than one tool in each case.  It took me about five minutes for gauss 10 and 20 and half an hour for gauss40.

Saludos!, Alejandro

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2173
  • Join the dark side... we have cookies
    • View Profile
    • http://www.astrophoto.cl
Re: Noise Reduction Challenge
« Reply #6 on: 2012 June 13 14:29:26 »
Very nice! I'm impressed by your last result. I'm gonna check your icons tonight.
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2173
  • Join the dark side... we have cookies
    • View Profile
    • http://www.astrophoto.cl
Re: Noise Reduction Challenge
« Reply #7 on: 2012 June 14 11:59:35 »
New challenge, now with real world data:

This is a crop of CCD data, green filter, already stretched, but no further processing. This corresponds to an integrated and calibrated master light.

Good luck!


PS: Alejandro, clever approach. I played with your icons. It seems that a soft threshold yields a bit better results, in ATWT. Also, the first two processes of your chain worked quite well with this new challenge, but the third failed completely.
PS2: As before, I'll upload the results of the new algorithm in a couple of days.
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline Philippe B.

  • PixInsight Old Hand
  • ****
  • Posts: 389
    • View Profile
    • CIEL AUSTRAL
Re: Noise Reduction Challenge
« Reply #8 on: 2012 June 16 02:22:56 »
Hi Carlos

Here is your original image :


Here is my processed version
I wanted to preserve maximum details, I used MMT with your original image as a mask :




a blink is interesting to do to compare 2 images.

cheers
« Last Edit: 2012 June 16 02:51:55 by Philippe B. »

Offline Alejandro Tombolini

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1267
    • View Profile
    • Próxima Sur
Re: Noise Reduction Challenge
« Reply #9 on: 2012 June 16 14:45:36 »
Hi Carlos

I started with CBR to minimize the lines and then used deconvolution and ATWT, in both cases with mask. I also deleted a black pixel with CloneStamp.
Attached are the process.

Rotate 90º, apply CBR and rotate back again.



Repair black pixel


Generate starmask to use in deconvolution to protect stars



Clone the image, apply LHE and ATWT and use as mask



Noise reduction with ATWT protecting stars with mask



Final


Saludos. Alejandro.

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2173
  • Join the dark side... we have cookies
    • View Profile
    • http://www.astrophoto.cl
Re: Noise Reduction Challenge
« Reply #10 on: 2012 June 20 17:42:38 »
Hi guys

Sorry for the delay. Here is the result with the current implementation of the new noise reduction algorithm. No masks, and only 2 parameters to fine tune. I hope you like it.

Now I'm working on an adaptive version of the algorithm, that optimizes the search of the data consistency parameter. This will yield, in practice, only one critical parameter to fine tune.
Another path I'm following is using this algorithm to regularize deconvolutions. In the next post I'll show you an example with MRI data.
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2173
  • Join the dark side... we have cookies
    • View Profile
    • http://www.astrophoto.cl
Re: Noise Reduction Challenge
« Reply #11 on: 2012 June 20 18:02:05 »
Here are the results of the use of TGV as a regularization constrain in a deconvolution.

This is an angiography image. I'm showing the original image. If we simulate a compressed sensing capture (that means, taking less samples... it would be something like having pixels not exposed to light) in the fourier space, this would be seeing like a complex bluring pattern. In this example, I simulated a 8x subsampling (randomly taking only 1/8th of the entire fourier space). At last, it is shown the reconstruction made with TGV.
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline Philippe B.

  • PixInsight Old Hand
  • ****
  • Posts: 389
    • View Profile
    • CIEL AUSTRAL
Re: Noise Reduction Challenge
« Reply #12 on: 2012 June 21 01:54:58 »
Thanks Carlos
Looks very promising  8)
For sure PI will rock at the top of the top with these new functions !


Note there is some denoising process rendering difference between USA and Europe ! :police:
US users tend to completely denoise their images whereas europeans tend to preserve a little touch of noise to make the image more realistic. 
It is not a critic. Just a way to see an image.  Noise can be nice if well processed  ;)
But Juan tends to be on the US way of denoising  >:D >:D >:D >:D
So, sometimes, it is hard to exactly follow a processing tutorial. Some rendition are too much processed with not noise at all. So where you apply coefficient of 100%, I must go between 20 to 50%  O:)
In my example I tried to go in US direction  :angel:
 

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2173
  • Join the dark side... we have cookies
    • View Profile
    • http://www.astrophoto.cl
Re: Noise Reduction Challenge
« Reply #13 on: 2012 June 21 06:47:10 »
Well, in fact, we started the "US way", quite a long ago... so it is the "hispanoamerican way". ;)

Talking seriously, I was very impressed by the first examples I saw with this algorithm. And it is quite new. The first publication was in 2009, while the papers I'm basing most of my implementations are from February and April this year. So, we can say with property that we are in the top of the wave.
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline Philippe B.

  • PixInsight Old Hand
  • ****
  • Posts: 389
    • View Profile
    • CIEL AUSTRAL
Re: Noise Reduction Challenge
« Reply #14 on: 2012 June 21 07:14:21 »
Yes, I think this new algorithm is very powerful.
I read some papers about TGV (Total Generalized Variation, 2nd and 3rd order )