Author Topic: Estimating Noise using NGC7000 script gives strange results  (Read 3667 times)

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Hello,

I have been using the script in http://pixinsight.com/examples/wavelets/NGC7000/en.html to estimate the noise of some of my pictures. The results were strange. So I created a test image with value 0.2 (Process/Image/New Image), added gausian noise with "Amount 1" (Process/Noise Generation/NoiseGenerator), and had a look at the statistics:


http://cid-c56e60845cfa6790.skydrive.live.com/self.aspx/Pixinsight/PixNoise.jpg

clearly show there is something strange: The statistics tool reports a StdDev of 0.06, while the script estimates 0.63. I would expect minor differences, but not a factor of 10...

What is wrong here?

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Nocturnal

  • PixInsight Jedi Council Member
  • *******
  • Posts: 2727
    • http://www.carpephoton.com
Re: Estimating Noise using NGC7000 script gives strange results
« Reply #1 on: 2009 May 17 14:54:01 »
You're a busy man Georg :) Great work.

It's unlikely that the statistics module is wrong but if you'd like to check you could write a JSR script that calculates the stddev of that image and compare.
Best,

    Sander
---
Edge HD 1100
QHY-8 for imaging, IMG0H mono for guiding, video cameras for occulations
ASI224, QHY5L-IIc
HyperStar3
WO-M110ED+FR-III/TRF-2008
Takahashi EM-400
PIxInsight, DeepSkyStacker, PHD, Nebulosity

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Estimating Noise using NGC7000 script gives strange results
« Reply #2 on: 2009 May 17 15:21:32 »
Well, there are those days when the family is very understanding... and on other days, you have got to paint the conservatory (Greetings to Harry ... ;) ).

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Estimating Noise using NGC7000 script gives strange results
« Reply #3 on: 2009 May 17 15:45:08 »
Indeed this is very strange. I've made a similar test to yours, and my results are correct.

I've used this script:

Code: [Select]
/**
 * Evaluation of Gaussian noise by the iterative k-sigma thresholding method.
 *
 * Reference:
 *    J.L. Starck, F. Murtagh, Astronomical Image and Data Analysis, Springer,
 *    1st ed., 2002, pp. 37-38.
 */

// Working array to gather noise at each iteration.
var N = new Array;

/**
 * Image callback routine to extract noise wavelet coefficients.
 *
 * s     is the value of the current sample, which is actually a wavelet
 *       coefficient in the first wavelet layer
 *
 * ks    k*sigma threshold, with k=3 and sigma is the standard deviation of the
 *       noise in the previous iteration.
 */
function GetNoise( x, y, c, s, ks )
{
   if ( Math.abs( s ) < ks )
      N.push( s );
   else
      this.currentSample = 1.0; // tag this coefficient as significant
}

/**
 * Iterative calculation of the standard deviation of Gaussian noise noise.
 * We use k=3 and return an estimate to within 1% accuracy.
 */
function Noise( img )
{
   // Perform a one-layer wavelet transform using a B3 spline scaling function.
   var W = img.aTrousWaveletTransform( [ 1.0/256, 1.0/64, 3.0/128, 1.0/64, 1.0/256,
                                         1.0/64,  1.0/16, 3.0/32,  1.0/16, 1.0/64,
                                         3.0/128, 3.0/32, 9.0/64,  3.0/32, 3.0/128,
                                         1.0/64,  1.0/16, 3.0/32,  1.0/16, 1.0/64,
                                         1.0/256, 1.0/64, 3.0/128, 1.0/64, 1.0/256 ], 1 );
   for ( var i = 0, s0; ; )
   {
      // Standard deviation of the noise for this iteration.
      var s;
      if ( ++i == 1 ) // first iteration?
         s = s0 = W[0].stdDev();
      else
      {
         // Standard deviation of the set of thresholded coefficients.
         s = Math.stddev( N );

         // Return an estimate to within 1% accuracy.
         var converges = (s0 - s)/s0 < 0.01;
         var enough = i == 20;
         if ( converges || enough )
         {
            console.writeln( (converges ? "Convergence reached" :
                                          "*** Warning: No convergence"),
                             " after ", i, " iterations." );
            return s/0.8908; // back to the image space
         }

         s0 = s;
      }

      // Extract the set of noise pixels for the next iteration.
      N.length = 0;
      W[0].forEachSample( GetNoise, 3*s );
      if ( N.length < 2 )
         return 0;
   }
}

function main()
{
   // Get access to the current active image window.
   var window = ImageWindow.activeWindow;
   if ( window.isNull )
      throw new Error( "No active image" );

   console.show();
   console.writeln( "<b>" + window.currentView.fullId + "</b>" );
   console.writeln( "Calculating noise standard deviation..." );
   console.flush();

   // Obtain an estimate of the noise standard deviation for the current view.
   var s = Noise( window.currentView.image );

   // Output results.
   console.writeln( format( "Noise standard deviation = %.8f, N = %u", s, N.length ) );
}

main();

which is a slightly improved version using the B3 spline scaling function and a more accurate value of the noise sigma in the first wavelet layer. However, the original script from the NGC7000 tutorial yields nearly the same result (actually, this algorithm cannot provide more than about 1% accuracy, so only two decimals are really significant).

The 0.8908 constant is the standard deviation of the first wavelet layer for a random Gaussian distribution of zero mean and unit sigma. I calculated this value using high-quality synthetic data and a large number of experiments (it is an improved value with respect to the original one published by Starck in his book, 0.889).
« Last Edit: 2009 May 17 15:49:15 by Juan Conejero »
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Estimating Noise using NGC7000 script gives strange results
« Reply #4 on: 2009 May 17 18:08:41 »
Hola Juan,

your improved script gives the right answer:
Code: [Select]
Noise Standard Deviation = 0.068...

The difference is in line 60. It reads
Code: [Select]
return 10*s/0.8908; // back to the image space
instead of
Code: [Select]
return s/0.8908; // back to the image space

Apparently I got the code from your post in http://pixinsight.com/forum/index.php?topic=1105.msg5234#msg5234, that contains this line. Maybe you fix your post before the wrong code spreads further. Anyway, thanks for investigating, and for giving us this great tool. :)

Georg
« Last Edit: 2009 May 17 18:10:24 by georg.viehoever »
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)