CanonBandingReduction sometimes generates hairlines

[ruediger]

When using the CanonBandingReduction script for reducing banding noise, I often (but not always) see subtle hairlines in the result. This most often happens with the following settings:

Code: [Select]

Amount: 1.0
[x] Protect from Highlights
1/SigmaFactor: 3.0
The result has to be controlled very carefully under high magnification with aggressive STF setting to detect these lines. They may vanish with some trial&error changes of amount and 1/sigmafactor.

Rüdiger

[tommy_nawratil]

hello,

noticed this too, I mainly get one or two horizontal greenish 1px lines.
I easily camouflage them later on in processing.
In PixInsight you could use Pixelmath to define a horizontal white line in a black mask,
soften it a bit and apply a softening process or replace by intact pixels.

Here in this video Gerald Wechselberger explains how to draw a line without a pencil 
http://dl.dropbox.com/u/57910417/Video08_PixelMath_IIa_english.wmv

Tommy

[ruediger]

In PixInsight you could use Pixelmath to define a horizontal white line in a black mask,
soften it a bit and apply a softening process or replace by intact pixels.

So you think we need a CanonBandingReductionHairlineRepairScript after we used CanonBandingReduction? 

I'm varying the parameters until no hairline appears. This usually takes less than four attempts.

Georg hasn't been active in this forum for awhile. If he doesn't catch up this thread, I will have a look into the source code and try to find the reason for this strange behaviour.

Rüdiger

[georg.viehoever]

Ruediger,

I am seeing this thread, but currently I am too busy with other things. My guess would be that the script somewhere encounters an unusual numeric value like NaN when computing the averages for a line, and this value basically destroys the whole line of the image. Feel free to hunt down the bug...

Georg

[ruediger]

Feel free to hunt down the bug...

I'll try this weekend

Rüdiger

[ruediger]

I'm quite sure now, it's a bug in ImageStatistics module when High Rejection Limit is enabled and set.

After adding some debug code  for a test picture where row 370 gets overcorrected:

Code: [Select]

rejectionHigh = 2558
row = 367, rGBGlobalMedian = 2532, rGBRowMedian = 2528, fixFactor = 3
row = 368, rGBGlobalMedian = 2532, rGBRowMedian = 2527, fixFactor = 4
row = 369, rGBGlobalMedian = 2532, rGBRowMedian = 2527, fixFactor = 4
row = 370, rGBGlobalMedian = 2532, rGBRowMedian = 2509, fixFactor = 22
row = 371, rGBGlobalMedian = 2532, rGBRowMedian = 2528, fixFactor = 3
row = 372, rGBGlobalMedian = 2532, rGBRowMedian = 2527, fixFactor = 4
I converted all values to [0..65535] range for better visibility. So you see, there is a discontinuity in "fixFactor" for row 370 (fixFactor is the difference of global and row median)

But when I compute the median myself with the following code:

                 

Code: [Select]

if ( row == 370 ) {
                     var ca = new Array();
                     var rej = 0;

                     for ( var col = 0; col < targetWidth; col++ ) {
                        var pv = targetImage.sample( col, row );

                        if ( pv < aStatistic.rejectionHigh ) {
                           ca.push( Math.floor( pv * 65535.0 ) );
                        }
                        else {
                           rej++;
                        }
                     }
                     ca.sort();

                     console.writeln( 'reject = ' + rej );
                     console.writeln( 'median = ' + ca[ Math.floor( ca.length / 2 ) ] );
                  }
I get the following result for row 370:

Code: [Select]

reject = 857
median = 2528
And with this median value of 2528, the line will get fixFactor = 3 and not overcorrected (i.e. hairline).

So in essence: I could easily fix the script by replacing the ImageStatistic computation with my own code, but someone of the PTeam member should have a look in the ImageStatistic code.

If test data is needed, please tell me.

Rüdiger

[georg.viehoever]

Anyone in PTeam looking into this? It is a problem in the core application that potentially can strike anyone one using ImageStatistics.
Georg

[Juan Conejero]

Image statistics are correctly calculated. However, there is a small bug that may cause problems with ImageStatistics when setting rejection property values. See this GitHub commit that I've just pushed now, where I fix the bug in question:

https://github.com/PixInsight/PCL/commit/c4a4390185db3ccb04471f845d92917cc1bc1113

As a result of this simple mistake, assignments to the following properties of the ImageStatistics object may cause problems with current core versions 1.8.4.119x:

   ImageStatistics.lowRejectionEnabled
   ImageStatistics.highRejectionEnabled
   ImageStatistics.rejectionLow
   ImageStatistics.rejectionHigh

The following script demonstrates correctness of median calculations and describes two simple workarounds to prevent the effects of this bug:

Code: [Select]

function bruteForceMedian( I, low, high )
{
   let a = [];
   I.initSampleIterator();
   do
   {
      let v = I.sampleValue();
      if ( v > low )
         if ( v < high )
            a.push( v );
   }
   while ( I.nextSample() );

   let n2 = a.length >> 1;
   if ( n2 == 0 )
      return [-1, 0];
   a.sort();
   if ( a.length & 1 )
      return [a[n2], a.length];
   return [(a[n2] + a[n2-1])/2, a.length];
}

/*
 * Routine to test calculation of median values with rejection using
 * ImageStatistics, Image.median(), and a brute force safe implementation.
 *
 * N    : Number of tests
 * wmin : minimum image width in pixels.
 * wmax : maximum image width in pixels.
 * hmin : minimum image height in pixels.
 * hmax : maximum image height in pixels.
 */
function TestMedian( N, wmin, wmax, hmin, hmax )
{
   let S = new ImageStatistics;

   // Disable unnecessary statistics
   S.avgDevEnabled = false;
   S.bwmvEnabled = false;
   S.extremesEnabled = false;
   S.madEnabled = false;
   S.meanEnabled = false;
   S.medianEnabled = true;
   S.pbmvEnabled = false;
   S.varianceEnabled = false;

   // Enable low and high pixel rejection
   // ### BUG: lowRejectionEnabled must be set before highRejectionEnabled.
   S.lowRejectionEnabled = true;
   S.highRejectionEnabled = true;

   for ( let i = 0; i < N; ++i )
   {
      // Random image dimensions
      let W = wmin + (wmax - wmin)*Math.random();
      let H = hmin + (hmax - hmin)*Math.random();

      // Random image
      let I = new Image( W, H );
      I.initSampleIterator();
      do
         I.setSampleValue( Math.random() );
      while ( I.nextSample() );

      // Random rejection limits
      let l = Math.random();
      let h = Math.random();
      if ( h < l )
      {
         let t = l; l = h; h = t;
      }

      /*
       * Brute force median calculation.
       */
      let m = bruteForceMedian( I, l, h );
      let m1 = m[0]
      let n1 = m[1];
      if ( m1 < 0 )
      {
         --i;
         continue;
      }

      /*
       * ImageStatistics median value.
       */
      // ### BUG: When reusing an ImageStatistics object, rejectionLow must be
      //          reset to zero before setting new rejection limits, and
      //          rejectionHigh must be set before rejectionLow.
      S.rejectionLow = 0;
      S.rejectionHigh = h;
      S.rejectionLow = l;
      S.generate( I );

      let m2 = S.median;
      let n2 = S.count;

      /*
       * For odd image dimensions, compute also Image.median().
       *
       * For performance reasons, Image.median() always computes high median
       * values for images of even dimensions with more than 65536 pixels. See
       * the following PCL documentation entry for more information:
       *
       *    http://pixinsight.com/developer/pcl/doc/html/classpcl_1_1GenericImage.html#a38d72dd2ffbaa5051cc31f916d2b2398
       */
      let m3 = -1;
      if ( I.numberOfPixels & 1 )
      {
         I.rangeClippingEnabled = true;
         I.rangeClipLow = l;
         I.rangeClipHigh = h;
         m3 = I.median();
      }

      if ( m1 != m2 || m3 >= 0 && m1 != m3 )
         throw Error( format( "TestMedian(): Failed.<br>m1=%.15lf m2=%.15lf m3=%.15lf n1=%u n2=%u l=%.15lf h=%.15lf", m1, m2, m3, n1, n2, l, h ) );
   }

   console.writeln( "<end><cbr>TestMedian(): Success." );
}

console.show();
TestMedian( 100, 256, 256, 1024, 1024 );

Summarizing, the workarounds are the following:

- ImageStatistics.lowRejectionEnabled must always be set before ImageStatistics.highRejectionEnabled. For example:

   let S = new ImageStatistics;
   ...
   S.lowRejectionEnabled = true;
   S.highRejectionEnabled = true;

- When reusing an existing ImageStatistics object, rejectionLow must be reset to zero before setting new rejection limits, and rejectionHigh must be set before rejectionLow. For example:

   let S = new ImageStatistics;
   let I = new Image;
   ...
   S.lowRejectionEnabled = true;
   S.highRejectionEnabled = true;
   S.rejectionLow = 0.25;
   S.rejectionHigh = 0.85;
   S.generate( I );
   ...
   S.rejectionLow = 0;
   S.rejectionHigh = 0.9;
   S.rejectionLow = 0.1;
   S.generate( I );


With these precautions you can use ImageStatistics without problems, as shown with the above script. This will be fixed in the next version of the PixInsight Core application. Thank you for detecting and reporting this issue.

[georg.viehoever]

Thank you for fixing this. Small bug, big effect....
Georg

[Juan Conejero]

Yep, this happens. The smallest bugs are often the worst ones, both in their difficulty to be found, and in their potential for catastrophe. I suppose this is one of the things that make programming difficult