Random Banding

[georg.viehoever]

Hi,

I conducted an experiment where I used PI to create a 1000x1000 random picture, and then I ran the CanonBandingReduction script on them. The parameters can seen in the screenshot. Also in the screenshot are 3 separately generated images that have been treated this way. It appears to me that the resulting banding structure is similar in all 3 of them. Either the banding script has a problem, or the random number generator of PI.

Any comments?

Georg

[Niall Saunders]

Hi Georg,

My immediate thought would have been to run the 'noise' image through an FFT. If there truly WAS any patterning in the 'random' noise algorithm, then I would expect to be able to detect this post-FFT.

And, in the (current) absence of an FFT process in PI, try running the PI-generated 'random' data through a third-party algorithm. I will try this once I can get my TeamViewer link running back to my home PC, but not from this 'cheap-ass' hotel, with no Internet in the rooms!! (my current preferred choice would be AIP4WIN)

Cheers,

[Juan Conejero]

Hi Georg,

For complete information about the random number generator used by PixInsight/PCL (and hence by PJSR), see:

http://pixinsight.com/developer/pcl/doc/html/classpcl_1_1RandomNumberGenerator.html

As you see, PI uses an adaptation of the Marsenne Twister algorithm by Makoto Matsumoto  and Takuji Nishimura. In particular, I implemented a variation of the SIMD-oriented Fast Mersenne Twister algorithm (SFMT) that has a period of 2¹⁹⁹³⁷-1.

I personally can't see a significant (apparent) correlation between the three images that you show on your screenshot. To substantiate a hypothesis of correlation between different results of the random generator, a larger set of tests should be performed. I like Niall's idea about using a FFT-based procedure. Currently the PJSR offers everything needed to conduct that test: Image.FFT() and Image.phaseCorrelationMatrix(). If you find a significant peak in a phase correlation matrix computed for the DFTs of two different random images, then we certainly have a problem.

[georg.viehoever]

The Marsenne Twister algorithm certainly is a good algorithm. If implemented properly, this should not be the cause of the issues.

I have not yet tried Niall's FFT proposal, but I had a look at the "fixImage" that the script generates based on the line wise medians of the random images. They have a normalized histogram as shown in the screenshot, which shows significantly lower "fix"-values for green. For other random images, I have seen larger fix values for green than for RB. This does not appear to the characteristic of a truly random process. To reproduce: Generate 1000x1000 random images as shown in my original post, and apply the modified script attached below (it does not produce properly corrected images due to the normalize() statement in line 316. Then display the histogram of the FixImage.

Strange, isnt it...

Georg

[georg.viehoever]

Juan,

below my naive try at FFT (I am not an expert for this), it fails with *** Error [000]: C:/PCLGeorg/scripts/testfft.js, line 5: Image.FFT(): invalid sample type: must be a complex image" when applied to an RGB image. Can you give me a hint?

Georg

Code: [Select]

     var window = ImageWindow.activeWindow;
     var targetView=window.currentView;
     var targetImage=targetView.image;
     targetView.beginProcess();
     targetImage.FFT(false);
     targetView.endProcess();


[georg.viehoever]

Hi,

attached a reduced version of the Canon Banding script (just 60 lines) that executes on the current image. The histogram of the "FixImage" shows the same effect, sometimes on the green, sometimes on the red channel. I think something is wrong here...

Georg

[NKV]

For complete information about the random number generator used by PixInsight/PCL (and hence by PJSR), see:

http://pixinsight.com/pcl/doc/html/classpcl_1_1RandomNumberGenerator.html

HTTP Error 404 

[Juan Conejero]

HTTP Error 404

Fixed! The correct URL is now:
http://pixinsight.com/developer/pcl/doc/html/classpcl_1_1RandomNumberGenerator.html

I'll be fixing many broken links on the forum during the next days.

BTW, it seems DNS propagation has reached your country. Great!

[georg.viehoever]

Hi Juan,

attached a script that will generate 2 FixImages for you: One from a random float image, one from a double (64 bit) image. In both cases, the histograms of the FixImage show the strange tendency to have different histograms for red, green and blue (see attached screenshot). This happens always, so it is not a random effect. The histogram of 2 colors match almost perfectly, while the 3rd color show a considerable skew. From my point of view this hints at a problem with the random number generator or in the image operations.

Please let me know how I can help to track down the issue.

Georg
 

[Juan Conejero]

Hi Georg,

I may be wrong (one always can be wrong!), but I am confident that PCL's random number generator is correctly implemented. I'll try to write a script to implement a battery of statistical tests to demonstrate its performance.

I'll write also a small script to compute a cross correlation matrix for two random images.

Just let me finish with all this migration stuff

[georg.viehoever]

Hi Juan,

the problem may not be the random number generator, it could also be the something in the arithmetic operations (e.g. incorrect rounding, loss of a few bits, ...). These can also be the cause for the strange histograms.

Georg

[georg.viehoever]

Juan,

I am very glad to say that this was my bug ! In line 3 of my test script test3.js (and at the corresponding place in the CanonBanding Script), a  "targetImage.resetSelections();" was missing. The resulted in computing the image statistics for the whole image being computed using only the last row of the image... Sorry for the wrong alarm 

Georg

[Niall Saunders]

Hi Georg, is this a case of "Physician, heal thyself"  

Cheers,

[Juan Conejero]

Phew!  8)