Author Topic: Noise evaluation problems  (Read 20263 times)

ruediger

  • Guest
Noise evaluation problems
« on: 2011 August 08 06:42:29 »
Hi all,
I have some problems with noise-evaluation based weighting and overall noise analysis of the final picture.

During image stacking, on a set of calibrated, noisy RAW files the MRS noise evaluation is failing on some, but not all pictures of the stack and falling back to k-sigma noise estimation. For me, there is no visible difference to the pictures. They are taken directly one after another with no changing light condition. The odd thing is, that the pictures with failing MRS noise evaluation get a much lower weight than the others. The weighting factors of the files with MRS succeeding is about 1,5x higher.
After completion of image integration, the MRS noise evaluation and reference SNR increments are computed relative to the first picture of the stack (with weight 1.000) and only meaningful, if either both time MRS failed or not.

For comparison of different integration parameters I therefore now use the "Script->Image Analysis->NoiseEvaluation" Tool. This tool never failed so far. Wouldn't it better to have a robust (at least user selectable) noise evalution method in image integration?

Also, just by coincidence, I found out, that a sub-pixel shift of a given picture has a large impact of the evaluated noise, at least when evaluated with the NoiseEvalution script. For example, shifting a picture by (x,y)=(0.5,0.5) gives on the R-channel:

Original
?R = 8.808e-003, N = 57793 (7.20%)
Subpixel shifted
?R = 6.847e-003, N = 36744 (4.58%)

Does this mean, that the lowest achievable noise also depends on the reference frame I pick for the star alignment? During the alignment process, pictures get (at least) shifted, so maybe one reference frame is better suited than another?

Regards,
 Rüdiger

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Noise evaluation problems
« Reply #1 on: 2011 August 11 01:02:43 »
Hi Rüdiger,

Could you upload one of those images where MRS noise evaluation is failing? I need it to understand and fix the problem. I guess I know how to solve this issue once and for all in the ImageIntegration tool, but currently I have no real image where this is happening.

Quote
I found out, that a sub-pixel shift of a given picture has a large impact of the evaluated noise

Shifting by noninteger pixel distances involves interpolation of pixel values, as happens with arbitrary (non-orthogonal) rotation and rescaling. Interpolation inevitably acts like a low-pass filtering process, so the final image is always smoothed to some degree (basically, the degree of smoothening depends on the applied interpolation algorithms). For this reason the noise evaluation algorithms will always give comparatively smaller estimates for interpolated images.

It's true that this can cause a known problem with ImageIntegration. When a set of images have been aligned (for example, with StarAlignment) taking one of them as the registration reference, and the whole set is being integrated, noise estimates will always be higher for the reference image, as it has not been interpolated as the rest of the set. This 'fools' ImageIntegration, in the sense that the registration reference image will always enter the integrated stack with an artificially smaller weight. We see two ways to handle this problem: (1) simply ignore it and avoid putting the registration reference image at the top of the list of integrated images, and (2) apply a slight (and controlled!) low-pass filtering procedure to the registration reference image, usually by decreasing bias for the first wavelet layer with the ATWT tool. (1) is a practical solution when you are integrating a large set of images, since in this case the relative loss of signal (caused by a 'wrongly' weighted image) can be negligible. (2) is for purists and for small image sets, where one wants to squeeze the last bit of signal from the data set. I'm not sure, however, if there's a real (practical) difference between (1) and (2).

Quote
Does this mean, that the lowest achievable noise also depends on the reference frame I pick for the star alignment?

Not at all. ImageIntegration will always estimate noise in the same way for each integrated image, irrespective of the image selected as reference for SNR increment evaluation (the first image in the integration list). For a more accurate evaluation of the achieved SNR increment after integration, you should refer to the average SNR increments, which take into account all of the integrated images, not just the first one.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline zvrastil

  • PixInsight Addict
  • ***
  • Posts: 179
    • Astrophotography
Re: Noise evaluation problems
« Reply #2 on: 2011 August 11 02:37:44 »
We see two ways to handle this problem: (1) simply ignore it and avoid putting the registration reference image at the top of the list of integrated images, and (2) apply a slight (and controlled!) low-pass filtering procedure to the registration reference image, usually by decreasing bias for the first wavelet layer with the ATWT tool. (1) is a practical solution when you are integrating a large set of images, since in this case the relative loss of signal (caused by a 'wrongly' weighted image) can be negligible. (2) is for purists and for small image sets, where one wants to squeeze the last bit of signal from the data set. I'm not sure, however, if there's a real (practical) difference between (1) and (2).

Hi Juan, just a thought - woudn't it be possible to use original (not-aligned) images for noise estimation? I'm not sure how to make link between original and aligned image though.

regards, Zbynek

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Noise evaluation problems
« Reply #3 on: 2011 August 11 02:40:53 »
...It's true that this can cause a known problem with ImageIntegration. When a set of images have been aligned (for example, with StarAlignment) taking one of them as the registration reference, and the whole set is being integrated, noise estimates will always be higher for the reference image, as it has not been interpolated as the rest of the set. This 'fools' ImageIntegration, in the sense that the registration reference image will always enter the integrated stack with an artificially smaller weight. We see two ways to handle this problem: ...

I think this is probably a problem for any frame that is -by chance- only shifted by on integer number of pixels (0,1,2,..) in x or y direction during StarAlignment, without any rotation. There are probably also certain rotations that might have unusual effects on the noise estimate, eg. 30, 45, 90 degrees. I wonder if this problem should be handled by StarAlignment (and not image integration) by applying an equivalent smoothing to *all* images, including the reference frame and those that have "unfortunate" shifts. Or it could be handled by adding a suitable FITS attribute that is evaluated by integration (which of course would then require the use of PI for image integration, since this is the only software that evaluates this attribute  >:D)


Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Noise evaluation problems
« Reply #4 on: 2011 August 11 04:21:23 »
Hi Zbynek,

Quote
woudn't it be possible to use original (not-aligned) images for noise estimation?

Yes, it is possible. It could be done with a dedicated tool (easy to write since all the routines are already available) and with the help of custom FITS keywords to store noise estimates, as Georg has suggested. However, would this be really correct? After all, what we are integrating is not the original images, but the registered --and hence interpolated-- images. Once the images have been interpolated (translated, rotated, etc.), the distributions of signal and noise no longer correspond to the original frames. Interpolation correlates the values of adjacent pixels causing a smoothing effect that is a function of the applied transformations and interpolation algorithms.
 
The goal of noise evaluation in ImageIntegration is to maximize the SNR gain in the final result with respect to the integrated data set. Evaluating noise in the images before alignment would provide estimates that don't correspond to the actual integrated data. In this sense, it would have the same problem as using exposure times as image weights.

In my opinion, accurate image weights can only be derived from direct noise estimates. For this to work correctly, the applied noise evaluation algorithms must be very accurate and robust. For this reason I invest a lot of work in improving our noise evaluation routines, and reports of failure in these routines (as this thread) draw my immediate attention.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Noise evaluation problems
« Reply #5 on: 2011 August 11 04:34:52 »
....It could be done with a dedicated tool (easy to write since all the routines are already available) and with the help of custom FITS keywords to store noise estimates, as Georg has suggested. However, would this be really correct? After all, what we are integrating is not the original images, but the registered --and hence interpolated-- images. Once the images have been interpolated (translated, rotated, etc.), the distributions of signal and noise no longer correspond to the original frames. Interpolation correlates the values of adjacent pixels causing a smoothing effect that is a function of the applied transformations and interpolation algorithms.
...

I am not sure that using the noise estimates of the aligned images for integration is correct. In essence, the noise estimates are used to estimate the information contents of the source images. In my opinion, there is no way that alignment or smoothing increase the amount of information - but this is what the changed noise estimates suggest to image integration (I hope I got that correctly !?! ).

Georg
 
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Noise evaluation problems
« Reply #6 on: 2011 August 11 04:35:23 »
Quote
I think this is probably a problem for any frame that is -by chance- only shifted by on integer number of pixels (0,1,2,..) in x or y direction during StarAlignment, without any rotation. There are probably also certain rotations that might have unusual effects on the noise estimate, eg. 30, 45, 90 degrees.

While this is certainly possible, I think (and hope!) the probability is sufficiently low as to allow us to live without thinking on that ... or if you are so purist then you can align all the images twice or thrice using different reference frames :)
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Noise evaluation problems
« Reply #7 on: 2011 August 11 04:58:04 »
Quote
there is no way that alignment or smoothing increase the amount of information

Of course not; we are magical but usually can't do miracles ;)

Quote
In essence, the noise estimates are used to estimate the information contents of the source images.

Nope. Noise evaluation does not try to evaluate information contents; that's a much more complex and difficult task. It just tries to make a relative measurement of the intensity of the noise in the image. Roughly, this is done through classification: for each image, the algorithm divides the whole set of pixels into two disjoint sets: noise pixels and signal pixels. Then the noise estimate is derived from the variance of the set of noise pixels.

The goal of noise estimation as an image weighing technique for image integration is generating the best possible result in terms of SNR improvement from the set of integrated data. Once the images have been interpolated, what we have are no longer the original noise and signal distributions, but other (altered, mixed, correlated) distributions which are what we have to deal with.

An interesting experiment would be comparing the results of two identical integrations: one made using noise estimates from the original images and a second with estimates from the interpolated images. The winner would be the integration whose result achieves the highest SNR improvement with respect the the average noise estimate. Hmm, this can be done with the current tools (noise evaluation script, custom FITS keywords, ...).
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Noise evaluation problems
« Reply #8 on: 2011 August 11 05:07:44 »
...The goal of noise estimation as an image weighing technique for image integration is generating the best possible result in terms of SNR improvement from the set of integrated data.

If I remember correctly, the images with the lower noise estimates will get a higher weight during integration. This is kind of counter-intuitive for me, since the information contents in those aligned images cannot be higher than in the unaligned images. It  puzzles me :-\

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Noise evaluation problems
« Reply #9 on: 2011 August 11 05:34:25 »
Don't think in terms of information, but just in terms of uncertainty (noise versus signal). A very noisy image can have lots of information, and a perfectly noiseless image can be perfectly useless :)

The interpolated images have lower noise estimates just because the variance of the set of noise pixels is smaller, as a result of low-pass filtering. The only problem comes from the fact that one of the images --the alignment reference-- has not been interpolated and the rest has been. This leads to an 'unfair' classification of the reference frame as 'poor', which in turn leads to a relative data loss. This problem is of very little practical importance, however.

Nevertheless, I'm going to carry out some tests as I described above (integration using noise estimates before and after interpolation as image weights - nice work to be on vacation ::)). If 'prior estimates' give better results then we'll have to talk about a new image weighting strategy :)
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

ruediger

  • Guest
Re: Noise evaluation problems
« Reply #10 on: 2011 August 11 05:55:38 »
Hi Juan,
here are two files to start testing (but I can provide you with as many as you need). These two are from my Garradd/Messier 15 stack (still experimenting on that one!):
ftp://ftp.pictstore.de/3330664c59d52a6b73f3a94fe4493d91/debayer_IMG_5486_c_r.fit
ftp://ftp.pictstore.de/3330664c59d52a6b73f3a94fe4493d91/debayer_IMG_5487_c_r.fit

You'll see, noise estimation failed on R and B-channel on IMG_5486. The following image IMG_5487 has therefore heigher weights on R and B. In these situations, changing the Image Integration weighting from "noise evaluation" to "don't care (all weights=1)" improves the SNR in the final result.
Code: [Select]
F:/Garradd/debayer_IMG_5486_c_r.fit
Reading FITS: 32-bit floating point, 3 channel(s), 4770x3178 pixels: 100%
MRS noise evaluation: |
** Warning: No convergence in MRS noise estimation routine - using k-sigma noise estimate.
** Warning: No convergence in MRS noise estimation routine - using k-sigma noise estimate.
Scale factors   :   1.05222  1.04846  1.09050
Zero offset     :  +2.354192e-003 +1.887822e-003 +1.664470e-003
Noise estimates :  4.309e-004 3.350e-004 1.877e-004
Weight          :     0.99948    0.81471    1.03138

F:/Garradd/debayer_IMG_5487_c_r.fit
Reading FITS: 32-bit floating point, 3 channel(s), 4770x3178 pixels: 100%
MRS noise evaluation: done.
Scale factors   :   1.06546  1.08421  1.10975
Zero offset     :  +2.357333e-003 +1.898855e-003 +1.658831e-003
Noise estimates :  3.320e-004 3.139e-004 1.441e-004
Weight          :     1.64150    0.86764    1.69073

Pictures were taken with Canon EOS 500Da at ISO800, 3min exposure. The EXIF recorded camera temperature was about 33°C. If you need more information or more files, please let me now.

Regards
 Rüdiger

ruediger

  • Guest
Re: Noise evaluation problems
« Reply #11 on: 2011 August 11 06:10:47 »
This leads to an 'unfair' classification of the reference frame as 'poor', which in turn leads to a relative data loss. This problem is of very little practical importance, however.
For a DSLR user with a similar shooting workflow like I have, the first picture in a stack is often really poor, because it was probably taken directly after live view focussing and framing. Live view usage pretty quickly heats up the camera by 10° degrees with a similar impact to noise and weight.

In these (rare) cases where the MRS noise evaluations works an all pictures of a stack, the computed weights often directly corresponds to the EXIF recorded internal camera temperature.

So at least for me it doesn't matter to much if the first picture gets an "unfair" classification. It deserves it ;)

Rüdiger

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Noise evaluation problems
« Reply #12 on: 2011 August 11 12:51:52 »
Quote
the computed weights often directly corresponds to the EXIF recorded internal camera temperature.

Nice experimental verification 8)

I think I have this issue definitely ironed out. Thank you for the images; I am downloading them right now. I'll let you know when I have news ;)
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Noise evaluation problems
« Reply #13 on: 2011 August 12 05:11:45 »
Hi Rüdiger,

I think the problem is now fixed. Please try the following modified version of the noise evaluation script with your images:

Code: [Select]
/*
 * NoiseEvaluation v1.1
 *
 * Automatic evaluation of Gaussian noise by the iterative multiresolution
 * support and k-sigma thresholding methods.
 *
 * Copyright (C) 2006-2011 Pleiades Astrophoto S.L.
 * Written by Juan Conejero (PTeam)
 *
 * References:
 *
 * - Jean-Luc Starck, Fionn Murtagh, Automatic Noise Estimation from the
 *   Multiresolution Support, Publications of the Royal Astronomical Society
 *   of the Pacific, vol. 110, February 1998, pp. 193-199.
 *
 * - J.L. Starck, F. Murtagh, Astronomical Image and Data Analysis, Springer,
 *   1st ed., 2002, pp. 37-38.
*/

#feature-id    Image Analysis > NoiseEvaluation v1.1

#feature-info \
Automatic estimation of Gaussian noise by the iterative multiresolution \
support and k-sigma thresholding algorithms.<br>\
<br>\
Written by Juan Conejero (PTeam).<br>\
<br>\
References:<br>\
<br>\
Jean-Luc Starck, Fionn Murtagh, <i>Automatic Noise Estimation from the \
Multiresolution Support</i>, Publications of the Royal Astronomical Society \
of the Pacific, vol. 110, February 1998, pp. 193-199.<br>\
<br>\
J.L. Starck, F. Murtagh, <i>Astronomical Image and Data Analysis</i>, Springer, \
1st ed., 2002, pp. 37-38.<br>\
<br>\
Copyright (c) 2006-2011 Pleiades Astrophoto S.L.

/**
 * Estimation of the standard deviation of the noise, assuming a Gaussian
 * noise distribution.
 *
 * - Use MRS noise evaluation when the algorithm converges for 4 >= J >= 2
 *
 * - Use k-sigma noise evaluation when either MRS doesn't converge or the
 *   length of the noise pixels set is below a 1% of the image area.
 *
 * - Automatically iterate to find the highest layer where noise can be
 *   successfully evaluated, in the [1,3] range.
 */
function NoiseEvaluation( img )
{
   var a, n = 4, m = 0.01*img.bounds.area;
   for ( ;; )
   {
      a = img.noiseMRS( n );
      if ( a[1] >= m )
         break;
      if ( --n == 1 )
      {
         console.writeln( "<end><cbr>** Warning: No convergence in MRS noise evaluation routine - using k-sigma noise estimate." );
         a = img.noiseKSigma();
         break;
      }
   }
   this.sigma = a[0]; // estimated stddev of Gaussian noise
   this.count = a[1]; // number of pixels in the noise pixels set
   this.layers = n;   // number of layers used for noise evaluation
}

function main()
{
   // Get access to the current active image window.
   var window = ImageWindow.activeWindow;
   if ( window.isNull )
      throw new Error( "No active image" );

   console.show();
   console.writeln( "<end><cbr><br><b>" + window.currentView.fullId + "</b>" );
   console.writeln( "Calculating noise standard deviation..." );
   console.flush();

   console.abortEnabled = true;

   var img = window.currentView.image;
   for ( var c = 0; c < img.numberOfChannels; ++c )
   {
      console.writeln( "<end><cbr><br>* Channel #", c );
      console.flush();
      img.selectedChannel = c;
      var E = new NoiseEvaluation( img );
      console.writeln( format( "&sigma;<sub>%c</sub> = %.3e, N = %u (%.2f%%), J = %d",
                               img.isColor ? "RGB"[c] : 'K',
                               E.sigma, E.count, 100*E.count/img.bounds.area, E.layers ) );
      console.flush();
   }
}

main();

Basically, the problem is that for some images, the noise can only be evaluated in the first two wavelet layers (scales of one and two pixels) with the MRS algorithm. This usually happens with images plenty of significant structures at many different scales. It also happens with images where the noise has a relatively large characteristic scale, as is the case with your images. Until now, my MRS implementation tried to evaluate noise within the first four or three layers, and this was seldom causing convergence problems. Now the idea is to let the algorithm converge at the highest possible layer, from layer 3 (8 pixels) to layer 1 (2 pixels), iterating as necessary.

The implementation is much more robust with this simple modification. Unless you tell me that you see no significant improvement with the above script (which I doubt), I'll incorporate this modified implementation to ImageIntegration.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

ruediger

  • Guest
Re: Noise evaluation problems
« Reply #14 on: 2011 August 13 01:35:13 »
I think the problem is now fixed. Please try the following modified version of the noise evaluation script with your images:
[...]
The implementation is much more robust with this simple modification. Unless you tell me that you see no significant improvement with the above script (which I doubt), I'll incorporate this modified implementation to ImageIntegration.
Hi Juan,
thanks so much for working on this. I tested the new script so far on about 100 images of two different stacks. The results are better (one stack is now flawless), but I still found lots of images where noise can't be evaluated on R and B -- didn't find any problem on the G-channel.

I don't have the upload bandwidth at home for uploading another 170MB of FITs-Files, but maybe you can check one simple observation on the files I already uploaded. From the same comet/M15-stack many files are now working, but at some point the noise evaluation starts to fail, e.g. on IMG_5537:
Code: [Select]
debayer_IMG_5537_c_r
Calculating noise standard deviation...

* Channel #0
** Warning: No convergence in MRS noise evaluation routine - using k-sigma noise estimate.
?R = 4.436e-004, N = 14696960 (96.95%), J = 1

* Channel #1
?G = 2.950e-004, N = 7772776 (51.27%), J = 3

* Channel #2
** Warning: No convergence in MRS noise evaluation routine - using k-sigma noise estimate.
?B = 1.924e-004, N = 14702471 (96.99%), J = 1
The major difference of this picture to the first ones of the stack is the larger amount of shift during alignment. It has a lot of black at the top and left, because it was taken hours later than the reference frame and I can't reach a better tracking accuracy of the Astrotrac (about 1 arcsec/min). The Astrotrac also needs to be rewinded and repositioned after 2 hours of shooting.

To play with the picture, I cloned it and cropped away a little bit of black space at the top. After this, the noise evaluation works:
Code: [Select]
debayer_IMG_5537_c_r_clone
Calculating noise standard deviation...

* Channel #0
?R = 3.192e-004, N = 6234283 (42.15%), J = 2

* Channel #1
?G = 3.077e-004, N = 4516728 (30.54%), J = 4

* Channel #2
?B = 1.384e-004, N = 6250576 (42.26%), J = 2

When thinking about the implications, I now fear that I have to completely change my workflow and include cropping to final image dimensions before stacking:
1: noise evaluation obviously takes the black areas into account, which are meaningless for the final result (see the changing green noise value)
2: sigma-based rejection algorithms need a normalized background value. Is the black area disturbing the normalization process?

Before completely switching to PixInsight, I used IRIS. Within IRIS, my workflow included the cropping of the aligned pictures before stacking. This was recommended and needed for the background normalization process which was carried out as another separate process before stacking.

Do you recommend to crop before stacking in PixInsight?

But the black areas can't be the only reason to make MRS fail. I also found some completely unmodified RAW-Files:
Code: [Select]
IMG_5466
Calculating noise standard deviation...

* Channel #0
** Warning: No convergence in MRS noise evaluation routine - using k-sigma noise estimate.
?R = 2.032e-003, N = 14430712 (95.20%), J = 1

* Channel #1
?G = 2.232e-003, N = 5035941 (33.22%), J = 4

* Channel #2
** Warning: No convergence in MRS noise evaluation routine - using k-sigma noise estimate.
?B = 2.628e-003, N = 14416396 (95.10%), J = 1

Odd enough, the calibrated and registered version of this RAW-file works.

Hopefully I didn't spoil the weekend with this post  O:)

Might a cop-out (giving user the option to use k-sigma noise estimation for weighting) be a simple solution?

Regards,
 Rüdiger
« Last Edit: 2011 August 13 01:51:15 by ruediger »