Author Topic: Removing Noise from an Image  (Read 2855 times)

Offline BobinBend

  • Member
  • *
  • Posts: 51
Re: Removing Noise from an Image
« Reply #30 on: 2019 July 03 13:07:02 »
pfile,

Thanks for looking at this. As you recommended I opened two subs, one from calibration and one from registration folders (the same sub files) and ran the NoiseEvaluation script on each one. I don't know how to read the results but I've attached the process console log for each.
Thanks for your help,
Bob

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: Removing Noise from an Image
« Reply #31 on: 2019 July 03 14:54:46 »
you are comparing a bayered image (_c) with a debayered image (c_d_r). these can not be correlated with one another... you need to compare the calibrated, debayered, unregistered image (c_d) with the calibrated, debayered and registered image (c_d_r)

rob

Offline BobinBend

  • Member
  • *
  • Posts: 51
Re: Removing Noise from an Image
« Reply #32 on: 2019 July 03 16:01:06 »
pfile,
Ok lets try it again...
Here's the two images processed with the NoiseEvaluation script. The channel percentages seem to look about the same.
Bob

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: Removing Noise from an Image
« Reply #33 on: 2019 July 03 16:38:42 »
well the % is just the percentage of pixels that were analyzed as contributing to noise. pixels with strong signal are excluded so this tool tends to measure the noise in the background, unless it is restricted to high SNR areas using a preview.

the leading number is the standard deviation of the pixels under consideration and smaller numbers are better. as you can see, the registered image actually has lower noise than the unregistered image. this is because registration involves interpolation, which involves averaging neighboring pixels together... and that boosts the SNR of a given pixel.

anyway, armed with this tool you should be able to go thru some subs from each side of the meridian and find out if one side truly has worse SNR than the other. or, you can make two integrations from equal numbers of east subs and west subs, and see how the noise looks in each stack. however, i'm not sure that this script is going to tell you if there are a bunch of hot pixels hanging around, which actually seems to be the main problem in your image.

anyway i feel that all this "upside down image" stuff is a diversion. i think there is a very narrow set of circumstances where PI could read in the calibration frames upside-down with respect to the light frames, and since you are exclusively using CR2 files with 'pure raw' this should not ever happen. note that just because the image is sometimes formed upside-down on the sensor does not imply that the sensor data has been read in upside-down.

i think you first have to definitively determine if the subs from one side of the meridian are truly worse than the other side, and if not, then start looking at why there are so many hot red pixels in your images. it could be simply down to dark scaling during calibration; any time a master dark is scaled, you risk under-correction of hot pixels. of course you can clean this up with CosmeticCorrection. or it could have to do with the camera being hotter for later subs than it was for earlier subs which increases the dark current (and the associated dark noise) in the sensor.

rob

Offline BobinBend

  • Member
  • *
  • Posts: 51
Re: Removing Noise from an Image
« Reply #34 on: 2019 July 03 17:10:36 »
Rob,

Thanks for your input and suggestions. A couple of discoveries have been uncovered from all this;

1. I did integrate one side of the data by itself and the results were very good with a much lower amount of noise. I didn't integrate the other M.F. side of the data but I will.

2. If both sides of subs integrate with low noise then I believe the 'noise' from the BatchProcessing script integrating all the subs is really not camera noise but instead an artifact generated from post processing software. Agreed?

3. I'm starting to question the performance of the BatchProcessing script. Could this script be the source of this artifacts? On the other hand I combined all these light subs using DSS and the results were the same with noise in the final stacked image.

4. Can I integrate these two sides of subs separately and then integrate the two resultant images into one?

5. I'll do some more testing using your suggestions. I'll also integrate the subs using the ImageIntegration process to see what I get.

Thanks for your help,
Bob

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: Removing Noise from an Image
« Reply #35 on: 2019 July 03 17:21:26 »
on #2 i can't say exactly, i mean as i mentioned it is possible subs from later in the evening are worse, either due to camera noise increasing, or maybe shooting into a light dome on that side of the meridian, or even clouds that moved in over time...

on #3 all BPP does is to call the individual processes in turn, handing all the file naming and propagating the name of the reference frame along as it morphs (-> calibrated -> debayered, etc.) obviously you have more control over everything if you do it manually but the BPP defaults for StarAlignment for instance are pretty sane. it's really just the ImageIntegration step that juan recommends that you do by hand so you can tweak the rejection method and parameters, since every stack is different...

#4 yes you can just integrate the stacks, but since ImageIntegration wants a minimum of 3 input files, what you have to do is add both files twice. the result will be mathematically the same as though you were able to integrate just the two files. normally when you integrate stacks together you turn off pixel rejection since 1) each stack should, if comprised of enough input images, already be free of outlier pixels and 2) generally you only have a handful of stacks and so pixel rejection is difficult or impossible when you don't have enough input images.

rob

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Removing Noise from an Image
« Reply #36 on: 2019 July 04 02:10:25 »
As a general rule, comparisons of unscaled noise estimates are meaningless. To compare noise standard deviations, you must scale them with statistical scale (or dispersion) estimates. In this way you'll be comparing compatible statistical descriptors.

You can scale the noise estimates easily using console commands in PixInsight. However, to simplify these operations, here is a modified version of the NoiseEvaluation script (which we call ScaledNoiseEvaluation) that you should use to compare noise estimates calculated for different images:

Code: [Select]
/**
 * Estimation of the standard deviation of the noise, assuming a Gaussian
 * noise distribution.
 *
 * - Use MRS noise evaluation when the algorithm converges for 4 >= J >= 2
 *
 * - Use k-sigma noise evaluation when either MRS doesn't converge or the
 *   length of the noise pixels set is below a 1% of the image area.
 *
 * - Automatically iterate to find the highest layer where noise can be
 *   successfully evaluated, in the [1,3] range.
 *
 * Returned noise estimates are scaled by the Sn robust scale estimator of
 * Rousseeuw and Croux.
 */
function ScaledNoiseEvaluation( image )
{
   let scale = image.Sn();
   if ( 1 + scale == 1 )
      throw Error( "Zero or insignificant data." );
   
   let a, n = 4, m = 0.01*image.selectedRect.area;
   for ( ;; )
   {
      a = image.noiseMRS( n );
      if ( a[1] >= m )
         break;
      if ( --n == 1 )
      {
         console.writeln( "<end><cbr>** Warning: No convergence in MRS noise evaluation routine - using k-sigma noise estimate." );
         a = image.noiseKSigma();
         break;
      }
   }
   this.sigma = a[0]/scale; // estimated scaled stddev of Gaussian noise
   this.count = a[1]; // number of pixels in the noisy pixels set
   this.layers = n;   // number of layers used for noise evaluation
}

function main()
{
   let window = ImageWindow.activeWindow;
   if ( window.isNull )
      throw new Error( "No active image" );

   console.show();
   console.writeln( "<end><cbr><br><b>" + window.currentView.fullId + "</b>" );
   console.writeln( "Calculating scaled noise standard deviation..." );
   console.flush();

   console.abortEnabled = true;

   let image = window.currentView.image;
   console.writeln( "<end><cbr><br>Ch |   noise   |  count(%) | layers |" );
   console.writeln(               "---+-----------+-----------+--------+" );
   for ( let c = 0; c < image.numberOfChannels; ++c )
   {
      console.flush();
      image.selectedChannel = c;
      let E = new ScaledNoiseEvaluation( image );
      console.writeln( format( "%2d | <b>%.3e</b> |  %6.2f   |    %d   |", c, E.sigma, 100*E.count/image.selectedRect.area, E.layers ) );
      console.flush();
   }
   console.writeln(               "---+-----------+-----------+--------+" );
}

main();


Juan Conejero
PixInsight Development Team
http://pixinsight.com/