Author Topic: New script for denoising linear monochannel images  (Read 45582 times)

Offline blinky

  • Member
  • *
  • Posts: 65
    • View Profile
Re: New script for denoising linear monochannel images
« Reply #150 on: 2019 October 07 14:08:49 »
Only thing is, if I load the variance scale from the stack, then the denoise does not work, the before & after images are the same, leave it at 1 and its fine though

Offline mschuster

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1058
    • View Profile
Re: New script for denoising linear monochannel images
« Reply #151 on: 2019 October 18 20:37:05 »
Only thing is, if I load the variance scale from the stack, then the denoise does not work, the before & after images are the same, leave it at 1 and its fine though

If the script appears not to be working, check the "Method noise" value in the process console log. If non-zero, then the script is in fact doing something. Depending on the image, the current STF stretch, and especially for very deep stacks where the noise level has been driven down significantly, the difference can be hard to see or distinguish from typical background variations due to unresolved stars, galaxies, and nebulae.

Regarding the loaded variance scale, the loaded value is an estimate that unfortunately does not completely account for the image normalization and weighting operations done by ImageIntegration. Feel free to adjust this value. As a double check, you can redo your processing using Nearest Neighbor interpolation in StarAlignment, and both output normalization and image weighting turned off in ImageIntegration. Doing so basically removes all of PI's pixel level modifications and hence best preserves image noise statistics, and so makes the script's job more straightforward.

Offline mschuster

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1058
    • View Profile
Re: New script for denoising linear monochannel images
« Reply #152 on: 2019 November 11 08:11:58 »
MureDenoise 1.23 should appear soon as an auto update. Change log:

  • Addressed a PixInsight 1.8.8 metadata compatibility issue involving load variance scale.
  • Removed support for load variance scale from a process log file.
  • Improved the handling of ill-conditioned linear systems that may arise when optimizing the denoising process.
  • Included an optional gradient classifier to provide a minor increase in the adaptivity and accuracy of the denoising process, but also increases processing time by about 50%.

Include gradient classifier is a new checkbox in the dialog. At least on my projects, its denoising improvements are minor and typically visible only with boosted STF stretch. Two examples are shown below. The differences may be more visible if you view them on your computer in a darkened room.

Example 1: 20 hour integration


Example 1: Include gradient classifier disabled


Example 1: Include gradient classifier enabled


Example 2: 15 hour integration


Example 2: Include gradient classifier disabled


Example 2: Include gradient classifier enabled


Offline mschuster

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1058
    • View Profile
Re: New script for denoising linear monochannel images
« Reply #153 on: 2019 November 11 19:55:23 »
Here is another example of the Include gradient classifier option. A 20 hour Ha integration of a higher contrast target, for which the gradient classifier appears to be effective. The crops are shown at boosted stretch.

Original 20 hour integration, showing the crop location


Crop of original


Include gradient classifier disabled


Include gradient classifier enabled

Offline mschuster

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1058
    • View Profile
Re: New script for denoising linear monochannel images
« Reply #154 on: 2019 November 19 07:39:10 »
MureDenoise 1.24 should appear soon as an auto update.

MureDenoise 1.24 uses a new scheme to account for ImageIntegration's normalization, weighting, and output rescaling operations. Rather than load Variance scale, you enable the option Use image metadata. When enabled, the new scheme estimates and applies a linear image transformation for improved accuracy in the accounting.

When Use image metadata is enabled, Variance scale remains unchanged. Start with a value of 1, and then as in previous versions, you may increase or decrease its value to control the amount of denoising. Combination count will be loaded.

The following ImageIntegration settings are supported: Image Integration > Combination: Average. Image Integration > Normalization: No normalization, Additive, or Additive with scaling. Image Integration > Weights: all values. The script is not smart enough to handle other settings, it will apply no transformation in these cases.

Thank you Juan for providing the metadata that makes this improvement possible.

« Last Edit: 2019 November 19 20:05:56 by mschuster »

Offline mschuster

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1058
    • View Profile
Re: New script for denoising linear monochannel images
« Reply #155 on: 2019 November 20 17:29:44 »
Here are two tips now how to use the latest version of MureDenoise. The UI has changed (which can be annoying sometimes), so you might find this helpful:

1) The normal case, do this:
  • Select the image
  • Check Use image metadata
  • Verify that Combination count is set correctly
  • Set Interpolation method correctly
  • Set Variance scale to 1
  • Denoise
Increase or decrease Variance scale to modify the amount of denoising.

For increased accuracy, but at a price of increased processing time, do one or both of these:
  • Check Include gradient classifier
  • Increase Cycle-spin count
2) A special case where you used Load variance scale... in a previous version of the script and want to recreate that result as close as possible for some reason:
  • Do 1) above, the normal case
  • Look in the process log for the line Use image metadata: true [scale: <scale number>, offset: <offset number>]
  • Uncheck Use image metadata
  • Uncheck Include gradient classifier
  • Set Variance scale to the square of the scale number (eg, if scale number is 1.02, set Variance scale to 1.02 * 1.02 = 1.04)
  • Denoise
Note that the result of 2) will not be as accurate as the result of 1) due to the new version's improvements.
« Last Edit: 2019 November 20 17:40:00 by mschuster »

Offline mschuster

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1058
    • View Profile
Re: New script for denoising linear monochannel images
« Reply #156 on: 2019 November 23 08:43:35 »
I found a bug in the new version 1.24 Use image metadata option, a typo in a variable name.

I apologize for the problem. I will release a new version asap.

Mike

Offline mschuster

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1058
    • View Profile
Re: New script for denoising linear monochannel images
« Reply #157 on: 2019 November 23 19:10:36 »
MureDenoise 1.25 should appear soon as an auto update.

Fixed bug in Use image metadata due to a typo.

Increased maximum Cycle-spin count to 48.

The text Use image metadata [none] indicates either unavailable image metadata or unsupported ImageIntegration options.
« Last Edit: 2019 November 23 19:24:46 by mschuster »

Offline Sedat

  • Member
  • *
  • Posts: 50
  • Newbie
    • View Profile
Re: New script for denoising linear monochannel images
« Reply #158 on: 2019 November 24 04:45:55 »
Hello Mike,

Thanks for the update.

Just wanted to make sure one point; are we still entering the detector data like the gain and the gaussian noise, or is the script getting those from the image somehow ?


Cheers

Sedat
Sedat

Offline mschuster

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1058
    • View Profile
Re: New script for denoising linear monochannel images
« Reply #159 on: 2019 November 24 07:22:14 »
Hello Sedat,

Yes, you still need to enter detector Gain, Gaussian noise, and Offset.

Here are tips on how to find these numbers:

1) Camera manufacturer data: Manufacturers sometimes provide gain in e-/DN and read noise in e-. Use this gain value and for Gaussian noise use read noise divided by gain. For example, if gain is 0.5 e-/DN and read noise is 8 e-, use 8 / 0.5 = 16 DN for Gaussian noise. Set Offset to 0 DN (because the image to be denoised has been calibrated). Note that some cameras allow you to switch between low and high gain, or change the amount of binning, and manufacturers may not provide data covering all of these cases. In situations like this you may need to follow 3) below.

2) Camera manufacturer gain parameter charts: Manufacturers sometimes provide gain parameter charts, here the gain parameter is not actually detector gain but rather a camera control parameter. You have to read gain in e-/DN and read noise in e- from charts. Then you calculate Gaussian noise in DN as in 1) above. Set Offset to 0 DN. See this post for an example.

3) Scripts Image Analysis > FlatSNREstimator and DarkBiasNoiseEstimator: These two scripts provide Gain in e- and Gaussian noise in DN using calibration frames you supply. You need two bias frames and two flat frames. DarkBiasNoiseEstimator provides a value which it calls Temporal noise in DN. Use this value for Gaussian noise. DarkBiasNoiseEstimator also provides Offset, but don't use this value, use 0 DN instead as above. FlatSNREstimator provides Gain, use this and ignore the other provided values.

For long exposures where dark current is more of a factor, you can supply to two dark frames to DarkBiasNoiseEstimator rather than two bias frames. This will give you a Gaussian noise value that accounts for the dark current noise. On the other hand, for FlatSNREstimator, a bias frame along with two well-exposed flat frames suffices.

Offline Sedat

  • Member
  • *
  • Posts: 50
  • Newbie
    • View Profile
Re: New script for denoising linear monochannel images
« Reply #160 on: 2019 November 24 09:48:33 »
Hello Mike,

All clear now! Thanks a lot!

Clear skies

Sedat
Sedat

Offline mschuster

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1058
    • View Profile
Re: New script for denoising linear monochannel images
« Reply #161 on: 2019 November 25 20:04:11 »
I wrote a new script that estimates detector settings for use by MureDenoise.

It will appear soon as an auto update as Scripts > Noise Reduction > MureDenoiseDetectorSettings.

The new script is very easy to use and fast. You supply two uncalibrated flat frames and either two bias or two dark frames.  The use of two dark frames rather than two bias frames allows the script to account for dark current noise.

The script provides estimates for detector gain, Gaussian noise, and offset for use by MureDenoise.

The script also provides a flat frame exposure estimate. As a double check on exposure, IMO you typically want to see value between roughly 30% and 70% of detector full-well in e-, to provide sufficient SNR and avoid problems with near saturation nonlinearity.



Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4531
    • View Profile
Re: New script for denoising linear monochannel images
« Reply #162 on: 2019 November 25 21:45:28 »
MureDenoise 1.25 should appear soon as an auto update.

Fixed bug in Use image metadata due to a typo.

Increased maximum Cycle-spin count to 48.

The text Use image metadata [none] indicates either unavailable image metadata or unsupported ImageIntegration options.

does LocalNormalization count as an unsupported ImageIntegration option? if so, what should be done in this case? not sure what to configure and where when you uncheck the 'use image metadata' checkbox.

rob

Offline mschuster

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1058
    • View Profile
Re: New script for denoising linear monochannel images
« Reply #163 on: 2019 November 26 07:03:02 »
does LocalNormalization count as an unsupported ImageIntegration option? if so, what should be done in this case? not sure what to configure and where when you uncheck the 'use image metadata' checkbox.

Hi Rob,

LocalNormalization is unsupported. Its pixel-wise modifications are difficult to model. Have you been using it previously with MureDenoise?

You have two options now:

1) Uncheck Use image metadata, set Variance scale to 1 and then adjust as needed.

2) Open File > FITS Header on your image, search for the keyword entry HISTORY > ImageIntegration.outputNormalization: Local, replace "Local" with "Additive + scaling", apply the change, and save. Then check Use image metadata, set Variance scale to 1 and then adjust as needed.

Either way, if you get an acceptable result, please let me know. Also, if you have been using LocalNormalization previously with the script please let me know.

Edit: Rob, please place your image in Dropbox so that I can take a look at its metadata. I am looking now at PCL II to see what it actually does, but I would like an example to double check. Thanks.
« Last Edit: 2019 November 26 07:20:10 by mschuster »

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4531
    • View Profile
Re: New script for denoising linear monochannel images
« Reply #164 on: 2019 November 26 09:28:55 »
i have probably not been using it with LocalNormalization; i usually don't use LN but i have a dataset with a very difficult flattening problem and so i tried it out.

if "use image metadata" is checked, does the script ignore the Variance Scale setting? it seems like it doesnt, so does it treat the value 1 as a special case to internally compute a different value?

i will upload the image.

rob

edit: https://drive.google.com/file/d/1RzE9i1boGHdsK65VJyFALyRD2HRBNvhw/view?usp=sharing