PixInsight Forum

Software Development => New Scripts and Modules => Topic started by: mschuster on 2015 November 03 09:11:49

Title: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 03 09:11:49
MureDenoise Version 1.21

Thanks,
Mike Schuster

The MureDenoise script denoises linear monochannel images corrupted by mixed Poisson-Gaussian noise. MureDenoise is applicable to single frame images and average combinations of equally exposed and registered frames.

The script supports an astronomical image processing workflow in which the denoising step occurs immediately after the calibration and optional average combination steps and prior to other linear or nonlinear processing steps.

The script applies an interscale wavelet mixed noise unbiased risk estimator (MURE) to find a denoised output image that minimizes an estimate of the oracle mean-squared error (MSE), or "risk", between the denoised output image and the unknown noise-free image.

Note: For linear multichannel images from monocolor detectors, run the monochannel denoiser on each channel separately. The script does not work properly for images from one shot color (OSC) detectors.

Warning: The script is adapted to denoise linear monochannel images mainly corrupted by shot noise, read noise, and dark current noise which is typically the case for astronomical data. The script does not work properly for other noise distributions, for saturated images, for debayered images, for linearly or nonlinearly processed images, for median combinations, or for drizzle combinations.

Warning: Do not combine denoised images. Signal-to-noise ratio (SNR) will be enhanced by combining noisy images and denoising the result. Combined images must be equally exposed, have the same pixel resolution, and be registered by projective transformation with no distortion correction.

More more information and examples, please see the script's documentation.
Title: Re: New script for denoising linear monochannel images
Post by: pfile on 2015 November 03 13:18:40
Mike, this is fantastic. i've used it already to denoise some narrowband masters and the results are impressive.

thanks for all your continued development

rob
Title: Re: New script for denoising linear monochannel images
Post by: cdesselles on 2015 November 03 13:51:59
Would it be possible to adapt this script to OSC images shot with a DSLR?
Title: Re: New script for denoising linear monochannel images
Post by: msmythers on 2015 November 03 13:56:53
If you can get the input parameter information for your camera all you would need to do is split the RGB channels.


Mike
Title: Re: New script for denoising linear monochannel images
Post by: msmythers on 2015 November 03 16:57:28
Mike

Very nicely done. Makes me wish I had real equipment and a good location to shoot from. Since I don't have either I wanted to see how I could adapt your script for my needs. I use a 12 bit Sony Nex-5 Mirrorless camera, don't laugh, and shoot from the coast of east Florida. A lot of my images are widefield with a 45mm lens.

Anyway since I don't have the gain or noise numbers for my camera I tried a little experimenting to see how I could use the script. That wasn't so bad.

The images I am showing are a crop of Cygnus with my 45mm lens. The image is 4608x3072. It is 23, 2 minute exposures stacked. I used ABE and then AutoHistogram with the average mean of the images 3 channels. Not a major stretch but enough to get a decent color balance. I then extracted the 3 channels for use with MureDenoise. After the different denoise tries or no denoise I then combined the channels and processed the image. No more noise reduction was applied after stretching.

There are 3 separate images. One with no denoise. One with my MureDenoise settings which included 23 combination count and 8 Cycle-spin count. The last was 2 passes of MureDenoise with the previous settings. All 3 images were then combined and the same exact processing was done to each combined image.

Thanks Mike!



Mike
Title: Re: New script for denoising linear monochannel images
Post by: RickS on 2015 November 03 22:47:54
That's very cool, Mike!  I have 50+ hours on a very dim Oiii target and I'll give this a try.
Title: Re: New script for denoising linear monochannel images
Post by: chris.bailey on 2015 November 04 01:22:44
Wow! I have just thrown about a dozen Ha stacks at it and in every case the results are better than I have been getting with MLT noise reduction, which has been my linear weapon of choice recently but have always had to follow this with a round of ACDNR post stretch. This seems to do it in one pass.

Does the Auto pixel interpolation setting pick up the setting used during star alignment?
Title: Re: New script for denoising linear monochannel images
Post by: jkmorse on 2015 November 04 08:00:39
Mike,

Can you explain a little more about the cycle-spin count.  From what you say in your note, it looks like it operates not as an intensity multiplier (like iterations in Deconvolution) but instead like the iterations in Masked Stretch where more means a better solution, not more intensity.  As such, if you are willing to trade time for quality, does it make sense to up the number beyond 8 and, if so, is there a practical limit?

Btw, the script looks fantastic!  Its the ability of talented people like you to supplement its workings that makes PI so much fun to use.

Thanks,

Jim 
Title: Re: New script for denoising linear monochannel images
Post by: jkmorse on 2015 November 04 08:19:54
Mike,

A second follow-up.  Can you expand a bit on the following statement:

Warning: Do not combine denoised images. Signal-to-noise ratio will be enhanced by combining noisy images and denoising the result.

Are you saying that the single frame image denoising is NOT an option if you are working with a stack?  If so, are there any limitations in running ImageIntegration beyond using an averaging method?   For example, are there any limitations on which rejection method we can use, whether we are better off with normalization set at additive or additive with scaling, or which scale estimator to use?

Thanks,

Jim



Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 04 08:54:39
Hi Rob,

Thank you so much, I appreciate your feedback! It is great to hear you got good results!

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 04 09:02:13
Would it be possible to adapt this script to OSC images shot with a DSLR?

Yes, msmythers is correct. For linear multichannel images, run the monochannel denoiser on each channel separately.

For linear OSC images, denoise the CFA channels not the de-Bayered RGB channels. Recombine the denoised channels and then de-Bayer. And as msmythers mentioned, be sure to use the correct detector parameters as they typically vary with ISO settings.

For a monochrome detector, again just denoise the filtered channels L, R, G, B, S, H, O, etc and combine the results as usual.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 04 09:18:43
Anyway since I don't have the gain or noise numbers for my camera I tried a little experimenting to see how I could use the script. That wasn't so bad.

Thank you Mike for posting feedback, your comments and images! I am glad to hear that your experiments worked!

If you can run the script twice and see improvements, chances are that the detector parameters are not as accurate as they could be. That is OK, no problem. With more accurate parameters, typically you can not run the script twice. Doing so results in obvious artifacts and loss of detail.

Also, I suggest running denoising before ABE or any other processing. Anything that modifies the signal modifies the noise statistics also. MureDoise does not know these transformations were performed, and so its noise estimates may be less accurate.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 04 09:25:20
That's very cool, Mike!  I have 50+ hours on a very dim Oiii target and I'll give this a try.

Thank you Rick! I hope you get good results. FYI: my 3nm H-alpha targets run 8 to 20+ hours and I am getting really good results on all of them.

Thank,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 04 09:30:23
Wow! I have just thrown about a dozen Ha stacks at it and in every case the results are better than I have been getting with MLT noise reduction, which has been my linear weapon of choice recently but have always had to follow this with a round of ACDNR post stretch. This seems to do it in one pass.

Does the Auto pixel interpolation setting pick up the setting used during star alignment?

That is excellent news Chris, thank you so much for posting!

Only choose Auto in the script when you selected Auto in StarAlignment. If you selected something else in StarAlignment, be sure use the same setting in the script. The settings need to match.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 04 09:52:03
Can you explain a little more about the cycle-spin count.  From what you say in your note, it looks like it operates not as an intensity multiplier (like iterations in Deconvolution) but instead like the iterations in Masked Stretch where more means a better solution, not more intensity.  As such, if you are willing to trade time for quality, does it make sense to up the number beyond 8 and, if so, is there a practical limit?

Thank you Jim for your kind comments.

Yes, a larger cycle-spin count means more iterations and a better solution. You can specify up to 32, at which point the results are at nearly highest quality possible in my testing. More than that give negligible improvements to the underlying risk estimator. Here is the idea:

The method uses a wavelet transform that subsamples at each level of refinement, each successive level having dimensions half the size of the prior level. The orthogonality of this transform allows the method to denoise each level independently, yet still achieve a globally optimal solution in terms of SNR across all levels. But the solution is shift variant, i.e. it depends on the locations of the subsampling boundaries. The solution to this problem is cycle-spinning: Run the method multiple times with different image shifts or translations and average the results. The final result is then nearly shift invariant.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: Roger B on 2015 November 04 10:00:10
noise reduction has always been by sore point for some reason , but this looks like it's my answer.  :D


Thanks

Roger
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 04 10:31:01
A second follow-up.  Can you expand a bit on the following statement:

Warning: Do not combine denoised images. Signal-to-noise ratio will be enhanced by combining noisy images and denoising the result.

Are you saying that the single frame image denoising is NOT an option if you are working with a stack?  If so, are there any limitations in running ImageIntegration beyond using an averaging method?   For example, are there any limitations on which rejection method we can use, whether we are better off with normalization set at additive or additive with scaling, or which scale estimator to use?

Hi Jim,

Yes. If stacking frames do not denoise the individual frames, rather denoise the resulting stack. If you denoise the individual frames you risk the loss of weak signals that rise above noise only after combination. Basically a weak signal might look like noise in the frame and be removed, but after combination the signal is stronger than noise and would not be removed. I have observed this loss and it can be significant, the deeper the stack the worse the loss.

Denoising a single frame is useful for survey work, for example, where you capture one frame and want to see how it might look with more time.

There are no limitations or preference on rejection method. No limitations or preference on scale estimator. For output normalization I suggest Additive with scaling. The ImageIntegration documentation recommends Additive with scaling for lights. I am experimenting with a scaling only option (no additive correction, not yet supported by ImageIntegration) that may turn out to be an good alternative.

Also, the script doesn't work properly for combinations of images captured at different pixel resolutions with different equipment.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: chris.bailey on 2015 November 04 10:42:45
Mike

It seems to work better if its the first operation post stacking (other than perhaps cropping dither borders) but certainly doing it before DBE seems to me to yield better results than the other way around. Is that to be expected ?

Chris
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 04 10:50:36
Yes Chris that is expected. Thank you for reporting your experience. The script's noise estimates will be less accurate on processed images.

Thanks
Mike
Title: Re: New script for denoising linear monochannel images
Post by: aworonow on 2015 November 04 13:39:56
Just tried it on a frame with a dozen faint galaxies that I got last night. Pretty impressive!

Thanks.....
Alex W
Title: Re: New script for denoising linear monochannel images
Post by: jkmorse on 2015 November 04 19:14:08
Mike,

I ran a quick test and got great results as shown on the links below to my before and after images from the Pickering's Triangle portion of my Western Veil image.  Really happy with the result, especially since I really didn't do any fine tuning of the inputs. 

Now my question.  At 2.59 arcsec/pixel, this image just screams for drizzle treatment but I do not see a way to use both drizzle and MureDeNoise.  Am I missing a step that will allow me to use both?  Of course, even if I can't it will be perfect for images from my Planewave CDK where the image scale ranges from 0.43 to 0.73 arcsec/pixel depending on my CCD.

One last point.  I assume in the DarkBiasNoiseEstimator, the number to use is the temporal noise and not the offset.

Thanks,

Jim

https://drive.google.com/file/d/0ByrTommGQM20Vy1pcVFjU1A0U3M/view?usp=sharing

https://drive.google.com/file/d/0ByrTommGQM20NWpfUUJGZGRYNDA/view?usp=sharing
Title: Re: New script for denoising linear monochannel images
Post by: gvanhau on 2015 November 05 07:22:44
Hello Mike
Thanks for sharing this script.
I run a quick test las night on a single and on a a stack of monochrome images form my atik 383L cam.

The noise reduction on the single image is really dramatic.
On the 41 images stack there was an improvement, but not so dramatic as in the single image I asume that this is because SNR is already very improved.

Regards
Geert
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 05 08:39:59
Hi Alex,
Thank you for your report!
Mike
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 05 08:52:01
Hi Jim,

Thank you for posting images! I am glad you got great results.

Drizzle combinations are not supported, no workaround is available. I am working on it, but not yet sure of success.

Yes, use the temporal noise number. I hope to release an update to DarkBiasNoiseEstimator soon with consistent labeling to make it less confusing.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 05 08:56:27
Hi Geert,

Thank you for testing!

Yes, you are correct, the stack has enhanced SNR and hence there is less noise to remove. My head post shows an example, at 1x actual size, the difference is harder to see in the 8 image combination.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 05 10:42:10
Hi Geert,

More info on stack SNR:

When you stack 100 frames, the usual rule says noise goes down by 10x, the square root of frame count. The top blue line below shows this as a function of count. At count 100 noise is scaled by 0.1.

But when you stack 100 frames, 99 of them were registered. The interpolation method used by registration averages adjacent pixels, introduces correlation, and hence reduces noise some. Curves show this for three StarAlignment pixel interpolation methods.

With drizzle the correlation is larger. Adjacent, smaller output pixels are more influenced by the same input pixels due to the fact that they are smaller. The dashed two curves show this. Increasing the drop size increases the correlation and further reduces noise. Also, noise does not drop as fast with drizzle due to the partial coverage.

All of this is relevant to the script, it needs to estimate (guess) how much noise is present in order to remove it.

Thanks,
Mike

(http://mschuster.zenfolio.com/img/s8/v75/p1646119980.png)
Title: Re: New script for denoising linear monochannel images
Post by: aworonow on 2015 November 06 08:40:00
Mike, et al.

Wouldn't most of the problem Mike describes disappear if the noise reduction was applied to the individual images before stacking? Clearly, that order of operations carries a computational penalty, but keeps pixel-to-pixel correlated noise out of the stack, as much as possible. Remediating correlated noise seems far more difficult than does simply avoiding it as much as possible. Also correlated noise fosters the introduction of artifacts by other processing tools that presume it is just not there. It's like trying to get OSC/debayered noise out...not easy because of the long-range correlation of noise that debayering introduces.

Can a script be applied to an image container?

Alex
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 06 10:15:47
Hi Alex,

I agree remediating correlated noise is difficult and best avoided.

Consider a 0.5 SNR signal in an individual image, a signal likely confused as noise by pre stack denoising, and removed. But in a 100 frame stack of noisy images the signal would be 5 SNR, not confused as noise by post stack denoising, and not removed. Pre stack denoising is not recommended for this reason. In my data this is not a theoretical problem.

With testing (thank you everyone) it appears that MureDenoise underestimates the amount of noise in a stack, and I think this is due to the correlation. And I do not have a solution. The only alternative at this point is to allow the Amount parameter to be larger than one. But this interferes with ease of use because now you have something that needs to be tweaked for stacks. This would allow you to get "more" denoising, but it comes at a cost of potential loss of signal due to the correlation.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: chris.bailey on 2015 November 07 01:17:18
Mike

I have thrown a lot of images at it over the past day or so and I would say it works VERY well where the images in the stack are of similar S/N weighting say in the range 0.95 to 1.0. It does a really fantastic job on some of my long sub-frame Ha image stacks! Where it starts to underestimate the noise is when the contributing images are more variable e.g where some of the contributing images have a weighting of less than say 0.85. Now there is an argument these should not be in the stack in the first place but 30 minute Ha subs are hard won when you only get one clear night a month! On these sort of images I have tried re-integrating with weights all set to unity and it does seem to improve NR but with other unwanted side effects. In my simple logic, when the S/N contribution of images in a stack is variable, the quadrature theory starts to break down so this somehow needs to be taken into account.

Chris
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 07 07:39:42
Hi Chris,

Thank you for your feedback. I agree that this is an issue that needs to be addressed. I am working on it. There is a similar issue due to flat fielding, I am working on this as well.

Also, on the noise correlation issue discussed above, I have made progress on this problem, and was able to fix some bugs. I hope to have a new release of the script in 10 days or so.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 09 16:55:12
The new MureDenoise version 1.8 is available in the head post.

The default denoising performed is unchanged.

The new Variance scale parameter lets you decrease or increase the default amount of denoising. Note that excessively large variance scale values risk the generation of denoising artifacts and the loss of signal-to-noise ratio (SNR).

The variance scale parameter can also be set based on information from an ImageIntegration process log file. The log file is a manually created .txt file containing a copy of the Process Console log generated by ImageIntegration. This feature results in more accurate denoising by taking into account the image output normalization and weighting performed by the combination process. Chris, this feature should help address the issues you see.

In the head post I added a warning concerning the script's inability to remove correlated noise introduced by the image registration process. For combinations, the script estimates the amount of correlated noise introduced by the registration pixel interpolation method, and then reduces the nominal amount of denoising so as to not inadvertently reduce SNR.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: chris.bailey on 2015 November 10 00:53:50
Mike

I have run a few of the previous 'problem' images through it this morning using the variance scale set from the Image Integration Log and it is a vast improvement. Typically the Variance Scale on these was in the 1.1 to 1.2 range but that was enough to make the difference. Doing the same on some of the better image run resulted in a Variance Scale of less than one (as low as 0.8) but the end result was better than setting it at unity. Variance scale is VERY sensitive to very small changes so setting it using the Integration Log seems the best approach based on the limited testing I have done, the extra step of cutting and pasting is well worth it.

Chris
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 10 07:12:42
Thank you Chris, this is helpful information.

Please post a 0 or near 0 log file in dropbox or inline in a post. I will investigate.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: chris.bailey on 2015 November 10 08:29:03
Mike

Apologies. For some reason the forum changed zero point nine to a zero point smiley face! I have not seen a variance scale of less that zero point eight five in about 20 images.

Chris
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 10 08:59:00
OK Chris, thanks.

The range of values you are seeing is roughly what I would expect. On my data I see maybe 0.9+ to 1.2+.

Note that the numbers you see will depend on which image you select as a reference in ImageIntegration. This is because images will be scaled differently by the process with a different reference.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: starhopper on 2015 November 10 13:38:56
Hello Mike,
thank you very much for this great work. I installed and used it with success. I was able to examine nearly all parameters for my images, even the variance scale. But I am not familiar with the DarkBiasNoiseEstimation script. What image should I provide for darkbias1 and 2?

Quote
The standard deviation of detector additive white Gaussian noise parameter is set to an estimate of the quadrature sum of read noise and dark current noise in a 40 minute dark provided by the DarkBiasNoiseEstimator script. Detector offset is set to 0 because the image is dark-subtracted.

I there a possiblility to determine the StarAlignment pixel interpolation. I have set it to auto, and does not find it in the fits header. Is this information lost?

Thomas
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 10 16:14:23
Hi Thomas,

Thank you very much.

For DarkBiasNoiseEstimator, please use two good bias images. The script will report "Temporal noise", which is an estimate of the "read noise" or "system noise" of the detector in DN units. Use this value for "Gaussian noise" in MureDenoise.

Regarding StarAlignment pixel interpolation, in my testing Auto means Lanczos-3 when aligning images with the same pixel resolution. So select Lanczos-3 in MureDenoise. You can double check this choice by looking at the StarAlignment process log, where the actual interpolation method used is listed. Note: some PI documentation says Auto means Lanczos-4 for the no rescaling situation, but in my testing Lanczos-3 is listed in the process log. Probably best to double check.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: starhopper on 2015 November 11 12:40:25
Hello Mike,

thanks for your explanation. Look it seems to work fine with my image. It is a stack of 11x5min OIII frames, registerered with lanzos3 and stacked with average sigma clipping.


Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 11 17:59:15
Hi Thomas,

Thank you for posting your results, I am glad to know it is working!

Regards,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: cs_pixinsight on 2015 November 11 19:37:06
Mike, all I can say is WOW!  That last example posted by starhopper is amazing  :surprised: :surprised: :surprised: :surprised:

Great work as usual.

Craig
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 14 08:26:15
Thank you Craig!

The new MureDenoise Version 1.9 is available in the head post.

This version includes a new large scale flatfield compensation option, which improves denoising quality by compensating for spatial noise variance scaling due to flatfielding.

Flatfield compensation is useful for telescopes with more than ~10% optical vignetting. For telescopes with less vignetting, flatfield compensation results in negligible denoising quality improvement.

The denoising performed by the new version is unchanged unless you use the new option.

Thanks,
Mike

Update: Here is an FSQ flatfield. The thick lines mark the location of the mean exposure, the center is ~2% brighter, the corners up to ~4% dimmer. SNR is lower in the corners (fewer photons). Flatfielding equalizes the exposure, SNR remains unchanged, and so the noise level ends up being relatively higher in the corners. Given a flatfield, the new MureDenoise option varies the amount of denoising across the image appropriately.

(http://mschuster.zenfolio.com/img/s8/v15/p1718911853.png)
Title: Re: New script for denoising linear monochannel images
Post by: Philippe B. on 2015 November 16 06:13:35
Hi Mike
Very nice work ! Thank you !

Title: Re: New script for denoising linear monochannel images
Post by: jkmorse on 2015 November 16 19:25:10
Mike,

I have been playing around with the script and making some interesting findings I want to confirm with you:

1.  While only 19 subs deep, the data set I used for testing is very good, well focused, etc, from my F16M - Planewave CDK12.5 system;
2.  The "default" settings you recommend are not coming up with the best fit solution but by making adjustments in the settings, I am able to get excellent results (almost miraculous actually given that it cleans both the background and the target so well);
3.  The two most sensitive settings for me are Gaussian noise and variance scale (which is probably to be expected).  I get my best results in getting the Gaussian noise set first, then fine tuning using the variance scale;
4.  Here is an example using a year old M81-M82 data set of 19 10 minute images (working in blue):
     a.  Reported gain on my F16M is 1.5 but I get better results lowering to 0.8
     b.  Gaussian noise as measured using two 10 minute dark subs is 26.98 but I have to lower that to around 10
     c.   Measured variance scale shows 1.312 but at that level (and in fact for anything above around 0.85 which I am using), I get "blocky" artifacts, almost like it is pixelating the image.

To put this into visual context, attached in this and the next two posts are excerpts representing my base, recommended settings and adjusted settings images.

Would love to get your thoughts on this experiment.

Thanks,

Jim

p.s. If I forgot to say thank you for this incredible tool, THANKS!!!!



Title: Re: New script for denoising linear monochannel images
Post by: jkmorse on 2015 November 16 19:26:04
Recommended settings image:
Title: Re: New script for denoising linear monochannel images
Post by: jkmorse on 2015 November 16 19:26:43
Adjusted settings version:
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 16 22:55:34
Hi Jim,

Thanks for posting! Here are my thoughts: I would expect for the F16M a gain of ~1.5 e-/DN and a Gaussian noise ~7 DN assuming a read noise of ~10 e-. In general, if you see denoising artifacts like these, the cause may be one or more of these setting problems: gain is too low, Gaussian noise is too high, variance scale is too high. Any of these things can (will) cause the script to overestimate noise. If possible, please post fits/xisf of integration, two darks, two biases, and the ImageIntegration log in dropbox and I will investigate further.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: chris.bailey on 2015 November 17 07:36:59
Jim

Just a thought but are those images cropped with any sort of rotation applied to the crop selection?

Chris
Title: Re: New script for denoising linear monochannel images
Post by: jkmorse on 2015 November 17 08:41:15
Mike & Chris,

No, just straight clips, no rotation.

Let me do some further testing before I send you samples, Mike.  Based on your note, it looks like I still may have the Gaussian noise set too high at 10 and need to test at ~7.  I also want to retest using two bias frames rather than the dark frames.

I will report back.

Best,

Jim 
Title: Re: New script for denoising linear monochannel images
Post by: jkmorse on 2015 November 17 08:54:19
Mike,

I have a side question about how scripts work generally.  Unlike regular process tools, when the script is open you lose the ability to manipulate other controls.  For example, when doing testing with Mure on a preview, I cannot use the undo button and instead have to close Mure, hit undo and then reopen the script.  This is not limited to Mure but seems to happen across the board (for example I see the same thing with SHO_AIP). 

Is there something fundamental to the difference between processes and scripts that causes that?

Thanks,

Jim 
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 17 09:14:37
Unlike regular process tools, when the script is open you lose the ability to manipulate other controls. ... Is there something fundamental to the difference between processes and scripts that causes that?

Yes, this modal nature is expected for scripts. The PixInsight scripting environment prevents access to other controls when a script dialog is open.

Also, andor.com lists F16M read noise at 22e-. This is ~2x higher that I expected, and also ~2x higher that other 16803 vendors claim. Maybe this includes noise from anti-RBI IR preflash?

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: jkmorse on 2015 November 17 17:28:15
Mike,

I suspect but don't know that its a bust in the Andor/Apogee publication.  Not the first time I have seen that.  Following your advice I dropped the  Gaussian Noise and found the sweet spot for my F16M at 0.8.  I combined that with variance scale of 0.85 to get my best results, however to optimize the noise reduction I had to drop my gain from 1.5 to 0.8.  That produced superb results without any loss of detail or blockiness (I just made up a new word  ^-^).  All in all, I am more than happy to accommodate numbers that don't seem to add up when I get these kinds of results  :D.

Now, having seen that your tool is supremely versatile, I am going to play with it on a drizzle integrated image.  Ya just never know until you try.  I will obviously report back.

Best,

Jim
Title: Re: New script for denoising linear monochannel images
Post by: Tromat on 2015 November 19 07:45:33
Hi,

the script works very nice on CFA images but the problem is that it requires a lot of work and the result is not ok.
I'm using a DSLR and tried to split the raw files with SplitCFA so I have splitted darks, flats, biases and light, then I made all the preprocessing for each CFA list of files (4 of them so). On the stellar registration part I used the same original image so the bayer array is at the right position.
I then merged the 4 integrations to get the final color picture but there's many bayer artifacts. I tried the script on each CFA image, it works nice, but as the merging leaves many artefacts it's not usable :/

Did I make something wrong or the script cannot be used on DSLR pictures ?

Here's what it looks like on the merge image (2:1 ratio).
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 19 08:37:06
Did I make something wrong or the script cannot be used on DSLR pictures ?

Hi Tromat,

Thank you for posting. DSLR denoising should be possible, but remains untested. I may have overlooked a big issue that prevents the script's use.

I would suggest trying your entire workflow without denoising to see if you can recover a good image. If not, then a workflow problem or maybe this separate channel calibration, registration, and integration workflow is flawed and simply cannot work. If so, then some unforeseen problem prevents separate channel denoising from working properly. I wish I could say more but I don't have any experience with DLSR.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: pengsloth on 2015 November 19 10:54:29
I just used this process last night for a set of DSLR (caon 6d) images. After registering the files I split the image into RGB and then did the process over each of them and then put them back together. It worked really well I think.
Title: Re: New script for denoising linear monochannel images
Post by: pfile on 2015 November 19 11:10:38
right - that seems like the right place to do the split, after the master is made, rather than up front before calibration...

rob
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 19 11:15:04
Rob and pengsloth, thank you for this information.

The new MureDenoise Version 1.10 is available in the head post.

The denoising quality is improved on average combinations. The improvement follows from a better covariance hypothesis that compensates for the correlation introduced by the registration process.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: Tromat on 2015 November 20 04:57:54
I would suggest trying your entire workflow without denoising to see if you can recover a good image.
The whole workflow doesn't really work actually, that's the point, the resulting picture suffers of bayer artifacts even without noise reduction.

Quote from: Pengsloth
After registering the files I split the image into RGB and then did the process over each of them and then put them back together. It worked really well I think.
The problem is that each channel is "corrupted" by the interpolation, it doesn't seems to be the most clean way to denoise. I should try with Bayerdrizzled images.
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 20 07:55:05
Hi Tromat,

You are correct on both points. I apologize for my mistake in suggesting an OSC workflow that does not work.

MureDenoise does not support DLSR and OSC detectors, neither traditional debayer nor bayer drizzle. There is no workaround. MureDenoise lacks a model for the interpolation involved.

Thanks,
Mike

Update: I will modify the head post to indicate this limitation shortly. The script documentation will be updated on the next release.


Title: Re: New script for denoising linear monochannel images
Post by: pfile on 2015 November 20 10:10:52
superpixel debayering should be an option for OSC, i don't see why not.

rob
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 20 10:48:08
superpixel debayering should be an option for OSC, i don't see why not.

Yet another form of interpolation not properly modeled by the script. I am working on monocolor detector drizzle support, maybe 50/50 chance of it working out. If this does work then maybe bayer drizzle support is possible. Don't know yet. On the other hand, for traditional debayer support, little chance, not working on it.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: pfile on 2015 November 20 11:20:32
is it because the pixels have physical separation between them? because there's no interpolation in superpixel debayering...

rob
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 20 12:04:56
I see your point. Depends on how you define interpolation I guess. Each output pixel is a weighted sum of input pixels. This means correlation and problems for the script.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: jkmorse on 2015 November 20 12:38:10
Mike,

Just so you know, I have my fingers crossed on your drizzle integration efforts (I am confident they are way better than 50/50, but that may be a Christmas wish).  I continue to play with and apply the script to older image sets and they are getting a new life from the script.  I am sure you know this and its why you developed the script, but one of the frustrations in all the other denoising tools is the fact that you either lose detail in the target or have to mask and protect the target which means you have an unnatural break between the denoised background and the still noisy target space.  Mure gets around that issue completely and that is why it is so brilliant in execution and incredibly satisfying to use. 

As an aside, at what stage will this become a part of the regular distribution in PI updates like your other scripts?

Best,

Jim 
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 22 10:09:39
Thank you Jim,

The script should be part of the PI distribution soon. Drizzle support is looking less and less like a short term effort (a lot more correlation between pixels). A new version 1.11 (bug fixes, improvements) is available in the head post.

Regards,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: jkmorse on 2015 November 22 14:04:00
Mike,

One thing I am finding is that I am not getting the same level of noise reduction with narrowband images as I do with RGBs.  Is that to be expected and is it a function of the lower SNR?

Thanks,

Jim
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 November 22 14:56:52
There is less noise in dimmer signals (less shot noise). Maybe as a double check, generate the method noise image, create a few small previews (16 x 16 roughly) in various areas with differing brightnesses, and measure the standard deviation (or better yet Sn) in each. Method noise is the actual amount of noise the script guessed and removed. A bit tedious, but it is possible to compare where more or less noise got removed both within and between images.

Thanks,
Mike

Update: For the KAF-16803 (gain 1.5 e-/DN, gaussian noise 7 DN) this plot shows how noise DN varies with signal DN (from 1 to 65535). For the dimmest signals noise floor is 7 DN. Noise ramps up from there due to shot noise, by more than a factor of 10 for bright signals.

(http://mschuster.zenfolio.com/img/s5/v120/p1833552258.png)

This plot shows the same data as the signal divided by the noise (SNR). Dim signals have low SNR. Dim signals look "more noisy", but they actually have "less noise".

(http://mschuster.zenfolio.com/img/s3/v23/p1866203014.png)
Title: Re: New script for denoising linear monochannel images
Post by: jerryyyyy on 2015 December 21 07:30:03
Hi,

I find this very helpful, at least on one image so far. 

http://www.astrobin.com/233107/B/

Revision done with MureDenoise script instead of TGVDenoise (all other processing the same):

Settings: Gain 0.483 from  FITS
Gaussian Noise 29.51 from DarkBiasNoiseEstimator

DarkBiasNoiseEstimator script also gives Offset 339.25 but not used (as per PI instructions not too clear).  Should this be put into the parameters?  I have another DSW image I am processing so can test it out I guess. 
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 December 21 08:40:25
Hi jerryyyyy,

Thank you.

Gain seems OK. Gaussian noise seems a bit high. I would expect ~20 DN for bias frames and maybe ~24 DN for 600" dark frames given specs from sbig.com. But 29 DN might be true.

Offset should be zero (the usual case when denoising a calibrated image). The exception is when you want to denoise a raw, uncalibrated frame. Then use a nonzero offset. Basically in this case the denoiser needs to discount the bias or dark frame median as a data "pedestal" or offset).

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: pfile on 2015 December 21 14:27:06
mike, these darks have RBI preflash turned on which might affect the numbers. i don't know if you've ever run your estimation scripts on darks with RBI mitigation...

rob
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 December 21 14:40:14
Hi Rob,

OK, yes preflash would result in more noise and higher estimates. Maybe you could dropbox a pair of bias frames without preflash and another pair with preflash? It would be great to have something to test and compare.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: pfile on 2015 December 21 16:41:55
OK, let me look back thru what i've got - if i don't have anything appropriate i'll make new bias frames with and without

rob
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 December 21 23:18:03
Thanks for the files Rob.

Noise in the biases (preflash and no-preflash) are about the same, ~7 DN. Maybe with bias there is no time for preflash leakage to add noise?

The average noise in the 600" darks (preflash and no-preflash) is more, ~9 DN, with preflash noise about 15 to 25% more than no-preflash depending on how measured. (Would need pairs exposed identically both ways to get a better estimate, but for this purpose really no need.)

So the 29 DN mentioned above with preflash is reasonable IMO.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: jerryyyyy on 2015 December 22 07:13:59
Guys,

How are you getting the noise?  Different numbers from what I get with DarkBiaNoiseEstimator.... maybe I am using the script wrong?  Seems to compare two random images from set of darks.  I got similar numbers from my STT 8300M and pfile's camera, which is suspicious. 

But, regardless images look good.  When I look in detail at the images there seems to be a sorta checkerboard pattern to the background... noise is reduced overall but there is a funny variance pixel to pixel to my eye.  When you do LHE this is obviously increased.  See screen shot.
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 December 22 07:58:10
Hi jerryyyyy,

The checkerboard pattern is an artifact due to excessive denoising. Typically one or more parameter is bad: Gain is too small; Gaussian noise is too big; Offset is nonzero (for calibrated images); Variance scale is too big.

To estimate Gaussian noise, pick two good bias frames (temperature matched to lights) and run DarkBiasNoiseEstimator. As a sanity check, you should get a noise estimate (in DN) roughly equal to manufacture specs "read noise" or "system noise" (in e-) divided by "gain" (in e-/DN). Or a little bit larger (say 10%) as detectors get a bit more noisy with age.

Longer light exposures will have additional noise due to dark current and preflash leakage (if any). So rather than using two biases for a Gaussian noise estimate, you can use two good dark frames (time, temperature, and preflash matched to lights). Expect a higher noise estimate. There is more room for error here so I recommend using bias frames to start with.

Thanks,
Mike

PS: Maybe you can dropbox a pair of matching 8300 biases and I will check them out?
Title: Re: New script for denoising linear monochannel images
Post by: jerryyyyy on 2015 December 23 07:17:40
Have packed up some Darks and Biases that I have been using.  Hummm, I was using the darks...  BTW, even with the problems I have had with script my first effort using it on DSW data got IOTD on astrobin.... good publicity for this script. 

http://www.astrobin.com/233107/B/

Can you send me a PM with your real address so I can share the link. 
Title: Re: New script for denoising linear monochannel images
Post by: pfile on 2015 December 23 20:30:34
jerry i had previously sent mike two fresh 600s darks and two fresh bias, 1mhz, with and without RBI - that's what he's referring to above.

rob
Title: Re: New script for denoising linear monochannel images
Post by: Dimitris Platis on 2015 December 24 03:04:57
I should add that testing the effects of the script in a Drizzled image (variance 0.8, and Noise taken from bias) I had a significant noise reduction.
Perhaps its worth just testing different parameters of the scripts at random even in Drizzled images
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 December 24 08:50:31
Hi Dimitris,

For drizzled images, here is a starting point for experimentation. Set Combination count to the number of images combined in the drizzle, set Interpolation method to Nearest Neighbor, set detector parameters as usual (gain and Gaussian noise from bias), and set Variance scale as shown in the table below. This table shows values for various Drop shrink values and an Output scale of 2. The resulting denoising will likely be conservative and minimal, but the result will be in a sense "SNR safe" as it accounts for the relatively large noise correlation introduced by drizzle up-sampling. Use this as a starting point, you may try increasing Variance scale to get more denoising at risk of artifacts and loss of SNR.

For bayer drizzled images, you can try something similar. (Warning: this application to bayer drizzle is theoretical and has not been tested so it may be wrong.) For R and B channels, do as above using a Variance scale of 0.11 (the entry in the table corresponding to a Drop shrink value of 0.5). For the G channel, use the same variance scale of 0.11, but double the Combination count value to account for the two G channels. Again, use this as a starting point and increase Variance scale at your own risk.

I want to say again that the results with these suggestions will likely be minimally denoised. Almost or completely to the point of not even worth doing. But they might be a useful starting point for playing around with the script.

Thanks,
Mike

Drop shrinkVariance scale
0.50.11
0.60.13
0.70.15
0.80.16
0.90.17
1.00.17
Title: Re: New script for denoising linear monochannel images
Post by: Dimitris Platis on 2015 December 24 11:18:46
How exactly would u measure loss of SNR for different Variances?
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 December 24 17:55:43
Unfortunately I don't. Variance scale contributes to an input noise hypothesis, the denoiser gives an optimal solution for this input and a built-in set of denoising processes. The denoiser does not give comparisons between solutions for different inputs, in a form that would help answer your question. As a result, my SNR comments for different inputs are qualitative: the higher the specified noise (maybe beyond a justifiable uncorrelated detector and shot noise specification), the higher the risk of confusing signal and noise and the introduction of artifacts. When playing around with drizzle images, just be aware of the risk and judge for yourself whether the result looks acceptable or not.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: Dimitris Platis on 2015 December 24 23:17:48
At the completed Drizzled image state I agree it is difficult to estimate SNR.....however, I must say that even at Variance 0.8 (tested with a Heart and Soul image) no loss or blurring if u wish of structures where observed.
The noise reduction seemed to be done correctly in a manner that seemed natural and not artificial.
Title: Re: New script for denoising linear monochannel images
Post by: jerryyyyy on 2015 December 25 10:39:39
Hi Dimitris,

For drizzled images, here is a starting point for experimentation. Set Combination count to the number of images combined in the drizzle, set Interpolation method to Nearest Neighbor, set detector parameters as usual (gain and Gaussian noise from bias), and set Variance scale as shown in the table below. This table shows values for various Drop shrink values and an Output scale of 2. The resulting denoising will likely be conservative and minimal, but the result will be in a sense "SNR safe" as it accounts for the relatively large noise correlation introduced by drizzle up-sampling. Use this as a starting point, you may try increasing Variance scale to get more denoising at risk of artifacts and loss of SNR.

For bayer drizzled images, you can try something similar. (Warning: this application to bayer drizzle is theoretical and has not been tested so it may be wrong.) For R and B channels, do as above using a Variance scale of 0.11 (the entry in the table corresponding to a Drop shrink value of 0.5). For the G channel, use the same variance scale of 0.11, but double the Combination count value to account for the two G channels. Again, use this as a starting point and increase Variance scale at your own risk.

I want to say again that the results with these suggestions will likely be minimally denoised. Almost or completely to the point of not even worth doing. But they might be a useful starting point for playing around with the script.

Thanks,
Mike

Drop shrinkVariance scale
0.50.11
0.60.13
0.70.15
0.80.16
0.90.17
1.00.17

Just like to report this is working fine on a new image.  No checkerboarding.  Must have been that image.

To use these Table, save as a txt file?  Did not seem to identify a .txt file. 
Title: Re: New script for denoising linear monochannel images
Post by: javajunkie2121 on 2015 December 29 18:17:14
Hi:

Has anyone had image size errors when trying to use a master light and a master flat in the script?

I have a red filter light master 2x2 bin and a matching master red flat 2x2 bin and the script is giving me the unequal size error? 

I hadn't had trouble using the master flat for image calibration.  Not sure if I'm using the flatfield compensation correctly?

jeff
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 December 29 19:02:12
Hi Jeff,

Sorry for the trouble, the script does test for equal width and height. I double checked the code and don't see a bug. Maybe you could put both in dropbox and I will test.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: javajunkie2121 on 2015 December 30 07:32:23
Hi Mike:

here's link to the 2 files: https://www.dropbox.com/sh/tldr6nwgzi648g0/AACAyK-Rn9_1fxOXSREz86SLa?dl=0 (https://www.dropbox.com/sh/tldr6nwgzi648g0/AACAyK-Rn9_1fxOXSREz86SLa?dl=0)

master flat was 25 flat frames, bias subtracted, combined with image integration
master red light was 11 frames, flat and dark calibrated, combined with image integration

jeff
Title: Re: New script for denoising linear monochannel images
Post by: martin farmer on 2015 December 30 07:40:43
Hello,

I have run into a message from this excellent noise routine:-

Attempt to execute a Script instance recursively. Sorry, but this is not supported in this version of Pixinsight.

Version 1.14 of MureDenoise
Pixinsight Version 01.08.04.1195 Ripley (x64)
Running on a MacBook Pro under OSX 10.11.2 El Capitan

Is there a newer version of this script?

Thanks

Martin
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 December 30 08:59:48
Hi Jeff,

Thanks for the files. The images do have unequal size, so the script is working properly: Master red is 3352x2532 and Master flat is 1676x1266.

For denoising, you may upsample the flat using the process IntegerResample with the parameters Resample Factor set to 2 and Upsample selected. For image calibration I would not want to upsample a flat, but for denoising upsampling will not introduce problems since the script applies a large scale smoothing filter.

Note: Master red might be a drizzle result. If so, note that the script does not denoise drizzle properly.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2015 December 30 09:15:59
Hi Martin,

Thank you, sorry for the problem. 1.14 is the current version. Can you explain how to reproduce the error?

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: martin farmer on 2015 December 31 01:03:16
Hello Mike,

I have to hold my hands up on this. It was my fault.
I got this message by setting the parameters in the dialogue box then dragging the triangle over to apply it to the image. Instead of clicking the Denoise button.

I put it down to my age!

The script works very well on my mono narrowband frames prior to stacking. I find it does not alter the structure contained in the picture but the pepper like noise.

Many thanks Mike for writing this useful script.

Martin
Title: Re: New script for denoising linear monochannel images
Post by: joelshort on 2016 January 06 19:56:46
I just became aware of this script and boy does it look promising.  Initial tests really impressed me.  I have a couple questions:
1.  Am I correct in assuming that if using a master flat I cannot perform DynamicCrop on the light frames prior to MureDenoise?

2.  In reply #30 it was suggested to use information in the ImageIntegration log file as a guide for setting the variance scale.  What information should I look for in the log?

Thanks for your work on this script.  Again, it looks to be an amazing tool.
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2016 January 07 09:47:29
Hi Joel,

1) You can crop before denoising. The flatfield must be cropped likewise. With DynamicCrop, set the crop bounds on the image, but before committing, drag the process icon to the flatfield, then commit on the image. Both should be cropped the same way.

2) Select and copy all of the lines in the Process Console window generated by ImageIntegration, use the Script Editor window to paste the lines into a new .txt file, save, and then load in the script. The denoiser will set Variance scale to account for the average frame normalization and weighting.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: joelshort on 2016 January 07 09:55:37
Got it, thanks Mike.  Is there any reason, as far as MureDenoise is concerned, that I SHOULD crop before running the script?  Will black borders from registration negatively affect the denoising? 
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2016 January 07 10:01:30
Hi Joel, this is something I have not tested. Guessing I think the script will work fine with black borders. Maybe you could try and let me know if you see anything bad? I will also test by adding a synthetic black border on my data.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2016 January 07 10:23:55
Hi Joel,

A quick test with synthetic black borders on one of my images: The script works fine with black borders.

I added a black border around my image, and a matching border on the flatfield, not black but rather a fill color of 0.5, as a zero in a flatfield would not be valid.

The denoised results look great. I generated the method noise image and see no artifacts in the image, along the border edges or elsewhere. In the black border areas there is a extremely small random looking texture in the range of 0.0 to ~0.01 DN visible only with an extreme stretch. This is of course negligible.

So the only restriction is no black borders in the flatfield. Denoising with black borders and no flatfield works fine likewise.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: joelshort on 2016 January 07 10:32:49
Just to be clear, I understood you to say previously that one should use a flat if denoising a stacked image, correct?

I just can't believe what kind of results I'm getting with this script.  It's so easy and appears to work really well, as well as correctly handling signal vs. noise. 
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2016 January 07 10:41:25
Thank you Joel,

Using a flatfield is optional. A flat can be used either on a single frame or on a stacked image. Either way specifying a flat helps the script account for differing noise levels due to vignetting. This is more useful on telescopes with lots of vignetting. When vignetting is small the improvement is negligible.

Regards,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: joelshort on 2016 January 07 18:45:40
OK Mike, I've got a new one for you.  I'm currently working on an image of M42.  The main Red stack has 12-15min subs.  Of course I needed to take shorter exposures for the core, so I also took 12-1min and 12-30sec exposures for an HDRCombination.  In the .zip file I've linked to below there are 3 files:

"Orion_R" is the original stack with 12-15min images
"Orion_R_HDR" is the HDRCombination stack including 12-15min, 12-1min and 12-30sec images. 
And a master FLAT

When I used MureDenoise on the "Orion_R" image I get a very pleasing result.  Here were the parameters I used:
Gain = 0.335
Gaussian Noise = 16.02
Offset = 0
Variance Scale = 1.0
Cycle Spin Count = 16
I also used a flat.

Now here's the problem: when I use these same parameters on the "Orion_R_HDR" image it turns out horribly.  I first tried lowering the Variance Scale all the way down to 0.1 but it was still ugly.  Realizing that the HDR image now contained 36 images rather than 12, I changed the Combination Count to 36 and with a Variance of 0.1 the result was a lot better but I still saw some artifacts. 

How does using HDRCombination mess up the denoising process?  It seems to me like the only parts of the short exposure images that would be added to the main image would be the core of M42 and would not really affect the overall noise of the main image, yet it really messed up MureDenoise.

Here's a link to a zip file containing the images:
https://www.dropbox.com/s/mely703edrtkzgs/Orion.zip?dl=0

Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2016 January 07 19:19:29
Hi Joel,

Sorry for the trouble. The script will not denoise HDR combinations properly. In general the script does not work properly on linearly or nonlinearly processed images. Processing alters noise statistics in ways the script cannot handle.

Regards,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: joelshort on 2016 January 07 19:24:02
I knew that, but I guess I didn't think that the HDRCombination script really "processed" the image at all...guess I was wrong. 
Title: Re: New script for denoising linear monochannel images
Post by: pengsloth on 2016 January 07 22:36:52
I used the script on a HDR image. However what I did was run the script on the long exposure image only and then combined the denoised long image with the normal short exposure. Because the short exposure is generally much better signal to noise it turned out great. Though I guess if your short image was also noisy you could denoise it too before combining.
Title: Re: New script for denoising linear monochannel images
Post by: joelshort on 2016 January 08 05:47:18
Yeah, that's what I did too.  Worked fine.
Title: Re: New script for denoising linear monochannel images
Post by: jerryyyyy on 2016 January 18 10:24:23
Hi,

Have been working away with the script and generally very happy but thought, Hey if this is good on a regular L image, what about a Super_L, where the RGB are added in.  I have some 28 L subs and an additional 16 each of RGB. 

I am still getting noise reduction, but it looks "pasty" unless I set the G noise very low....  When I run the ImageIntegration Proc and compare for example the G noise they are 2.09 versus 0.94 (to the -4) for the L versus Super L. 

Does this make any sense (sanity check please). 

Keep up the good work!

BTW, I have drizzle files on these data if you have any suggestions, guess that will not help. 
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2016 January 19 14:00:10
Hi Jerry,

Sorry for the trouble, MureDenoise will not denoise multifilter combinations properly.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: joelshort on 2016 February 16 11:42:48
Mike,
The more I use MureDenoise the more I really like it.  Fantastic!  The one suggestion I might make is that when I load the variance scale from a txt file created from the ImageIntegration process there is a pop up window that states the variance scale and the number of images used.  The variance scale automatically gets changed in the main window, but the combination count does not.  If you're forgetful like me, I might forget to change the combination count or worse yet can't remember how many images are in the stack.  Auto-populating that field would save me from myself.
Title: Re: New script for denoising linear monochannel images
Post by: mmirot on 2016 February 17 15:28:51
Mike,
The more I use MureDenoise the more I really like it.  Fantastic!  The one suggestion I might make is that when I load the variance scale from a txt file created from the ImageIntegration process there is a pop up window that states the variance scale and the number of images used.  The variance scale automatically gets changed in the main window, but the combination count does not.  If you're forgetful like me, I might forget to change the combination count or worse yet can't remember how many images are in the stack.  Auto-populating that field would save me from myself.

Can't we get the variance and number images put in the image header with XSIF after integration?
The script could automatically populate this info. Juan should jump at the chance for another use of XSIF ;)

Also, can some post a screen shot on how to make the integration txt file?

And Mike this is a kill app!
Title: Re: New script for denoising linear monochannel images
Post by: joelshort on 2016 February 17 16:10:38
Immediately after ImageIntegration, copy the info from the Process Console that ImageIntegration created.  Then open the Script Editor tab and click the "T" to create a new text file.  Paste the copied info in the right hand panel and then save it somewhere.  Very easy, even I figured it out. 
Title: Re: New script for denoising linear monochannel images
Post by: joelshort on 2016 February 17 16:11:26
A couple more screenshots
Title: Re: New script for denoising linear monochannel images
Post by: pfile on 2016 February 17 21:28:52
on osx, if you are comfortable with the terminal, you can highlight and copy what joel has shown above and then in the shell, issue the command: pbpaste > integration.txt

rob




Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2016 February 18 09:17:53
Joel and mmirot, thank you very much, Rob thanks also. Joel, I will implement your suggestion. mmirot, your suggestion remains a wish list item.

Regards,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: mmirot on 2016 February 18 19:03:06
Thanks Joel and Mike.
Title: Re: New script for denoising linear monochannel images
Post by: dmcclain on 2016 February 25 14:23:12
I know this isn't kosher from the math perspective... but I have had tremendous success going back over old images taken with an ATIK 490 OSC, already deBayered, registered, and integrated. Out of more than 100 such images, only one failed to show drastic improvement applying MURE denoising against the integrated stack image.

For parameters, I'm just putting in the measured gain and read noise of the sensor, treated as a mono sensor. So I'm probably not even using the correct noise setting, since you indicate earlier in the post that it should be the quadrature sum of read noise and dark current noise.
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2016 February 26 05:28:42
Ok, thanks, I recommend checking carefully for artifacts and loss of detail.

Also, version 1.15 just released as an auto update. Load button loads combination count also (thanks Joel). New note that flatfield must be bias or dark-subtracted. No other changes.

Regards,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: steveyzh on 2016 March 11 08:28:00
Hello Mike,
I got this error msg, the script console shew running but no result after 15 mins.

Processing script file: C:/Users/Administrator/Desktop/MureDenoise.1.15/MureDenoise.1.15/MureDenoise/MureDenoise.js
*** Error [022]: C:/Users/Administrator/Desktop/MureDenoise.1.15/MureDenoise.1.15/MureDenoise/MainViewController.js, line 568: TypeError: console.beginLog is not a function

Steve
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2016 March 11 09:20:09
Hello Steve,

Sorry for the trouble.

This error likely is due to an attempt to run the script in an older version of PixInsight that does not support the console logging functions (beginLog/endLog/logText introduced in version 1115).

Please double check that you are running the latest version (1195 on Mac/Win, 1198 on Linux/FreeBSD). If you are running the latest version, then I am not sure what to suggest, other than to try a system reboot and a PixInsight software reinstall.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: gnomus on 2016 July 10 08:02:11
This is a great script.  Thank you.  A friend suggested some settings for my KAF8300 chip, and these work well.  I now have a QSI 690.  Could anyone recommend some starting settings for that camera?
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2016 July 11 08:47:12
Thank you gnomus. The QSI spec here (http://www.qsimaging.com/690-overview.html) gives typical high gain 0.16 e-/DN (default), low gain 0.36 e-/DN, and read noise 3 e- RMS in high quality mode. For the script's gain value, specify one of these. For Gaussian noise in DN units, convert read noise in e- to DN by division by gain, eg either 3 / 0.16 = 18.8 DN or 3 / 0.36 = 8.3 DN.

To find a more accurate gain value for your camera, open a frame, choose File > FITS Header, choose the frame's view, and scroll through the list to find the EGAIN keyword. Use this value for the script's gain, and again divide 3 by this value to get Gaussian noise in DN.

To find a more accurate Gaussian noise value, open two good bias frames and run the script DarkBiasNoiseEstimator. Select the two views and click Estimate. Use the temporal noise value which is in DN.

As a sanity check, both EGAIN and the temporal noise value from DarkBiasNoiseEstimator should be roughly equal to the spec values (noise after division).

Best,
Mike

PS: Some vendor drivers do not include the EGAIN keyword, or the value may be generic and not specific for the camera. In this case, to find a more accurate gain value for your camera, open two well-exposed uncalibrated flat frames and one good bias (or flat-dark) frame and run the script FlatSNREstimator. Select the three views appropriately, select 20% as the foreground quantile, and click Estimate. Use the gain value which is in e-/DN units.
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2016 December 19 08:25:11
MureDenoise Version 1.18 soon to be released as an update with small denoising quality improvements.

• Exploits the decorrelation of noise at coarse wavelet scales.
• Refines the load variance scale calculation in some cases.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: Duncan on 2016 December 20 19:06:58
This looks great - but, dumb question: how do I install the script?

I've been searching the forum trying to figure this out but without any success. Any help much appreciated...
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2016 December 20 19:13:18
The script is part of the standard install, with the new version released in today's auto update.

In PI's menu bar Script > Noise Reduction > MureDenoise.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: Duncan on 2016 December 20 19:28:21
Thanks for the super quick response! I guess I need to update my install...  ;)
Title: Re: New script for denoising linear monochannel images
Post by: Duncan on 2016 December 25 17:22:17
With everything in my installation straightened out, I've been playing with this script for a few days.

Really pretty awesome! After extracting parameters from flats and darks, it was really easy to set up and get running, and I'm extremely happy with the results. Many thanks for this...
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2016 December 26 07:03:18
Thank you,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: samlising on 2016 December 28 10:07:21
Mike.
I've found this script to be really useful, especially when creating large format prints. One thing I have noticed is that the script can not be used with an "image container" for batch processing. I did a little digging around in your script and found a very simple solution to add a couple of lines to MureDenoise.js file.

   if (Parameters.isViewTarget) {
      controller.setImageView(Parameters.targetView);
      controller.denoise();
      return;

   }

It seems to work, but not knowing enough about the inner workings of your script I was wondering if you could check to see if this is a good solution. If possible, it would be great to add this functionality to a future build.

Thanks for the great work.

Sam
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2016 December 28 11:52:00
Hi Sam,

Thank you. Your solution seems OK for you to use as long as you are careful. Parameter sanity checking is bypassed and this could lead to trouble (e.g. if target image size doesn't match flatfield image size, etc). There might also be issues with GUI messages pausing the container processing. There may also be problems if you try to abort the processing. I will investigate and see if I can find a solution.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2016 December 29 10:04:38
Sam,

The next release will use your solution, along with additions to handle errors and process abort correctly. You will be able to use the script with an image container (be sure to check the process log for errors). You will also be able to drag a script instance icon directly onto a view, with one known exception (dragging the instance icon directly out of the dialog onto a view will not work due to a script API limitation).

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2016 December 29 20:59:34
MureDenoise Version 1.19 soon to be released as an update with several usability improvements.

• "Load variance scale..." loads variance scale directly from the image selected for denoising using metadata produced by the recent ImageIntegration update. No need now for a process log .txt file, except for a legacy integration that lacks the metadata. Thank you Juan Conejero for your help.

• The script now works with image containers. Create a script instance on the workspace and drag it to an image container instance or vice versa. You may also drag the script instance on the workspace to a view. Be sure to check the process log for errors, as no GUI error messages will appear. Abort processing by clicking "Pause/Abort" in the console. Thank you Sam Lising for your solution.

• If "Generate method noise image" is enabled, the log will contain an estimate of component Poisson and Gaussian noise variance in the sky background, defined by the 10th percentile exposure. If the Gaussian noise variance contribution is less than 10%, the image can be considered "sky background noise limited".

• Improved messaging, documentation, and tooltips.
Title: Re: New script for denoising linear monochannel images
Post by: samlising on 2016 December 29 23:16:22
Hi Mike.

That was fast!!! I knew it wasn't as simple as 2 lines. Thanks very much for your support.

Sam
Title: Re: New script for denoising linear monochannel images
Post by: chris.bailey on 2017 January 03 05:10:41
The load Variance Scale from Metadata is great work Mike/Juan. A great de-noise tool just got heaps better still.

Chris
Title: Re: New script for denoising linear monochannel images
Post by: hansheiser on 2017 January 14 13:11:00
Maybe I'm blind but I can't find any file attachment or download link in the forum?
Title: Re: New script for denoising linear monochannel images
Post by: msmythers on 2017 January 14 13:40:23
Mure Denoise is now part of the normal PI distrubution. It was updated around the 2nd of this month with a normal PI update. There is nothing download from the forum. You can check the installed updates within PI.


Mike
Title: Re: New script for denoising linear monochannel images
Post by: hansheiser on 2017 January 14 13:42:01
Ah, okay. My last update of pixinsight was long ago ... Thank you !

Hannes
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2017 January 26 10:50:36
MureDenoise Version 1.20 soon to be released as an update.

• Enabling the use gradient/noise thresholds parameter increases the adaptivity of the denoising process by incorporating interscale gradient to local noise ratio relationships into the thresholding functions. Increasing the adaptivity of the denoising process improves output quality, but also increases processing time by about 40%.

• Performance optimizations, documentation updates, stability improvements, and bug fixes.

Expect subtle quality improvements with gradient/noise thresholding, in both small gradient, non-edge and large gradient, edge regimes as well as in the transitions between them. The increase in processing time is noticable. With the option disabled expect a 10% processing time decrease in comparison to the prior released version.

Thanks,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: jkmorse on 2017 January 28 17:37:04
Mike,

Looking forward to the update.  Didn't comment earlier but the new "load variance scale" is a great add and has really improved the noise reduction on my images.

Thanks,

Jim
Title: Re: New script for denoising linear monochannel images
Post by: Juan Conejero on 2017 January 29 03:06:11
Hi Mike and Jim,

The updates are now available, sorry for the delay. Excellent work as usual, Mike!
Title: Re: New script for denoising linear monochannel images
Post by: mar504 on 2017 February 09 14:00:42
This script is so awesome, wonderful work!!!!
Title: Re: New script for denoising linear monochannel images
Post by: eganz on 2017 March 01 13:55:44
Mike,

I was enjoying the new gradient/noise thresholds parameter method, and it was improving the results for my images.
However, now in the latest version 1.21, this functionality has disappeared.

What's up?

Eric
Title: Re: New script for denoising linear monochannel images
Post by: Buzz on 2017 March 04 08:13:49
I have been having fantastic success with MUREDenoise on my images.  I tried it today and it is repeatedly creating a regular noise pattern on the images. Has something recently changed in the base algorithm? I need to open an old project and see if, indeed, something has changed.
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2017 March 04 16:27:28
Eric, sorry, I found a gradient/noise classifier problem, the option is removed until I find a solution. Within a pixel or two of a sharp, high contrast edge (more common in natural images) denoising tends to be a bit too conservative.

Chris, yes changes (runs faster, less roundoff error, better cycle-spin effectiveness), but should be nothing that introduces this problem. Typical cause is incorrect script parameter values. Can you provide the image in dropbox, along with parameter values? I will run the previous versions for comparison.

Thank you,
Mike
Title: Re: New script for denoising linear monochannel images
Post by: Saad on 2018 February 20 09:16:04
Hello,

I am trying out the Mure Denoise camera for my images. I use a SBIG ST8300M for narrowband imaging (30 min subs) at -15C and 0C in summers.

I have computed a gain of 0.371e- and a gaussian noise of 29.52 DN. However, the result of DEMURE seems to produce blotches in the image. It is not as smooth as it is for others in this thread. What can I do to improve this?



Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2018 February 20 13:30:24
"Smoothness" is somewhat subjective and also somewhat a function of how much you stretch.

Your detector gain and noise values seem reasonable for unbinned 8300.

A couple things to try:

Double check the Image Combination count and Interpolation method settings. Incorrect values cause trouble.

Double check the Detector Offset setting. The default 0 is appropriate only for calibrated images (bias and/or dark subtracted). If the image has not been bias/dark subtracted, then a non-zero value must be used. The DarkBiasNoiseEstimator script can provide an estimate (when run on a bias or dark frame).

Try reducing the Variance scale value from default 1 to 0.9 or less. A value less than 1 will reduce the amount of denoising.
Title: Re: New script for denoising linear monochannel images
Post by: troypiggo on 2018 June 02 16:07:03
I notice that the number of interpolation algorithms in the script's drop-down do not match all of the options that star alignment.  eg Bicubic spline is available, but SA has bicubic spline and bicubic b-spline and so on.  I've been using b-spline is SA.  Results using just bicubic spline as closest match in Mure Denoise appear ok.  Just curious if that's right, or should we limit use of SA algorithms to match the options in MureDN?
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2018 June 02 21:16:31
Troy, what you are doing is fine. When using any of the missing methods, check for excessive denoising artifacts (eg, pixel checker-boarding) and reduce Variance Scale slightly if you see problems. Sorry for the lack of support, I didn't spend the necessary development time thinking that these other methods were not often used.
Title: Re: New script for denoising linear monochannel images
Post by: troypiggo on 2018 June 03 03:37:51
Thanks mate.
Title: Re: New script for denoising linear monochannel images
Post by: mschuster on 2019 February 12 15:13:43
MureDenoise 1.22 release, should appear as an auto-update soon.

When you apply several processes to a preview, the preview's base image is normally used as input to each process. The results from previous processes are normally discarded, unless you use the command Preview > Store.

Up until this new release, MureDenoise did not work this way. If you applied MureDenoise multiple times to a preview (say to test different parameter settings) previous results were not discarded and you ended up denoising a previously denoised result, a bad thing to do. The new release fixes this problem. Thank you ngc1535 very much for finding and reporting this bug.

Also, the new release has updated documentation and tooltips for several of the script's parameters. This should help with ease of use.

 
Title: Re: New script for denoising linear monochannel images
Post by: Juan Conejero on 2019 February 14 01:54:38
I have just released the corresponding updates for MureDenoise 1.22 and its documentation.

Mike, thank you so much for your continued excellent work. Thank you also Adam for your help and support.