New script for denoising linear monochannel images

Mike,

I ran a quick test and got great results as shown on the links below to my before and after images from the Pickering's Triangle portion of my Western Veil image.  Really happy with the result, especially since I really didn't do any fine tuning of the inputs. 

Now my question.  At 2.59 arcsec/pixel, this image just screams for drizzle treatment but I do not see a way to use both drizzle and MureDeNoise.  Am I missing a step that will allow me to use both?  Of course, even if I can't it will be perfect for images from my Planewave CDK where the image scale ranges from 0.43 to 0.73 arcsec/pixel depending on my CCD.

One last point.  I assume in the DarkBiasNoiseEstimator, the number to use is the temporal noise and not the offset.

Thanks,

Jim

https://drive.google.com/file/d/0ByrTommGQM20Vy1pcVFjU1A0U3M/view?usp=sharing

https://drive.google.com/file/d/0ByrTommGQM20NWpfUUJGZGRYNDA/view?usp=sharing
 
Hello Mike
Thanks for sharing this script.
I run a quick test las night on a single and on a a stack of monochrome images form my atik 383L cam.

The noise reduction on the single image is really dramatic.
On the 41 images stack there was an improvement, but not so dramatic as in the single image I asume that this is because SNR is already very improved.

Regards
Geert
 
Hi Jim,

Thank you for posting images! I am glad you got great results.

Drizzle combinations are not supported, no workaround is available. I am working on it, but not yet sure of success.

Yes, use the temporal noise number. I hope to release an update to DarkBiasNoiseEstimator soon with consistent labeling to make it less confusing.

Thanks,
Mike
 
Hi Geert,

Thank you for testing!

Yes, you are correct, the stack has enhanced SNR and hence there is less noise to remove. My head post shows an example, at 1x actual size, the difference is harder to see in the 8 image combination.

Thanks,
Mike
 
Hi Geert,

More info on stack SNR:

When you stack 100 frames, the usual rule says noise goes down by 10x, the square root of frame count. The top blue line below shows this as a function of count. At count 100 noise is scaled by 0.1.

But when you stack 100 frames, 99 of them were registered. The interpolation method used by registration averages adjacent pixels, introduces correlation, and hence reduces noise some. Curves show this for three StarAlignment pixel interpolation methods.

With drizzle the correlation is larger. Adjacent, smaller output pixels are more influenced by the same input pixels due to the fact that they are smaller. The dashed two curves show this. Increasing the drop size increases the correlation and further reduces noise. Also, noise does not drop as fast with drizzle due to the partial coverage.

All of this is relevant to the script, it needs to estimate (guess) how much noise is present in order to remove it.

Thanks,
Mike

p1646119980.png
 
Mike, et al.

Wouldn't most of the problem Mike describes disappear if the noise reduction was applied to the individual images before stacking? Clearly, that order of operations carries a computational penalty, but keeps pixel-to-pixel correlated noise out of the stack, as much as possible. Remediating correlated noise seems far more difficult than does simply avoiding it as much as possible. Also correlated noise fosters the introduction of artifacts by other processing tools that presume it is just not there. It's like trying to get OSC/debayered noise out...not easy because of the long-range correlation of noise that debayering introduces.

Can a script be applied to an image container?

Alex
 
Hi Alex,

I agree remediating correlated noise is difficult and best avoided.

Consider a 0.5 SNR signal in an individual image, a signal likely confused as noise by pre stack denoising, and removed. But in a 100 frame stack of noisy images the signal would be 5 SNR, not confused as noise by post stack denoising, and not removed. Pre stack denoising is not recommended for this reason. In my data this is not a theoretical problem.

With testing (thank you everyone) it appears that MureDenoise underestimates the amount of noise in a stack, and I think this is due to the correlation. And I do not have a solution. The only alternative at this point is to allow the Amount parameter to be larger than one. But this interferes with ease of use because now you have something that needs to be tweaked for stacks. This would allow you to get "more" denoising, but it comes at a cost of potential loss of signal due to the correlation.

Thanks,
Mike
 
Mike

I have thrown a lot of images at it over the past day or so and I would say it works VERY well where the images in the stack are of similar S/N weighting say in the range 0.95 to 1.0. It does a really fantastic job on some of my long sub-frame Ha image stacks! Where it starts to underestimate the noise is when the contributing images are more variable e.g where some of the contributing images have a weighting of less than say 0.85. Now there is an argument these should not be in the stack in the first place but 30 minute Ha subs are hard won when you only get one clear night a month! On these sort of images I have tried re-integrating with weights all set to unity and it does seem to improve NR but with other unwanted side effects. In my simple logic, when the S/N contribution of images in a stack is variable, the quadrature theory starts to break down so this somehow needs to be taken into account.

Chris
 
Hi Chris,

Thank you for your feedback. I agree that this is an issue that needs to be addressed. I am working on it. There is a similar issue due to flat fielding, I am working on this as well.

Also, on the noise correlation issue discussed above, I have made progress on this problem, and was able to fix some bugs. I hope to have a new release of the script in 10 days or so.

Thanks,
Mike
 
The new MureDenoise version 1.8 is available in the head post.

The default denoising performed is unchanged.

The new Variance scale parameter lets you decrease or increase the default amount of denoising. Note that excessively large variance scale values risk the generation of denoising artifacts and the loss of signal-to-noise ratio (SNR).

The variance scale parameter can also be set based on information from an ImageIntegration process log file. The log file is a manually created .txt file containing a copy of the Process Console log generated by ImageIntegration. This feature results in more accurate denoising by taking into account the image output normalization and weighting performed by the combination process. Chris, this feature should help address the issues you see.

In the head post I added a warning concerning the script's inability to remove correlated noise introduced by the image registration process. For combinations, the script estimates the amount of correlated noise introduced by the registration pixel interpolation method, and then reduces the nominal amount of denoising so as to not inadvertently reduce SNR.

Thanks,
Mike
 
Mike

I have run a few of the previous 'problem' images through it this morning using the variance scale set from the Image Integration Log and it is a vast improvement. Typically the Variance Scale on these was in the 1.1 to 1.2 range but that was enough to make the difference. Doing the same on some of the better image run resulted in a Variance Scale of less than one (as low as 0.8) but the end result was better than setting it at unity. Variance scale is VERY sensitive to very small changes so setting it using the Integration Log seems the best approach based on the limited testing I have done, the extra step of cutting and pasting is well worth it.

Chris
 
Thank you Chris, this is helpful information.

Please post a 0 or near 0 log file in dropbox or inline in a post. I will investigate.

Thanks,
Mike
 
Mike

Apologies. For some reason the forum changed zero point nine to a zero point smiley face! I have not seen a variance scale of less that zero point eight five in about 20 images.

Chris
 
OK Chris, thanks.

The range of values you are seeing is roughly what I would expect. On my data I see maybe 0.9+ to 1.2+.

Note that the numbers you see will depend on which image you select as a reference in ImageIntegration. This is because images will be scaled differently by the process with a different reference.

Thanks,
Mike
 
Hello Mike,
thank you very much for this great work. I installed and used it with success. I was able to examine nearly all parameters for my images, even the variance scale. But I am not familiar with the DarkBiasNoiseEstimation script. What image should I provide for darkbias1 and 2?

The standard deviation of detector additive white Gaussian noise parameter is set to an estimate of the quadrature sum of read noise and dark current noise in a 40 minute dark provided by the DarkBiasNoiseEstimator script. Detector offset is set to 0 because the image is dark-subtracted.
Click to expand...

I there a possiblility to determine the StarAlignment pixel interpolation. I have set it to auto, and does not find it in the fits header. Is this information lost?

Thomas
 
Hi Thomas,

Thank you very much.

For DarkBiasNoiseEstimator, please use two good bias images. The script will report "Temporal noise", which is an estimate of the "read noise" or "system noise" of the detector in DN units. Use this value for "Gaussian noise" in MureDenoise.

Regarding StarAlignment pixel interpolation, in my testing Auto means Lanczos-3 when aligning images with the same pixel resolution. So select Lanczos-3 in MureDenoise. You can double check this choice by looking at the StarAlignment process log, where the actual interpolation method used is listed. Note: some PI documentation says Auto means Lanczos-4 for the no rescaling situation, but in my testing Lanczos-3 is listed in the process log. Probably best to double check.

Thanks,
Mike
 
Hello Mike,

thanks for your explanation. Look it seems to work fine with my image. It is a stack of 11x5min OIII frames, registerered with lanzos3 and stacked with average sigma clipping.


 

Attachments

  • MureDenoise.JPG
    MureDenoise.JPG
    108 KB · Views: 223
Mike, all I can say is WOW!  That last example posted by starhopper is amazing  :surprised: :surprised: :surprised: :surprised:

Great work as usual.

Craig
 
You must log in or register to reply here.
Back
Top