Mure Denoise not effective with reduced number of frames in stack

[astrovienna]

Normally Mure Denoise works like a charm, but on this data it's only working if I include the bad frames in the stack.    Below are links to two XISFs.  The first (integration 3) has 208 frames, and the second (integration 4) has 230 frames.  The only difference is that the extra 22 frames had problems - a bit of haze, less than perfect flats, etc.  Mure Denoise gives substantial noise reduction on the stack that includes the bad frames, but almost none at all on the good stack.

Settings for Mure Denoise are:  Lanczos-3, gain 0.294, and gaussian noise 21.93.  I think you can load the combination count and variance scale from the XISF, and you should see that the 230 frame stack has a variance scale of 0.863 and the 208 frame stack has a variance scale of 1.023.

Can anyone explain what's going wrong?  I'd love to think that the stack with only the good data has no noise, but perfection is a standard beyond my capabilities.   

https://www.dropbox.com/sh/6hnn7a0g53t1b83/AABXBzUwORdtPdzqhnoiJddza?dl=0
https://www.dropbox.com/s/jgg4yvay7py82ed/integration4.xisf?dl=0

Kevin

[mschuster]

Kevin,

The locations (ie medians) of the two integrations are significantly different, ~183DN and ~5086DN.

The script typically handles the varying scales (ie multiplicative scaling) of frames during integration via the variance scale parameter.

Unfortunately large frame location offsets are not handled well (ie not at all) by the scrlpt. There is no workaround.

[astrovienna]

Hi Mike,

I got the "no workaround" part but not sure what "location" means in this sense.  From the documentation, DN seems to be equivalent to ADU, right?  So one integration is significantly brighter than the other?  If you can give me some sense of what I'm trying to avoid in the future, that would be a help.

Thanks for Mure Denoise.  It's a great tool.

Kevin

[mschuster]

Location is the typical pixel intensity (eg median metric). Scale is the typical difference between a pair of pixels (eg Sn metric).

The script works best when all integrated frames share a common location and a common scale. IE, frames exposed similarly in similar observing conditions. Scale differences are handled reasonably well, location differences are handled only when they are relatively small.

So if sky brightness varies significantly across frames, the script will have trouble. By default the integration process adds a constant intensity to each frame to match frame locations. These constant offsets (if not small) mess up the script's noise estimator (detector noise plus shot noise which varies with intensity).

DN = ADU.

[astrovienna]

Thanks Mike. So if I understand correctly, MD will not work if the frames in the stack have a wide range of median values, like if there were clouds rolling through during the exposures.  That's what happened to the integration with the high DN - clouds and bad flats in the extra 22 frames must have greatly increased the average brightness.  But that's also the integration where Mure Denoise worked!  I would have thought it would be the other way round - that MD would fail on this integration and work on the other one, which had much lower DN because all the frames were similar.  Am I misunderstanding something?  Thanks for your patience!

Kevin

[mschuster]

As a sanity check, calculate the amount of noise you should expect to see in the background.

For your smaller stack, location = 183 DN, gain = 0.294 e-/DN, combination count = 208.

Standard deviation of noise in the background in DN = sqrt(location / (gain * count)) = 1.7 DN

So noise in the background is only a couple DN. You may not be stretching enough to see the difference in the denoised result.

You may want "more" denoising. But that would remove signal and lower SNR. The script is designed to try to avoid that.

On your larger stack, the script is not working properly due to the large intensity offsets. The result is basically undefined.

[astrovienna]

Okay, I get it now.  The script had a larger apparent effect on the bad integration, but it wasn't actually working the way it's supposed to.  That probably explains why I thought it was stronger than I expected.  And on the good integration, it appeared to have no effect just because there was little noise in it to begin with.  Thanks.

Kevin