New script for denoising linear monochannel images

[pfile]

is it because the pixels have physical separation between them? because there's no interpolation in superpixel debayering...

rob

[mschuster]

I see your point. Depends on how you define interpolation I guess. Each output pixel is a weighted sum of input pixels. This means correlation and problems for the script.

Thanks,
Mike

[jkmorse]

Mike,

Just so you know, I have my fingers crossed on your drizzle integration efforts (I am confident they are way better than 50/50, but that may be a Christmas wish).  I continue to play with and apply the script to older image sets and they are getting a new life from the script.  I am sure you know this and its why you developed the script, but one of the frustrations in all the other denoising tools is the fact that you either lose detail in the target or have to mask and protect the target which means you have an unnatural break between the denoised background and the still noisy target space.  Mure gets around that issue completely and that is why it is so brilliant in execution and incredibly satisfying to use. 

As an aside, at what stage will this become a part of the regular distribution in PI updates like your other scripts?

Best,

Jim 

[mschuster]

Thank you Jim,

The script should be part of the PI distribution soon. Drizzle support is looking less and less like a short term effort (a lot more correlation between pixels). A new version 1.11 (bug fixes, improvements) is available in the head post.

Regards,
Mike

[jkmorse]

Mike,

One thing I am finding is that I am not getting the same level of noise reduction with narrowband images as I do with RGBs.  Is that to be expected and is it a function of the lower SNR?

Thanks,

Jim

[mschuster]

There is less noise in dimmer signals (less shot noise). Maybe as a double check, generate the method noise image, create a few small previews (16 x 16 roughly) in various areas with differing brightnesses, and measure the standard deviation (or better yet Sn) in each. Method noise is the actual amount of noise the script guessed and removed. A bit tedious, but it is possible to compare where more or less noise got removed both within and between images.

Thanks,
Mike

Update: For the KAF-16803 (gain 1.5 e-/DN, gaussian noise 7 DN) this plot shows how noise DN varies with signal DN (from 1 to 65535). For the dimmest signals noise floor is 7 DN. Noise ramps up from there due to shot noise, by more than a factor of 10 for bright signals.



This plot shows the same data as the signal divided by the noise (SNR). Dim signals have low SNR. Dim signals look "more noisy", but they actually have "less noise".

[jerryyyyy]

Hi,

I find this very helpful, at least on one image so far. 

http://www.astrobin.com/233107/B/

Revision done with MureDenoise script instead of TGVDenoise (all other processing the same):

Settings: Gain 0.483 from  FITS
Gaussian Noise 29.51 from DarkBiasNoiseEstimator

DarkBiasNoiseEstimator script also gives Offset 339.25 but not used (as per PI instructions not too clear).  Should this be put into the parameters?  I have another DSW image I am processing so can test it out I guess. 

[mschuster]

Hi jerryyyyy,

Thank you.

Gain seems OK. Gaussian noise seems a bit high. I would expect ~20 DN for bias frames and maybe ~24 DN for 600" dark frames given specs from sbig.com. But 29 DN might be true.

Offset should be zero (the usual case when denoising a calibrated image). The exception is when you want to denoise a raw, uncalibrated frame. Then use a nonzero offset. Basically in this case the denoiser needs to discount the bias or dark frame median as a data "pedestal" or offset).

Thanks,
Mike

[pfile]

mike, these darks have RBI preflash turned on which might affect the numbers. i don't know if you've ever run your estimation scripts on darks with RBI mitigation...

rob

[mschuster]

Hi Rob,

OK, yes preflash would result in more noise and higher estimates. Maybe you could dropbox a pair of bias frames without preflash and another pair with preflash? It would be great to have something to test and compare.

Thanks,
Mike

[pfile]

OK, let me look back thru what i've got - if i don't have anything appropriate i'll make new bias frames with and without

rob

[mschuster]

Thanks for the files Rob.

Noise in the biases (preflash and no-preflash) are about the same, ~7 DN. Maybe with bias there is no time for preflash leakage to add noise?

The average noise in the 600" darks (preflash and no-preflash) is more, ~9 DN, with preflash noise about 15 to 25% more than no-preflash depending on how measured. (Would need pairs exposed identically both ways to get a better estimate, but for this purpose really no need.)

So the 29 DN mentioned above with preflash is reasonable IMO.

Thanks,
Mike

[jerryyyyy]

Guys,

How are you getting the noise?  Different numbers from what I get with DarkBiaNoiseEstimator.... maybe I am using the script wrong?  Seems to compare two random images from set of darks.  I got similar numbers from my STT 8300M and pfile's camera, which is suspicious. 

But, regardless images look good.  When I look in detail at the images there seems to be a sorta checkerboard pattern to the background... noise is reduced overall but there is a funny variance pixel to pixel to my eye.  When you do LHE this is obviously increased.  See screen shot.

[mschuster]

Hi jerryyyyy,

The checkerboard pattern is an artifact due to excessive denoising. Typically one or more parameter is bad: Gain is too small; Gaussian noise is too big; Offset is nonzero (for calibrated images); Variance scale is too big.

To estimate Gaussian noise, pick two good bias frames (temperature matched to lights) and run DarkBiasNoiseEstimator. As a sanity check, you should get a noise estimate (in DN) roughly equal to manufacture specs "read noise" or "system noise" (in e-) divided by "gain" (in e-/DN). Or a little bit larger (say 10%) as detectors get a bit more noisy with age.

Longer light exposures will have additional noise due to dark current and preflash leakage (if any). So rather than using two biases for a Gaussian noise estimate, you can use two good dark frames (time, temperature, and preflash matched to lights). Expect a higher noise estimate. There is more room for error here so I recommend using bias frames to start with.

Thanks,
Mike

PS: Maybe you can dropbox a pair of matching 8300 biases and I will check them out?

[jerryyyyy]

Have packed up some Darks and Biases that I have been using.  Hummm, I was using the darks...  BTW, even with the problems I have had with script my first effort using it on DSW data got IOTD on astrobin.... good publicity for this script. 

http://www.astrobin.com/233107/B/

Can you send me a PM with your real address so I can share the link.