Can you help troubleshoot my TGV Denoise steps?

[pvelez]

I've recently collected some good data and have appreciated just how deficient my skills are in PI - one of the key areas I am struggling with is noise reduction. So I thought I might post a few screenshots showing how I am going about the process and let the gurus tell me where I am going wrong.

I've reviewed the excellent tutorial on Noise Reduction posted here - http://lightvortexastronomy.blogspot.com.au/2015/08/tutorial-pixinsight-noise-reduction.html - sadly, I can't achieve the same results. If I had some pointers on how to tighten up my processing, I wouldn't feel as if I am simply engaging in trial and error fiddling with the settings. (I might add I have also dabbled extensively with the Multiscale Linear Transform process following the same tutorial and the one posted by Juan here - http://www.pixinsight.com/examples/M81M82/index.html#Noise_Reduction - again, without being able to replicate the results).

So here goes.

I have an image of NGC 6744 comprising 114 x 5 minutes subs ie total integration time of just under 10 hours. They have been calibrated (bias, dark and flat) and combined using the Winsorized Sigma Clipping setting. I've then applied cropped the integrated image and applied DBE. The image looks quite good but there is plenty of noise in the background. I've set up a preview from the edge of the galaxy which has some faint spiral arms and a reasonable amount of background. The first screenshot shows part of the image with the preview frame indicating what I am working with.

I'll post the remaining steps and images in succeeding posts

Pete

[pvelez]

Part 2 - Using some clean background and the Statistics process, I have found the standard deviation is around 1.3 x 10^-5 so I have set that as the Edge Protection parameter. Strength and Smoothness have been dropped by an exponent. I have taken a copy of the image and applied the STF to the Histogram Transform rendering the copy non-linear and set that as the Support Image. Iterations are increased to 250. The second screenshot shows the preview with settings before being applied.

Pete

[pvelez]

Part 3 -The third screenshot shows the after. I have large dark blotches in the background and there are clearly steps between different parts of the background. Not good at all.

[pvelez]

Part 4 -

So I lowered the Strength from 5 to 2. The next screenshot shows the result. The blotches have diminished in size but are still to be seen. There is still plenty of steps between darker and lighter areas and the overall reduction of noise is diminished - as you'd expect.

[pvelez]

Part 5

My next thought was to return the Strength setting to 5 and to increase the Smoothness - this was to smooth out the steps between darker and lighter areas. The next screenshot shows the result which is worse than all the earlier attempts. The steps are more pronounced as is the distinction between high and low signal areas.

[pvelez]

Lucky last!

So I thought I'd drop the Strength back to 2 and keep the Smoothness setting high - this time at at 4.15. The result is the final screenshot. The step artefacts remain and the noise is not reduced much.

Where do I go from here? I expect that if I want to increase protection for the high signal areas and increase impact on the background, I'd tinker with the support image by clipping the shadows and raising the mid-tones. But I don't think either protection of high signal areas or strength on the background is the issue. Perhaps I have unreasonable expectations about how much noise reduction I should be able to achieve - the Preview produced with Strength at 2 and Smoothness at 2 seems the best of the bunch but I'd like some more reduction than this if at all possible. Or are there other adjustments I could make?

I appreciate that its not easy to judge with Screenshots and I can post larger images if that helps. Alternatively, I could upload the original file for others to tinker with. The key for me is to understand what is reasonable to expect from the Process and then where I can improve my workflow to achieve those results.

Cheers

Pete

[pvelez]

If you'd like to have a play with the original data - here is a link to the file

https://www.dropbox.com/s/jr6cizsrfnolhjw/NGC%206744%20Lum.fit?dl=0

Pete

[msmythers]

Pete

I didn't use TGV but MultiscaleLinearTransform. I have an easier time with it but that is just me. I noticed some pixelation in your image that goes away when using the 24 bit lookup table selection for STF. Here is a screen shot of your last image and mine next to it. The MLT settings are shown in the image. Again notice that I have the 24 icon selected on the top toolbar. There is an excellent example of MLT in the M81/M82 example in the main PI website. http://www.pixinsight.com/examples/M81M82/index.html


Mike

[pvelez]

Mike

that is brilliant! Thanks for that. I've been able to replicate that on my machine so it really does work that well.

I have never heard about the 24 bit lookup selection for STF. What does it do? I see that its the difference between the nasty pixellated blotches I had and your smooth alternative. I'd best do some research on that now - and head back to the rest of my data. Looks like a busy weekend ahead of me

The M81 and M82 example was one I had tried to follow but couldn't get an acceptable result. Will have another play with it now. Interestingly, using the Real Time Preview function, I can't see any effect of the 24 bit lookup option - but I certainly do on the finished product

Pete

[pvelez]

So I've had a play with MLT on a few images since the weekend and have found that the 24 bit Lookup table function improves the performance dramatically.

So thanks again Mike.

Interestingly, I've found that TGVDeNoise is also improved but not by as much - and it has no discernible impact on Deconvolution.

I have a follow up question - for Mike and anyone else who'd like to respond - MLT provides a linear mask option. However, even when the slider is pushed to the end (turning it up to eleven as they say in Spinal Tap), the preview indicates that it is only the cores of stars and the brightest structures that are protected. The M81 tutorial suggests that the linear mask option is what makes MLT such an excellent process. But the Light Vortex tutorial promotes the use of a non-linear mask ie duplicated luminance image that is stretched and used.

Which is better? Or if it's a case of sometimes one and sometimes the other, when is it best to use the linear and when the other?

Pete

[jkmorse]

Pete,

Whether to use the built-in mask or a separate non-linear one all depends, at least for me, on the point you touched on, namely can I get the MLT mask to provide sufficient cover for what I am trying to protect.  For many images, it does a great job but for some the range is just too narrow, on either side of the equation (maybe something that can be addressed in a future update).  In those situations I always fall back on a separate mask and just turn the built in mask off.

Best,

Jim

[msmythers]

Jim and Pete

Here is something I have noticed with MLT and the linear mask. Remembering we are linear we normally are looking at the image to be worked on with STF applied. This affects the linear mask in MLT. With a regular non-linear mask you can apply more or less brightness to areas to protect. Since we are linear I don't think applying a non-linear mask is necessarily the best choice in non-linear noise reduction. Just my opinion.

Here is a screen dump to think about. It is a linear image with the MLT linear mask showing and SFT applied on the left side. On the right side are the same settings only SFT is not applied. Notice how different the mask is.


Mike

[jkmorse]

Mike,

Interesting and will need to test since I hadn't realized that before.  There are times, however, where even at 1000, the MLT mask is all white (or at 0 all black).  In those instances, the non-linear protection is the only option and I have not seen that it creates any noticeable downgrade in MLT's performance.  That said, I always prefer to use the MLT mask whenever possible.

Best,

Jim

[msmythers]

Jim

I've not had any images come close to going to 1000. The most I've had is 500 and I would say the normal range for my images is between 100 and 200. The only time I've had something even close to 0 is with linear images. If you have many saturated stars in a linear image that might drive the amplification very low I would think. Maybe check the statistics on those images where you need a very low or a very high amplification value, I'll bet you might be clipped or very close to clipped on the high or low end.


Mike

[jkmorse]

Mike,

We may be talking past each other but I want to make sure since you say the only time you come close to 0 is with linear images.  To date I have only used MLT with linear images and have not tried it after stretching.  Therefore no risk of clipping because they are still the raw, unstretched data. In the non-linear phase I always convert to TGVD, ATWT or MMT for noise reduction. 

In that regard, do you regularly use MLT in the non-linear phase and if so how to you find it compares to the others?

Best,

Jim