PixInsight Forum

PixInsight => General => Topic started by: dpaul on 2018 February 24 19:36:52

Title: TVG Denoise - creating unwanted contour lines around galaxies
Post by: dpaul on 2018 February 24 19:36:52
I've been using TVG denoise to remove the graininess of the background sky. Lets assume for example I have an integrated set of light frames but its still linear and to protect a galaxy structure's sharpness.

So I create a clone and stretch the clone/do background noise removal to get a dark sky background, then invert to create a mask. I then use this mask to protect galaxy structures from being softened by the denoise process.

I'm using only mild settings - strength, edge protection, smoothness all at 2.0.  The background gets smoothed out nicely but I then notice faint contour lines around the galaxy.

How can I avoid this. am I doing something wrong?  Also is there another way of getting what I want with a different process?

Thanks

David

Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: dpaul on 2018 February 25 14:28:16
Further to my note below, I managed to avoid most of the contour evidence with the right background darkness and background extraction.

Attached is an image of M65 - it was with a first quarter moon.

David

Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: John_Gill on 2018 February 25 23:08:15
Hi,

I see in your image there is a contour line around the galaxy core.  Perhaps try a mask on the entire galaxy and then apply the TGVD process.

Look up
John
Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: dpaul on 2018 February 26 01:47:47
Hi John,

I did use a mask - there is actually some subtle banding around the outside of the galaxy too - more obvious when the original higher quality image is opened up.

Regards

David
Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: Carlos Milovic on 2018 February 26 10:20:45
If you are using the mask, it is not the same as removing the object as it wasn't there. Let me explain further, if you have a very strong light source (i.e. a star) and you have no edge protection, it will be blurred and the surroundings will have larger intensities. If you use a mask to protect the star, you'll blur the image, and then replace the star with the old data. You'll end up with the same increase in brightness of the background.
So, what you need to do is to fine-tune the edge protection, to avoid such artifacts. Using the mask as Local support would also be beneficial since it acts internally to avoid the data "leakage". It is different to a standard mask.
Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: ChoJin on 2018 February 26 11:10:17
For a good edge protection value you might start with the following:

- create a preview over a background area without stars or other stuff
- run the statistics process
- in the parameters of the statistics process add the standard deviation.
- make sure the values are displayed normalized
- use the stddev value for your edge protection value
Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: dpaul on 2018 February 26 16:12:51
Hi Cho,

Thanks for the note.

Just to confirm a few thing:

1/.In statistics do you mean avgDev is the standard deviation?

2/. When I then ''use this value as the edge protection value'' do I simply adjust the exponent and the slider to match that number? 

Regards

David

Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: dpaul on 2018 February 26 16:40:18
Hi Carlos,

Thanks for the reply -
Before I ask some questions related to your note, I have a few fundamental questions:

1/. Which of the following is best:
(a) Use TVG denoise on the R, G, B and L separately?
(b) Channel combine the RGB and then use TVG denoise and separately on the luminance?
(c) Complete the LRGB combination and then (whilst still linear) use TVG denoise?

2/. What should be the order of priority out of the following (and why?):
(a) TVG denoise
(b) MT to reduce star sizes
(c) Deconvolution

3/. When creating a mask to protect a galaxy (for example), I make a clone then stretch it, remove background with DBE then invert it. Should the mask be a clone of a color image (whether single or RGB) or just extract the lightness?

4/. When creating a mask (as in question 3), currently I'm not blurring the edges in any way but I probably should be. I think I should use ''range selection'' on the mask and play around with the 'lower limit' and the smoothness (to blur the edges) - is this the best way?  Also what about star masks, do they need to be blurred?


Now my final questions relates to local support:

5/. When I check 'local support' which support image am I picking, presumably the mask?

6/. assuming I leave midtones, shadows and highlights as default, what about ''noise reduction'' should it be left as zero default?


Many thanks in advance.

Regards

David
Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: ChoJin on 2018 February 27 05:45:34
yes, avgDev, you either just copy the value to the box where you can see the full value (the one on the full left), or you copy it as float and exponent. I prefer to just copy it as is to the left box (and the two other boxes will update automatically)

With my very very limited experience:
- using TGV on linear gives me weird blotches, even after tuning the values for hours. I'm probably doing it wrong. I therefore used MLT following the M81&M82 tutorial from http://www.pixinsight.com.ar/en/info/processing-examples/48/m81.html, and got much better results.

- on non linear, I got quite nice results with ACDNR (using lightness mask enabled on both lum and chroma), I could match the results with TGV, but it wasn't really better (at least to my opinion on the data I tried on) to justify how slower it was. So even after hours of TGV Tuning, I went back to ACDNR with my default values.
Also note that I had run MLT during the linear process, hence I probably get better results on non-linear because of that.
Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: dpaul on 2018 February 27 12:33:07
Thanks Cho

Regards

David
Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: dpaul on 2018 March 01 16:33:45
Hi Carlos,

Did you get chance to see my reply below from February 27th -





Hi Carlos,

Thanks for the reply -
Before I ask some questions related to your note, I have a few fundamental questions:

1/. Which of the following is best:
(a) Use TVG denoise on the R, G, B and L separately?
(b) Channel combine the RGB and then use TVG denoise and separately on the luminance?
(c) Complete the LRGB combination and then (whilst still linear) use TVG denoise?

2/. What should be the order of priority out of the following (and why?):
(a) TVG denoise
(b) MT to reduce star sizes
(c) Deconvolution

3/. When creating a mask to protect a galaxy (for example), I make a clone then stretch it, remove background with DBE then invert it. Should the mask be a clone of a color image (whether single or RGB) or just extract the lightness?

4/. When creating a mask (as in question 3), currently I'm not blurring the edges in any way but I probably should be. I think I should use ''range selection'' on the mask and play around with the 'lower limit' and the smoothness (to blur the edges) - is this the best way?  Also what about star masks, do they need to be blurred?


Now my final questions relates to local support:

5/. When I check 'local support' which support image am I picking, presumably the mask?

6/. assuming I leave midtones, shadows and highlights as default, what about ''noise reduction'' should it be left as zero default?


Many thanks in advance.

Regards

David
Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: wimvb on 2018 March 03 02:36:06
Have a look here:

https://jonrista.com/the-astrophotographers-guide/pixinsights/effective-noise-reduction-part-2/

As for the standard deviation in the statistics tool; if you press the wrench button, you get a menu from which you can choose several statistical measures. Std Dev is in there.

Hope this helps,
Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: dpaul on 2018 March 03 03:27:17
Hi Wim,

That was a 'very' useful thankyou - explains why and not just how.

It clearly shows the order of priority with linear images:

Debanding if required
Background Neutralisation
Color Calibration
Noise reduction (SNCR then TVGDenoise)

Still have a few questions:

1/When should Morphological Transformation and Deconvolution take place compared to the above (and which first, Deconv or MT)?
2/.Until now I've done noise reduction, MT and deconvoltion on separate R, G and B integrated frames (e.g, 20 integrated Red frames), except of course color calibration which is done after channel combination. So I'm wonder if I should channel combine ''first'' then do all the above?

Thanks

David
Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: wimvb on 2018 March 03 08:53:43
Deconvolution should be done in the linear stage, star reduction in the nonlinear stage. However, it doesn't hurt to go away from common wisdom, and test tools in areas where they weren't meant to be used. At least you'll learn when (not) to use a tool.

My normal workflow for rgb after stacking:
1. inspect the r, g, and b masters. If there is a substantial difference in average/median pixel values, I use LinearFit to "align the histograms". Otherwise:
2. combine r, g, and b masters in channelcombination.
3. Crop any stacking artefacts with DynamicCrop
4. CanonBandingReduction if needed.
Cooled CMOS cameras can have horizontal bands when used with short exposures at high gain. If I do get this, I use CBR.
5. DBE with normalisation checked in the correction field.
6. Background neutralisation.
7. Colour calibration.
Nowadays I use Photometric Colour Calibration, in which case steps 5 and 6 are one process
8. Deconvolution. If I also have a luminance master, I only do convolution on this.
Sometimes I get weird artefacts (coloured pixels) around stars if I do deconvolution on an rgb image. If I don't have an L master, I extract a synthetic L from the rgb image and do a deconvolution on this. Later on I then do LRGB combination, in the nonlinear stage.
Not all images need deconvolution, but if you finetune the deringing and regularisation parameters, you can only target the stars in deconvolution, making them a bit tighter.
8. Noise reduction with Jon Ristas methods, TGV and MMT, or a variation thereof.
9. Histogram transformation on L, Arcsinh stretch on rgb. (But sometimes I find that Arcsinh stretch clips the blackpoint too much. I need to refine this step.)
10. Refining the stretch of L.
11. HDRcompression.
12. Make sure that the histograms of both L and rgb are roughly the same, then use LRGB combination with chroma noise reduction.
13. Further stretch refinements and colour saturation increase.
14. Star reduction using Morphology Transform.
This step not only reduces stars, it also dims them. I make a new star mask that only targets the center portion of stars and use MLT with a positive bias on layers 1, 2 and 3, to sharpen the stars a little.

If the background shows colour noise, I use ACDNR after stretching on chrominance only, to neutralise this.
Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: dpaul on 2018 March 03 17:36:26
Wim

Thanks for the detailed reply, much appreciated!

I was doing much of your methodolgy except the following:

1/ Never bothered with Linear Fit, but I will take note in future
2/.DBE I've usually done 'after' background neutralisation and color calibration but is probably causing ''splodgy background'' so I'll try later instead
3/. Deconvolution I do on RGB and Luminance but I appreciate the importance of the Luminance for detail so I'll just try that in future
4/. HDR Compression I haven't been using - it gives great contrast but seems to darken images a lot but I'll experiment more!
5/. Morphological Transformation I've alway done very early in the linear stage - I'll try it later after going non-linear.
6/. Currently I'm combing the RGB color calibrated image and the luminance image in linear state and then doing Arcsinh (or Histogram Transformation until very recently). However I agree the RGB and Luminance are not well matched (Lum dominates) so I'll try combining them after stretching.

Regarding Arcsinh, I do like this tool - so far I've set the black point ''just before'' it starts to crop pixels then adjust the stretch factor until. Interestingly some of my data shows much less cropping than others so this might be a useful indicator of how good the image calibration was done with the dark master (e.g. not setting optimisation too high).

What I've also tried is a less agressive Arcsinh and then do a slight bit of Histogram Transformation - the former gives better colors, the latter may help contrast (I think).

One question on using masks - there are two ways I've done this:

(a) Take a clone image and use range selection to get a mono image (of say a galaxy) then blurr it a bit.
(b) Take a clone image, DBE and stretch it

The otion (a) probably protects the areas you want to protects better and softer transitions than option (b). any suggestions on this?

Thanks a lot

David


Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: RickS on 2018 March 03 21:50:15
A clipped lightness/luminance mask is often useful.  If you have a grayscale luminance then stretch if linear and then clip the blacks with HistogramTransformation.  If you have RGB then extract CIE L*, stretch if linear, and black clip it.

These masks are good for noise reduction (inverted), saturation boost for RGB images, sharpening, etc. 

Cheers,
Rick.
Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: dpaul on 2018 March 04 01:38:01
Rick

Thanks for this, I've never used such a mask but will try

David
Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: dpaul on 2018 March 04 17:48:19
I've just been using a mask to protect a galaxy (M66) whilst using MLT for noise reduction. This was on both RGB al L images (still Linear).
The masks in each case were created by creating a clone then using range selection to only include the galaxy. I also made is blurred  it with the smoothness slider.
In both cases I still some subtle contours - see attached images, the RGB is still linear and the L is stretched but in both the contours are visible.

How can avoid these contour lines?

Thanks

David

Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: wimvb on 2018 March 05 00:23:47
Did you apply deconvolution before noise reduction? These contours around saturated stars are an artefact of deconvolution.
You can try noise reduction on a non-deconvoluted image.
The method described by Jon Rista has never resulted in such artefacts, as far as I know.
If you have the possibility to attach/link to the stacked but otherwise unprocessed lum master (xisf format), I can have a look at it.
Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: dpaul on 2018 March 05 02:56:26
Hi Wim,

I think I did deconvolute before noise reduction (could be wrong). I also made a mask to protect the galaxy by using arcsinhstretch on a clone and stretching / black level until everything but the galaxy was clipped. This I believe may have also create weird contours.

Either way, I've had another attempt, definitely convolution done first. Also a mask created with a light extracted clone then range selection to just show the galaxy and made is blurred with the smooth slider.

Tthe attached result seems a lot better, no contours.

The only issue I still had was creating a star mask and doing MT to reduce star size. No matter what star mask settings I chose, I couldn't include the brightest star near the galaxy. So some of the stars are still a little bloated but at least not too over-processed now. Advice on star masks would be useful.

Thanks

David
Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: wimvb on 2018 March 05 04:07:18
To include stars on a bright background:
Extract luminance from the stretched image. Apply HDRMT with layers down to 3. Then create a star contourmask from this image. You may need to blur (convolution) the image a little to produce a good mask.
General settings for contourmask:
Large scale and small scale = 0 - 1
Compensation = 1
Smoothness = 8 or less
If you need to soften a contourmask, it's faster to do so afterwards with convolution.
Title: Re: TVG Denoise - creating unwanted contour lines around galaxies
Post by: dpaul on 2018 March 05 15:16:54
Hi Wim,

Thanks for the note -

Still struggled to get a decent star mask from the stretched luminance image. HDRMT did help reduce the dominance of the galaxy core but I still barely detected the nearby brightest star.

In the end I redid the whole thing (to get better colors, etc) and taking a bit more care. Everything was in the order that you suggested except I did MT just prior to stretching which was much easier with the star mask generation. Also I did arcsinhstretch for the RGB then a little bit of HT too to stretch a bit more - better contrast. It meant the RGB was more balanced with the HT stretched luminance Image. The end result seems to be better and stronger colors.

Incidently, I tried PCC and color calaibration and the latter was the one final chosen, the colors seemed better 'on this occasion'. Sometimes I get the opposite.

See attached final image, not perfect - still a bit noisy. The reason is that the mask I used for the galaxy when reducing noise was probably too wide - better if I'd just protected the inner spiral arms.

Thanks

David