Noise Reduction in Low Signal Areas Only

[sreilly]

Just the other day I posted an image of the Iris Nebula and one suggestion I got on another forum was to use an inverted star map and use Noise Ninja to smooth the dimmer low S/N areas. While I have Noise Ninja I really don't like it's affect and think that PI has better noise reduction processes. With that in mind, any suggestions on reducing the low signal areas while leaving the high areas alone? The image in question is posted in this gallery posting http://pixinsight.com/forum/index.php?topic=7386.0

Thanks,

[jkmorse]

Steve,

Great image, I only wish I had some ideas beyond the standard of trying MLT and tweaking the noise reduction settings in the various layers.  I will be interested in the responses you get. 

I did have a couple of questions for you though, based on your first post.  First, you mentioned that just using RGB data did not provide enough detail.  I assume those were shot binned 2x2 and that was why you didn't try creating a synthetic luminance frame?

Second, I do not have cosmic correction as a part of my workflow.  Does it really help even though you have calibrated your images first?

Thanks,

Jim

[sreilly]

Jim,

The original image had 13 fifteen minute luminance images used and that added a good deal of noise to the low s/n areas which is basically anything outside of the bright reflection nebula. Adding more to that, a total of 12.25 hours of luminance data helped bring out the very faint areas by allowing more aggressive stretching. I never use a synthetic luminance image. Guess I'm too old to change but if I'm going to go for luminance I just take them. I use ACP for observatory automation, have a rotator, and weather safeties so leaving my system running while I go to bed doesn't bother me in the least. In this case I just added several nights more of luminance imaging. And yes, the RGB data is binned 2x2. All images shot for 15 minutes at -15C.

What I use to do is calibrate my images in MaxIm and then bring them into PI to finish. I'd create my master calibration frames in MaxIm, calibrate the images, and then run a plug-in that allowed me to remove hot/cold pixels. Real easy and then I'd save the lot. I do not particularly like using the BatchPreProcessing script for several reasons. Mainly I'm wanting more control and look at the results while processing. So I calibrate all my east side of pier images using the Calibrate Image process with appropriate master frames and then the west side of pier images. I use a rotator so I have E/W master flats for each filter. After calibrating the images I'll open them up and inspect for any defects that may be present from say a not so good flat or dark. If I'm happy with those I save the bunch. Then I generally cull the images 1st based on FWHM and then visually check them for any problems saving only the best of the lot.

Next comes the Cosmetic Correction. I load my high quality Master Dark frame and check both the Use Master Dark and Use Auto Detect boxes. Open an image and use the Real Previewer to see the effect. For large images make a preview of an area with lots of hot/cold pixels and use this for the preview. Usually the default values of 3 work best for both (they need to be checked). Set the path to save the images and run the process. Again I'll check some of the images to be sure it did a good job. Next, when you combine the images you should see a much cleaner result, at least I do. I dither my guiding so this really helps. Between CosmeticCorrection and combining the images, you should get a very clean master. It has made a big difference for me.

One thing I did find is that when calibrating my images I needed to use the flats marked as East for the West side of pier images and opposite for East side of pier images. I can only assume that MaxIm, the software used to acquire the images, and ACP, the automation software, see side of pier opposite. But hey it works after I finally figured it out. I must also say that my flats work much better in PI than in MaxIm. I create all my master calibration frames in PI.

At this point I do all my processing in PI and am feeling somewhat comfortable. There's a lot I don't know and probably never will but my processing is far better now then it's ever been and I give PI credit for giving us the tools to do better with much more control. I'd love to see layers which I think is being worked on but then I was never much of a layering guy in that other program. But I do see where it's strengths could be. I'm just happy they continue to grow PI. I can't imagine what it will be like 5 years from now!

[jkmorse]

Steve,

Thanks for the explanation.  I was in your camp for years, but a long discussion with Juan in a thread got me imaging using 1x1 RGBs and a synlum to maximize the number of frames (and I still take at least two nights to do one target anyway).  I have generally been happy with the result though it is probably time to run some comparison tests.

Have to say I am really envious of your setup.  I am still a few years from retirement but once I take the plunge, an observatory is first on my list.  Have you always been an ACP user or have you tried others, such as CCD AutoPilot?  And how much tweaking did it take to get everything working?

I will definitely try out your workflow with cosmic correction to see if it helps with my rig.  Since we are using similar imaging equipment I would guess I will see positive results as well.

Thanks for the great response and I couldn't agree with you more about how great it is that the team keeps making PI better and better.  I have a cribsheet you may have seen and am up to Rev26a just trying to keep up.  Their dedication and the results they produce are a joy.

Best,

Jim   

[sreilly]

I use to do RGB binned 1X1 but got away from that. If I had more pristine skies......but not sure. I'd like to hear the reasoning behind the binning argument but I suspect that seeing versus focal length would make a difference.  My image scale with the RC/STL-11002M is .64 at bin 1 and 1.28 at bin 2. My skies at best may get to 1.8 or so and probably average 2-2.5 FWHM. The data used in the Iris was all below 2.5 and I only had to throw out a handful but those were good to excellent nights. Some nights I only got a few hours before having to shut down.

When we bought this property some maybe 15 years ago, home and observatory was built 11 years ago, I was driving my agent nuts. I insisted on dark areas without a lot of development. What I bought into was a 10 acre lot, wooded, in a 140 acre development that only had 10 acre lots. Outdoor lighting is minimal throughout the development and none existent after midnight. My fist night time trip to the property was after an outreach event 12 miles away from the property. We had left after midnight and I decided even though it was out of my normal way home to have a nighttime look. We had just dug the hole for the walkout basement. It was so dark I could see the stars reflecting off my truck hood and it was pitch black with great summertime Milky Way definition. I got chills, wasn't New Mexico or Arizona, but is was darn nice for Central Virginia. The observatory was built while the house was going up. A friend who was a carpenter was assisted by me and it became what it is.

If you don't get the results with CC post and at least see what may be happening.

[sreilly]

What was proposed for noise reduction using another program was "Make a copy of the image so you have two layers. Run Noise Ninja or similar noise reduction algorithm on the top layer then add a hide all layer mask.  Copy the bottom layer (ctrl-a, ctrl-c) then hold down the Alt key and click in the layer mask. Paste (ctrl-v) the image in the layer mask.   This gives you a black and white version of the image as a mask with the light parts white and the dark parts black. Invert the image so the dark parts become white and the light parts become black. You can bring up levels and move the slider to the right to taste. Since white reveals and black conceals, you will let the noise reduction through in the dark parts of the image, but block it from the lighter parts."

I had been familiar with this type of technique before although not using this particular noise reduction plug-in in this manner. As great as Pix Insight is, with all it's abilities, I have no idea how this type of process can be accomplished in PI. Is this possible with PI as it is at this point or is the need for layers holding me back from being able to accomplish this. Or is there a way I'm not seeing that will allow very fine control over noise reduction on very selective areas?

I tried an inverted star mask and using TVGDenoise but that worked on the entire background, i.e. all but the stars and that softened the nebula itself losing detail. Is there a way to make the nebula a mask, invert the mask, and reduce noise in the rest of the image?

Any thoughts on this would be great. Even though I have long paid for other software and plug-ins for them, I really want my images to be 100% PI processed. I've argued that PI is the tool of choice for astro-imaging and I strongly believe this. Of course that also depends on one's knowledge of using this great software.

Thanks for any suggestions. The link to the image page is http://www.astral-imaging.com/NGC7023-RC2.html

[Jason Tackett]

Hi Steve,

Recently I had some luck making a nebula only mask using the following procedure.

1. Create a mask using RangeSelection which selects the nebula and stars
2. Create a star mask
3. Remove stars from range_mask using the PixelMath expression, range_mask - star_mask. At this point the nebula will be bright, noise in the background will be grey and the stars will be black.
4. Apply a contrast curve using CurvesTransformation to darken the background and brighten the nebula.
5. Blur the image by deselecting the detail in all wavelet layers except the residual layer with ATrousWavelets & apply to the image
6. Repeat steps 4 and 5 until there is a smooth mask where the nebula portion is bright and the background is very dark.
7. Subtract the stars out of the range mask again using PixelMath: range_mask - star_mask

This creates a mask that selects the nebula alone. I think this worked well to target nebulosity in my images because they are noisy, so by iteratively applying contrast curves and smoothing, I effectively darken the background of the mask (removing the noise along the way) while still preserving the transitions between the nebula and the background.

If you just want the background in your case (which is awesome, BTW), skip steps 2, 3 and 7. This will give a mask selecting the nebula and stars which can be inverted to select the background.

As an aside, the stars could have been removed from the range mask by adjusting the upper limit slider in RangeSelection, making step 3 unnecessary. Not sure why I didn’t do this, but the result worked well and I always look for an excuse to use PixelMath!

Hope this helps,

Jason

[jkmorse]

Jason,

Nice routine! I will give that a try. 

Best,

Jim

[gvanhau]

Hello

You can use a stretched inverted luminance of the image as mask, and then apply the noise reduction tool of your choise.
Note that ADCNR has its own lum mask.

Geert

[JoLo]

Yeah, I like using the lightness mask in ACDNR and is similar to results i used to get with an inverted object mask in Photoshop.  There is a lot of control over the contrast in the mask, then HT can be used to clean it up.

I have also used the procedure Jason refers to and it works well, but I have recently been using CloneStamp instead of the PixelMath approach.  Using the settings in RangeSelection can get you most of the way there, then the CloneStamp can clean up any residual.

I often, right after LRGBCombination, make a strong luminance mask and then use CloneStamp to make a mask of the object only, and a second with everything but the object.  The first can be inverted or not, to protect object, or not.  For the second mask, I have a good star mask that can be put to use.

I know I am bordering on heresy here, but I will say I still find the level of control over masks to be superior in Photoshop, especially feathering edges and such.  Now, I am completely converted to PI (put the bonfire out!) and am getting better at mask control and particularly edges in PI but I still find it easier and more intuitive in PS.  Trial and error with the RangeSelection tool can produce interesting results.

My current approach to NR is to use MMT over the whole image in the linear state (a very, very light touch of MMT) and then use ACDNR and its lightness mask in the non-linear state close to the end of the workflow.  I find this gets good smoothing, without it looking like it was smoothed.

Each image is different and I enjoy the dance between NR and sharpening the data.  Sometimes it takes me three or four runs through PI before I am happy with it, but mamma, that's where the fun is!

BTW, I do all subs 1x1 and only do a syn lum if i run out of time.....works for me!

[Juan Conejero]

You can use a stretched inverted luminance of the image as mask, and then apply the noise reduction tool of your choise.

In the latest processing example I have uploaded to our website I describe the new linear mask feature that I implemented in the multiscale processing tools (MultiscaleLinearTransform and MultiscaleMedianTransform). Jump to the Noise Reduction section of the tutorial.

Inverted lightness masks have been used routinely as noise reduction masks in PixInsight since the first version of PixInsight LE (2003). You can use a variety of tools to stretch and clip a duplicate of the image (or an extracted brightness or lightness component) as necessary, including HistogramTransformation, CurvesTransformation, PixelMath, RangeSelection, etc. StarMask is not required at all for a noise reduction mask.

A linear mask is much more efficient for noise reduction of linear images than a stretched nonlinear mask. By preserving linearity of the mask we ensure that noise reduction is strictly proportional to the SNR of the image. A linear mask can be generated very easily with PixelMath, but you can also use MultiscaleMedianTransform with the Linear Mask and Preview Mask options enabled.

[Juan Conejero]

...make a strong luminance mask and then use CloneStamp to make a mask of the object only...

Once you understand and gain control over the signal and the noise and how they are distributed and interrelated, you no longer want to apply this kind of arbitrary manual manipulation in PixInsight. Image processing must be applied algorithmically based on properties of the data. Of course this requires you to understand your data first in order to identify and solve the actual problems posed by each image.

The CloneStamp tool is intended to repair small and easily identifiable defects, exclusively when there can be no doubt that by doing that we are not creating artifacts or removing existing significant data.

...in PI but I still find it easier and more intuitive in PS.

Do these easiness and intuitiveness include painting masks by hand? These "procedures" pertain to the retouching and painting field; they have nothing to do and actually are opposed to image processing.

[jkmorse]

This is why I got a dedicated machine and purged all the other evil easy solutions from my toolbox.    PI makes me feel like an idiot most of the time, but that just means I haven't mastered it yet. 

Of course, listening to Juan and the debate, clearly there is a bit of a religious schism at work here too.  There are easier ways (sometimes) but they are not pure to the data.  As an aside, that's one of the reasons I struggle with the concept of BPP, it shortcuts what should be carefully handcrafted; but that is just a personal view that makes me sound like my dad ("when I was a kid we didn't even HAVE shoes and had to make them out of tree bark. . .")   

For what its worth (not much),

Jim

[jkmorse]

Steve & Juan,

First Steve.  Not sure if it will answer all questions but I did come across this link that looks like it may be helpful since it seems to be right on point to your query: http://dl.dropbox.com/u/57910417/Howto_enhance_nebuala_without_pushing_stars.wmv

For Juan, I typically build my masks for noise reduction by cloning my image and then adjusting through HistogramTransform and CurvesTransformation, etc., which you mention in your note below.  But you also state that

A linear mask can be generated very easily with PixelMath.

 

For those of us who are not sufficiently math savvy, can you elaborate on how to go about building a linear mask for linear noise reduction using PixelMath.

Thanks,

Jim

[gvanhau]

In the latest processing example I have uploaded to our website I describe the new linear mask feature that I implemented in the multiscale processing tools (MultiscaleLinearTransform and MultiscaleMedianTransform). Jump to the Noise Reduction section of the tutorial.

A linear mask is much more efficient for noise reduction of linear images than a stretched nonlinear mask. By preserving linearity of the mask we ensure that noise reduction is strictly proportional to the SNR of the image. A linear mask can be generated very easily with PixelMath, but you can also use MultiscaleMedianTransform with the Linear Mask and Preview Mask options enabled.

Very interesting, can this linear mask be exported in order to use it in other processes?

Geert