Gradient Domain Operations, take 4

[georg.viehoever]

Dear all,

here is another release of this module. A summary of the changes:

- I have improved the GradientsMergeMosaic module:

-- It is now much more resilient to to stars that are located at the borders of the contributing images. This problem was demonstrated by Harry in http://pixinsight.com/forum/index.php?topic=3211.msg22213#msg22213, and so far we had only a manual workaround http://pixinsight.com/forum/index.php?topic=3211.msg22244#msg22244. The improved method does not show this problem to this high extend as shown in the attached screenshot 1.
--- All problems at seams are gone, except for the slight problem at preview1. I am afraid this will need to be handled by the clone stamp method.
--- The remaining issues are where the algorithm tries to merge pictures with the background, i.e. at the border of the merged image. I am not yet sure how and if to handle this problem. For the moment, I guess, most people will be happy to sacrifice a view pixels in return for a perfectly matched inner region.

-- Another problem were aliased pixels at the border of the contributing images. The algorithm cannot match such images, as in cannot tell if such pixels at the border are intentional or artificial. This problem was experienced by Geert http://pixinsight.com/forum/index.php?topic=3211.msg22335#msg22335 and David http://pixinsight.com/forum/index.php?topic=3211.msg22498#msg22498, http://pixinsight.com/forum/index.php?topic=3211.msg22541#msg22541. The source of this problem can be aliasing that happens during interpolation (e.g. when sizing or rotating an image), or for example by wavelets or convolution.
--- The module now allows to "shrink" the border of an image using the "Shrink Count" parameter. Effectively, this will move the border of a contributing image by a n  pixels inwards, effectively eliminating the critical pixels. However, this requires an overlap of at least 2*n pixels between neighbouring images.
--- The second attached screenshot simulates this problem using Harry's NGC7000 mosaic. I blured both contributing images using the Blur process. You can nicely see the blured border in the closeup on the bottom left. Merging these images with the default Shrink Count of 1 leads to the result on the top left. Using a Shrink Count of 3 gives the image on the top right.

- No changes to GradientsHdrCompression.

- I also merged in the changes by Zbynek http://pixinsight.com/forum/index.php?topic=3211.msg22046#msg22046 and Juan http://pixinsight.com/forum/index.php?topic=3211.msg22047#msg22047, so with a little bit of luck this module should compile properly on all platforms supported by PI. However, I have no way to test this. This version was tested and developed on Fedora14-x64

Maybe Juan considers publishing compiled versions as a decent holiday activity  . If not, I guess you'll have to wait a little bit or compile yourself.

Any feedback welcome  8)

Georg

[David Raphael]

Awesome!  I will compile this and give it a try when I get home this evening.

Best,
Dave

[georg.viehoever]

Hi,
one more observation I would like to share with you:

• Occasionally, the current implementation still has issues with stars at the seams between two images - even if it is now much less of a problem than with the previous implementation.
• You saw such a problem in the "Preview 1" region of my previous post. In the attached screenshot, you see this situation on the left hand side with a close-up at the bottom left.
• On the right hand side of the screenshot, this problem magically disappears. How was this done? The answer is quite simple:
  
  • For the left hand side, I merged the linear images, before the non-linear histogram stretch.
  • The right hand side was merged after stretching the contributing images first.
• The problem disappears because in the images after the stretch, brightness differences of bright stars are much smaller than in the linear images.
• This procedure also signicantly reduces the artefacts at the border of the merged image.

I guess the consequence of this is the recommendation to do the GradientMergeMosaic step always after the non-linear stretch.

Georg

[Harry page]

Hi

Forward progress 

Look forward to having a go when Lord Juan returns 


Harry

[Juan Conejero]

Hi Georg,

I can build binaries for Linux 64/32 and Windows 64/32. However, I can't for Mac OS X, as I have no Mac machine here and Mac OS X cannot be virtualized (and I am a legal guy). I'll try to do this as soon as possible. If somebody else (David?) can build the module for Mac OS X, then we can upload all the files to the development repository.

I guess the consequence of this is the recommendation to do the GradientMergeMosaic step always after the non-linear stretch.

This is a serious limitation that we have to solve. Mosaics must be generated as linear images, or otherwise the possibilities to process these images would be severely reduced.

A stupid idea (of those that use to work): could you simply exclude the stars from the data that feeds the matching routine in the gradient domain? In other words: just don't try to match the stars; just match the background 'and a little more'. This could be done with the help of a simple binary mask built internally by the process. Sounds feasible?

[georg.viehoever]

Hi Juan,

thanks for doing this during your vacation  .

Regarding the star problem. I think there are two different situations here:

- A: stars at seams within the mosaic. The problem is caused when the two side of a star that extends across a seam have different profiles/different max. brightness values. MergeMosaic tries to smooth out the differences of both sides, causing a bright halo on the one side, and a dark halo at the other side. I currently have two ideas (not yet implemented) to handle these:
--A1. use something like a star mask that would attenuate the effect of MergeMosaic around starts, e.g. by reducing the gradients in a non-linear fashion around stars. This of course would destroy the linear character of the image.
--A2. use something like feathering http://en.wikipedia.org/wiki/Feathering at the borders of the contributing images. This would smoothly blend from the gradients of one image to the gradients of a second image, giving us the additional benefit that even transitions that cannot be smoothed by gradient merge (such as noise or background patterns) would be somewhat smoothed.
--A3. use different solver that would allow to exclude certain regions. For this the remarks in B3. below apply
--A4. manual clone stamp as demonstrated in http://pixinsight.com/forum/index.php?topic=3211.msg22244#msg22244 .

-B: stars at the border the background region of the mosaic. They are caused by MergeMosaic trying to smooth out the gradients between background (which has gradients 0 in every direction), and the signicantly different gradients of the image. The effect of this is most visible when a star is located there. While this is annoying, it can usually be tolerated by just throwing away a couple of pixels close to the border. Anyway, here are my ideas for handling this:
--B1. replicate the mosaic in a mirrored fashion, such that the is no background anymore. I am not yet sure on how to do this in a consistent fashion, but it would certainly remove any of those effects.
--B2. use feathering as described above.
--B3. use different solver: The current solver (DCT based) requires solved regions to be rectangles. Using sparse linear solver would enable us to handle arbitrary regions (that would for example exclude the background region). Based on my previous experiments with iterative solvers, I think these would probably not be fast and accurate enough for astronomical images (for daylight images which are handled by most panorama tools, 8 bit accuracy is ok and can quickly be achieved, but handling the dynamic range of floats or doubles as used for astronomical pictures is an entirely different story).
-- B4. manual clone stamp as in A4.

I would love to hear if someone has different ideas on how to handle this. I currently favor the feathering approach.

Georg

[Juan Conejero]

thanks for doing this during your vacation

Thank you for your excellent development work.

--A1. use something like a star mask that would attenuate the effect of MergeMosaic around starts, e.g. by reducing the gradients in a non-linear fashion around stars. This of course would destroy the linear character of the image

While we can't find a cleaner solution, this is the best option IMO, from a practical point of view. This breaks linearity, but only for the objects that are being protected and, if the whole procedure is being applied correctly, the differences should be negligible.

By 'applying the procedure correctly' I mean that the mosaic frames should always be matched with a linear fit (as SA's frame adaptation feature) before applying a gradient domain match. Then we only have to fix residual additive gradients, which should be weak. Naturally, we assume that the images have been accurately calibrated as a precondition; otherwise trying a mosaic is a nonsense.

[Carlos Milovic]

Regarding the star problem. I think there are two different situations here:

- A: stars at seams within the mosaic. The problem is caused when the two side of a star that extends across a seam have different profiles/different max. brightness values. MergeMosaic tries to smooth out the differences of both sides, causing a bright halo on the one side, and a dark halo at the other side. I currently have two ideas (not yet implemented) to handle these:

One thing to consider is that image quality is not the same across the entire field (well, in most cases). So merging two images with the same weight along the whole intersection is not the best approach, IMO. I would consider a scheme that gives more weight to the pixels that are closer to the center of the image, and fade that function to zero at the boundary.  Perhaps we may use another weight too, from information of the intersected area (feather), like a linear function perpendicular to the skeleton of that area.
Also, it is to consider the blending method. Is a weighted average the best method? What about multiplying and then taking the root? Or other method based on statistical properties? I've seeing in a paper that they used the maximum (gradient) value.

I would not follow the star masking path.

-B: stars at the border the background region of the mosaic. They are caused by MergeMosaic trying to smooth out the gradients between background (which has gradients 0 in every direction), and the signicantly different gradients of the image. The effect of this is most visible when a star is located there. While this is annoying, it can usually be tolerated by just throwing away a couple of pixels close to the border. Anyway, here are my ideas for handling this:
--B1. replicate the mosaic in a mirrored fashion, such that the is no background anymore. I am not yet sure on how to do this in a consistent fashion, but it would certainly remove any of those effects.

Yes, indeed this should solve a lot of artefacts at the borders. See the code of HDRC. There I have a function called Padding, that performs mirror padding for a user defined amount of pixels (well, I don't give the user the choise, I predict the size of the padding from the number of wavelet layers...). Also you may consider an extra zero padding to avoid other artefacts from the solver.

BTW, I discovered that this solver handles poorly "bad pixels", i.e. if we have a single pixel with an off value, it generates a gradient that spares several pixels (in both intensity sides). I solved that in my algorithm using a median filter to the attenuation mask. Maybe a thresholded median (modify only gradient values that are too far from the local median) may deal with most artefacts.

--B3. use different solver: The current solver (DCT based) requires solved regions to be rectangles. Using sparse linear solver would enable us to handle arbitrary regions (that would for example exclude the background region). Based on my previous experiments with iterative solvers, I think these would probably not be fast and accurate enough for astronomical images (for daylight images which are handled by most panorama tools, 8 bit accuracy is ok and can quickly be achieved, but handling the dynamic range of floats or doubles as used for astronomical pictures is an entirely different story).

I've got a very promising paper about a multigrid approach  for large images. Bad news is that it is way beyond my skills as programmer (at least, using just my free time), so it would be something for Juan, or maybe Javio. I think that I've sent it to Juan before.
I tried once to implement the multigrid solver that appears in the Numerical Recipes, but I never was able to make it work Also I have the solver that uses Fattal for his HDRC implementation, but I had no luck there neither. I may send that code to you, if you want to give it a try.

[Harry page]

Hi

A feather seems sooooo much easier 

Harry

[Nocturnal]

Hi,

I thought I would start by reading the included documentation (clap clap) but it's not showing in the process explorer. Sure wish module interfaces had an F1/Help button to bring up the help.

[georg.viehoever]

Hi Sander,
there is a help button on th bottom right....
Georg

[Nocturnal]

Holy cow Georg, this module is *terrific*! I'm trying to piece together a mosaic I started at the end of '09 and using Harry's excellent video (thanks!) I was able to finally make some progress here. Maybe I'll finally get all the panels to fit together. Super work.

[georg.viehoever]

Thanks  ! Also have a look at Steve's video http://pixinsight.com/forum/index.php?topic=3369.0 .

Georg

[Nocturnal]

Will do Georg. The mosaic consists of 6 panels each about 2.5 x 1.5 degrees. I used EQMosaic (free ASCOM client) to generate the slews. Needless to say it's a challenge to get all the images to register. I gave up 2 years ago because the result of the intermediate mosaics was so poor. Now that I see what a 2 panel mosaic looks like I will give it another try.

[RBA]

I'm using the GradientsMergeMosaic and I'm getting a couple of those funny dark-looking "smudges" (for lack of a better word) that were originally suspects of stars in the edges.

The seam areas are beautiful, the smudges are happening at two non-common edges in this particular case.
Well, there's in fact one tiny smudge at the seams, barely noticeable because it's very small. Not quite complaining about that one.

I have the latest version per http://update-devel.pixinsight.com/