PixInsight Forum (historical)
Software Development => New Scripts and Modules => Topic started by: georg.viehoever on 2011 July 01 14:05:32
-
Hi,
here is the third installment of the Gradient Domain Module. The new feature this time is the Gradient Domain Mosaic Composition process, something I played with a while ago (http://pixinsight.com/forum/index.php?topic=2124.0). The general idea is that you use gradient domain operations to smoothly connect a number of pictures, with no small scale gradients.
The effect is best explained be looking at the attached screenshot:
- left: Some portrait embedded into a black background
- center: Juan's portrait is smoothly embedded into to gray background, as merged by the Gradient Domain Mosaic Composition process.
-- the differences are adjusted by generally brightening the backgorund
-- remaining differences are smoothed over a large area, very much in the same fashion a thin elastic membrane would respond to external forces.
-- There are no longer any hard gradients between the two regions - if they can be avoided at all.
- right: the GUI.
-- The top is a list of files, very much like the list of files in ImageIntegration. Below this is the parameter section
-- You can select how two images are merged: Overlay means that images are stacked on top of each other, with the large image "winning". "Average" computes the average brightness in regions where some images overlap.
-- Black Point allows to specify the brightness level that is considered Background/transparent. 0 (black) is the default - just what ImageRegistration will produce.
-- Finally, "Generate Mask" causes the process to generate an additional picture showing the different image regions.
Other features of this release:
- Gradient Domain HDR compression is the second process in this module. No major changes here.
- By default, the module will now use the solver developed by Carlos (used with permission :) )
-- That means that no FFTW library is needed any longer (FFTW would prevent inclusion into the PI product - FFTW charges for commercial use.).
-- For most images, this solver is fast enough.
-- However, it occasionally shows some artifacts. I will have to work with Carlos to find the bug.
- If you like, FFTW can be used as an alternative by using the suitable #define in GradientsBase.h and linking with libfftw3.so. It is faster, consumes less memory, and does not show the problems.
- Again, the whole module has been tested on Fedora14-x64. I don't see any reason why it should not compile and work on other PI platforms.
Attached are the sources. An application example will follow later.
Georg
-
Hi
If I read this right could this be used with a deep sky mosaic to help merge the edges ? how would you do this ?
Harry
-
Hi,
Harry recently showed in a video http://pixinsight.com/forum/index.php?topic=1167.msg21839#msg21839 how StarAlignment supports merging moaics. Linear fit brings images into a similar brightness scale. This works fine - if you were able to remove any remaining background gradients during the procesing of the separate images. This is not always possible, for example if you cannot find sufficient free space for DBE - just what happened to me some time ago.
The first attachment shows:
- top rows: two shots of M31
- bottom left: StarAlignment in Mosaic mode without Linear Fit activated. We clearly see the huge difference between the two parts
- bottom right: same with Linear Fit. The images are almost the same. But: image 2 has an internal gradient that leads to clearly visible seems in the center of the image.
How can GradientDomainMergeMosaic help here?
- it can adjust for the brightness differences. It will not help to adjust for differencs in texture.
- Screenshot 2 shows the result of the process on the M31 images:
-- top Left: merge result
-- top middle: mask displaying the different regions
-- top right: GUI
-- bottom: result with mask as overlay.
- We still see some texture differences, but the brightness jumps are gone.
- There are still some artifacts where stars are just on the border between two regions, for example in the region marked as "Preview 1". I am not yet sure how to handle such cases, ideas are welcome.
Enjoy,
Georg
-
Hi Harry,
If I read this right could this be used with a deep sky mosaic to help merge the edges ? how would you do this ?
You are right. General procedure:
- align your mosaic tiles using Star Alignment.
-- First, create a big reference frame using the "Register/Union-Mosaic" functionality
-- Then register all your tiles again in "Register/Match Images" mode, using the reference frame as the "Reference Image"
- The merge the registered frames files using the MergeMosaic process.
BTW: Not my invention. Gradient Domain Compositing has been around for quite a while, and MS apparently has used it to stitch the images in its "WorldWide Telecope" project http://research.microsoft.com/en-us/um/people/hoppe/proj/spherestitch/.
Georg
-
Great work Georg!
-
Hi
I was suggesting this sort of thing to Juan the other day 8)
The ultimate thing would be if this was implamented in the image reg / mosaic tool , too much to ask I know
But great work though should really help and look forward to seeing a windoz version so I can Play ;D
Harry
-
Hi
I may be oversimplifying this but why cant the edge say 100 pixels feather in from 1 image to another without averaging all the overlapping pixels and avoiding the problems this causes >:D
Sorry if I am being dumb ???
Harry
-
Hi Georg,
this is quite interesting module! Great job!
I tried to compile it on windows. After some minor changes to code (attached) to satisfy not-so-standard M$ compiler, I compiled it successfully.
However, I'm not able to install it into PixInsight. I'm getting following error:
Installing 1 module(s):
C:/Program Files/PixInsight/bin/GradientDomain-pxm.dll
*** PixInsight API Error: MetaProcess: GradientsMergeMosaic: Invalid LengthQueryParameter routine: path
<* failed *>
*** PixInsight API Error: Module: C:/Program Files/PixInsight/bin/GradientDomain-pxm.dll: Module initialization error: Exit code: 1
*** Warning: No modules were installed.
I'm not sure what does it mean. It never happened to me with my modules.
regards, Zbynek
-
Hi,
Georg, this is a great tool. Well done!
Later this afternoon I'll try to post versions for Windows and Mac OS X, 32/64.
Zbynek, you've been faster than me :) I'll fix those little problems too. The problem is simply that the ProcessImplementation::ParameterLength() reimplementation does not provide a valid length for the "path" process parameter (which must be a descendant of MetaString; I haven't seen the code yet). Are you using the last PCL version? Here it is:
http://pixinsight.com/developer/pcl/download/PCL-01.00.94.0367-20110529.tar.gz
Unfortunately, there is an incompatibility with previous versions (http://pixinsight.com/forum/index.php?topic=3079.0), so using the latest PCL is necessary.
-
Hi Juan,
I have still PCL-01.00.91.0338-20101220 installed, so I'll update it immediately. To be honest, it's very difficult to even find the download link for PCL now, when it's no longer under Software Distribution link. I was never able to find the download link anywhere but in forum and in the storm of 1.7 posts, I missed the one about new PCL version. Would it be a problem to add PCL download link to Software Distribution? But let's not hijack Georg's module thread with this little bit off-topic theme :police:.
regards, Zbynek
Update: with new PCL, it compiles and installs without problem :).
-
Hi,
thanks for your kind words :) .
- What the module now needs most is experience from other users, so we can find possible bugs or or kinks. Therefore I am very grateful to Juan for publishing compiled versions.
- PCL download: see http://pixinsight.com/forum/index.php?topic=3060.0
- Invalid path problem: The code is basically taken from ImageCalibration, so if there is an issue then it would be in ImageCalibration as well.
Keep up the good work.
Georg
-
Hi all,
The new GradientDomain module is now available as an update from a new development repository that we have created today. To add the new repository, please follow these steps:
- From the main menu, select: Resources > Updates > Manage Repositories
- On the Manage Update Repositories dialog, click the Add button.
- Enter the URL of the new repository: http://update-devel.pixinsight.com/
- Click OK.
- Select: Resources > Updates > Check for Updates
Now the new module should be available for your platform as an update from the new update-devel repo. Enjoy!
-
That worked fine Juan thanks.
One minor bug: when I started PI it said I had an update. I ignored this, added the devel repository. PI then told me there were no updates. I then explicitly checked for updates and it showed me both. The update then succeeded. Not something to worry about but it could be confusing.
-
Hi Sander,
Thanks for the info. Perhaps the check for updates should be done automatically after modifying the update repositories?
-
I think that would be good Juan. It seems there is a check but due to the updated repository it returns '0' or 'nil'. Just a previous un-tested code path :)
-
Great, must play.....
-
Hi
I am having a play , and I cant get it to work ??? I know its me sorry
Could you explain the process in a little more detail please :D
Harry
-
Hi
Ok I have sussed it , but as you say the edges do cause big problems ???
-
Hi Harry,
thanks for the feedback.
- do you get those artefacts only at the borders between different images, or also elsewhere?
- can I have the data to play with?
Georg
-
Hi
It is just the border / edges I see the problem >:D
The files are at my FTP server
Harry
-
Good morning!
Thanks to Harry for providing me with his data. I think I have found a workaround for the issue:
- Harry's dataset are two images of the North America and the Pelican Nebula.
- Combining them into a mosaic works quite well using PIs StarAlignment process with Linear Fit, see first screenshot on the left.
- If you look closer, you find that Linear fit will still leave some seams, most notably somewhere in the mid-west (Preview 1, first screenshot right).
- Trying to combine using the GradientMergeMosaic process works quite well, except if a star lies just on the boundary of one of the images. In this case, the process produces artefacts as shown in Harry's post on the second screenshot top left.
- Bottom left image illustrates the condition for one of the worst instances.
- Workaround: CloneStamp along the borders, replacing the stars with black background, as shown on the bottom right. Choose softness 0 and opacity 1. I do not consider this "painting", because the contents will be filled in from the other image that does not have a border there.
- Now use GradientMergeMosaic, and you get a perfectly merged mosaic as shown on the top right.
I think what currently is done manually using CloneStamp can be done automatically by some algorithm, but my ideas have not yet converged...
Enjoy,
Georg
-
Hi
Yes I can see how this works as a stop gap , but do not understand how a average of the two images produces these defects , nor am I intelligent enough to tell you how to fix it ;)
Harry
-
Hi Harry,
here is my hypothesis regarding the cause of those artefacts:
- the algorithm first combines images only by overlay (as in StarAlignment) or by average (computing the average where images overlap). At this point, it does nothing to hide the seams - the results are simular to what you see after StarAlignment. The algorithms also generates a mask that identifies which pixel belongs to which image or combination of images (toggle "Generate Mask" to see it).
- It then computes x and y gradients of the combined image. If I would feed this information to the poisson solver, I would get the combined image again - with seams and everything.
- So what I do is first patch the gradients, by setting the gradient to 0 where two image regions meet. Feeding these patched gradients to the solver thus causes the solver to try to give neighboring pixels the same value, thus smoothing out possible seams at this location. Any remaining "tension" is smoothed out over a larger region. Visually, the patched pixels are almost invisible, but the are there...
- Now, if we have a star just at a seam, the star usually causes a strong gradient. The algorithm patches this gradient to be 0. The solver now finds that the core of the star should not be as bright as it currently is, it therefore reduces the core brightness. However, other gradients in the vicinity that are not patched still tell the solver to produce a bright star. So the solver tries to find a compromise between the conflicting gradients, slightly enhancing brightness with respect to the patched gradient, slightly reducing the brightness with regard to the other gradients, and distributing the remaining tension into the neighborhood.
This results in the artifact you see in the screenshot below. I guess the papers I have read about the Gradient Domain algorithms never mentioned this problem because daylight images rarely have the kind of intense points sources that astronomical pictures have.
My current idea to eliminate/reduce those is to limit the gradient patching to regions with low gradients, but I do not yet know if this produces other problems. I think I have also seen some article that suggest how to select optimal seems...but I have not yet found it.
Kind regards,
Georg
-
Hi
Thanks for explanation , what you are doing is far more complicated than I first thought :-[
I am, sure you will have this cracked soon 8)
Harry
-
Hi Georg
Why don't you perform the merging right on the gradient domain? I've seen some algorithm of mixed methods, where they take the maximum of the two shots. Maybe a weighted average may do the trick....
-
...Why don't you perform the merging right on the gradient domain? I've seen some algorithm of mixed methods, where they take the maximum of the two shots. Maybe a weighted average may do the trick....
That's indeed probably the path I will take. Stay tuned...
Georg
-
Hello
I Was testing this on some old mosaic candidate images.
In general it worked better than the linear fit. but... I got this pair of images on witch I got no match...
I tried using both options Average and Overlay.
Regards
Geert
-
Hi Geert,
thanks for trying this module. Practical experience is most valuable to improving this module.
With regard to the images that just do not match: my idea would be that something is strange at the border of those images.
- could you provide me with copies of those images?
- can you try to cut away a couple of pixels at the border of images (where you see the underdesired effect), and then try gradient merge mosaic again?
Georg
-
Georg it was my fault.
The bottom image had a dark line near the top that only was visible in 1x
the results are really very good.
Regards
Geert
-
This was indeed the type of problem that I expected. The next release will contain a workaround for this.
Georg
-
I tried compiling this on OS X, and I ran into some issuese...
...I'm only pasting some of the error...
g++ -arch i386 -Xarch_i386 -mmacosx-version-min=10.5 -fPIC -dynamiclib -install_name @executable_path/GradientDomain-pxm.dylib -L"/Users/draphael/PCL/lib/macosx/x86" -L"/Applications/PixInsight32.app/Contents/MacOS" -o "/home/georg/Gradients/macosx/g++/x86/Release"/GradientDomain-pxm.dylib ./x86/Release/GradientsBase.o ./x86/Release/GradientsHdrCompression.o ./x86/Release/GradientsHdrInstance.o ./x86/Release/GradientsHdrInterface.o ./x86/Release/GradientsHdrParameters.o ./x86/Release/GradientsHdrProcess.o ./x86/Release/GradientsMergeMosaic.o ./x86/Release/GradientsMergeMosaicInstance.o ./x86/Release/GradientsMergeMosaicInterface.o ./x86/Release/GradientsMergeMosaicParameters.o ./x86/Release/GradientsMergeMosaicProcess.o ./x86/Release/GradientsModule.o ./x86/Release/RgbPreserve.o -lpthread -lPCL-pxi
Undefined symbols for architecture i386:
"global constructors keyed to _ZN3pcl5AlphaEi", referenced from:
anon in GradientsBase.o
"global constructors keyed to _ZN3pcl23GradientsHdrCompressionC2Ev", referenced from:
anon in GradientsHdrCompression.o
"pcl::GradientsHdrCompression::GradientsHdrCompression()", referenced from:
void pcl::GradientsHdrEngine::Apply<pcl::UInt8PixelTraits>(pcl::Generic2DImage<pcl::UInt8PixelTraits>&, pcl::GradientsHdrInstance const&)in GradientsHdrInstance.o
-
Sorry, can't help you with this.
Is this with the PCL version released today? I did not yet try it with that one.
Maybe you can use the version that Juan provided in http://pixinsight.com/forum/index.php?topic=3211.msg22047#msg22047
Georg
-
Hi Georg
Have you made any head way ;)
Harry
-
The one that you linked worked. I'll test it out and let you know how it goes!
Best,
Dave
-
Isn't Juan's version avalaible through the "development" repository (for Mac)?
-
Nope - I added the repository, and it didn't seem to find anything. I'm running the 32-bit version...
-
And it isn't working on my Windows 64-bit install either...
-
I guess it is because the repository indicates an older version of PixInsight core:
<platform os="windows" arch="x86_64" version="1.7.0.695:1.7.0.698">
<package fileName="20110703-windows-x86_64-module.zip"
sha1="9407cdb8534cadf6d272a6a72cfcafa925424e8d"
type="module"
metadata="20110703-module"/>
</platform>
I am running 1.7.0.0702
/d
-
Sorry for the mistake —I simply forgot to update the development repository after the release of 702. This is now fixed and should work correctly; let me know otherwise.
-
No Problem!
The module worked great so far for me. I attached a screenshot of a sample mosaic I'm working on.
-
Hmmm... I feel that are still some unmatched features, or at least, you can tell differences in color. Georg, is the merge done only in the luminance?
-
Hi David,
not bad! Thanks for testing this. Some questions and remarks:
- what size is this mosaic? It appears to be much larger than anything I tried on my modest Linux machine.
- how long did the process run for this mosaic? Do you feel that runtime is acceptable?
- Some of your unmerged images have strong background gradients close to their borders. These are still visible in the merged images. I guess that either a DBE (before merge) or removing this part of the images would improve the result.
Hi Carlos,
no, the merge happens separately on each color channel. I suppose that the remaining artifacts are caused by the strong background gradients mentioned above.
Georg
-
Yes... so, be need to merge the gradients of those images... I'm wondering if there could be a multiscale approach to this.
-
Hi David,
not bad! Thanks for testing this. Some questions and remarks:
- what size is this mosaic? It appears to be much larger than anything I tried on my modest Linux machine.
This is a 6 panel mosaic that is approx 7000 x 7000
- how long did the process run for this mosaic? Do you feel that runtime is acceptable?
About 5 minutes - definitely acceptable
- Some of your unmerged images have strong background gradients close to their borders. These are still visible in the merged images. I guess that either a DBE (before merge) or removing this part of the images would improve the result.
Yea - it was a quick and dirty effort. I am redoing it now and I will let you know how that goes.
-
Well - that didn't go very well...
I've attached the result of my efforts - this time I removed the gradients prior to using the Merge Mosaic! But something must be wrong with how I am doing it...
Let me know what you think.
-
Typically, there are a few lines at the border of the images where the image slowly is going into the background. For GradientMergeMosaic, the border between image and background must be without any additional pixels. Zoom to 1:1, and have a look at border pixels.
Georg
-
Ok - here is an updated version. I did screw up and left some black lines before...there is still some transition that I have had to finish by hand. But this makes it much easier. Also, I don't think there is anything wrong with the module - it is user error...I can see I am not fixing the bright stars at the edges that throw things off etc...
-
Looking good! I am working on both the bright star and the border problem right now.
Georg
-
Older messags on this in http://pixinsight.com/forum/index.php?topic=3173.0
Newer messages on this in http://pixinsight.com/forum/index.php?topic=3283.0