Merge Mosaic Script

[georg.viehoever]

Hello,

I have recently had the problem to merge a couple of astronomical images into one. First I thought: Well, there is ImageCalibration, and Star Alignment also does Frame Adaption, so this should be simple. I was wrong:

- the images still had large scale gradients that I was not able to fix with DBE.
- just adapting frames using a linear eqution such as pixelNew=a+b*PixelOld just did not work here, because I would need different (a,b) values for each part of the image.
- the whole thing becomes a nightmare when you have to combine more than 2 images this way.

The attached screenshot shows what I mean: The 2 pieces of the Andromeda galaxy were shot on different nights, at different sky conditions. Using Star Alignment, I can get a frame adapted combination of both, but the seams still are clearly visible (highlighted using STF). Ultimately, I did the image combination with reasonable success using painted   layers in GIMP. But it was difficult. (The Andromeda pictures have other flaws as well, I chose them for the educational qualities here  )

I investigated the use of photo panorama stitcher programs (e.g. huggin http://hugin.sourceforge.net/), but these generally do not do a good job with astronomical images: They fail to find matching parts, and throw away parts of the information in the image stack - something that just is painful when you had so much trouble collecting your photons.

So here is my solution: I have written a script that combines a number of views in PI, using a method called "Gradient Domain Compositing". A good summary of it is the paper "Poisson Image Editing" by Perez, Gagnet, Blake (2003) http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.133.6932&rep=rep1&type=pdf . In essence, you no longer work on the images themselves, but their gradients. You merge the images in a way that their gradients are kept as much as possible. Of course, this is not perfectly possible due to the mismatches/boundary conditions where the images meet. So you need to add a correction, and this correction is smooth with having no second order gradients. If effect, this correction looks like a soap membrane in a deformed ring http://www.maths.tcd.ie/~foams/GALERY/mobiusBubble.jpg, or -if you are more  into architecture- the roof of the olympic stadium in Munich http://www.moiz.ca/sustainability/pics/munich.jpg.

The advantage of this correction: it is smooth, and it is spread over the whole image area. You can adapt complex gradients, if you like.
The disadvantage: It is computationally intense: You need to solve a linear equation with a number of variables equal to the number of pixels.

Fortunately, there are methods to do this efficiently. More on that in the next post.

Georg

[georg.viehoever]

Hello,

here is part 2 of this article.

As I said, the Gradient Domain Image Composition requires the solution of a linear equation with a number of variables equal to the number of pixels. This clearly becomes quite a challenge even for small images. Fortunately, solving linear equations is a fairly common problem in engineering, and there are a number of approximation methods that give decent (but not exact) results. Actually,  Microsoft recently used one of those methods to do Gradient Domain Image Composition for their WorldWide Telescope http://www.worldwidetelescope.org/Home.aspx representation of Digitial Sky Survey data http://research.microsoft.com/en-us/um/people/hoppe/spherestitch.pdf , http://research.microsoft.com/en-us/projects/terapixel/default.aspx (using a cluster of Windows Systems).

The implementation I am using here uses a solver created by Kazdhan and Hoppe http://www.cs.jhu.edu/~misha/Code/SMG/ in 2008, using a multigrid method http://www.cs.jhu.edu/~misha/MyPapers/SIG08.pdf. Its source code and executables for Windows are freely available, and have been used in this implementation. The PI script takes a number of Pixinsight views, combines them assuming that black parts of the image are supposed to be transparent, and then calls the external solver. Here are more details using the attached screenshot:
- input are the two partial images (top left, top middle) that I aligned using Star Alignment
- from the input images, the script generates a label image (top right) that has a different value for each region of the image. The solver will later try to combine the different regions without showing seams.
- from the input images, the script generates an integrated image (bottom left), using the pixels that are not black in the source images. Those pixels are combined using average, maximum, or priority operators (priority meaning that the first image that has a non-black pixel is used). Clearly this image has seams.
-finally, the external solver is called working on the label image and the integrated image, giving the result image without seams.

The result is shown on the bottom right (STF "enhanced"). The seams are clearly gone (but of course other image problems cannot be corrected this way...).

The next post will contain the script and instructions for its installation.

Georg

[georg.viehoever]

Hello,

so here it finally is, the script with installation instructions.

Installing the script:

- the script currently works on Windows only, because the solver is available for Windows only. The sources are available on http://www.cs.jhu.edu/~misha/Code/SMG/, and a quick glance suggests that it should not be very difficult to get this running on Linux et. al.. If you feel like it: Please port it. The script itself should be working on all platforms (currently untested, of course), and the solver has a fairly liberal license.
- As a Windows user: download the solver from http://www.cs.jhu.edu/~misha/Code/SMG/SMG.exe.zip , and copy it into the PixInsight installation directory. Your directory tree should be like this (important parts in bold):
C:\PCL64_161>dir
...
01.08.2010  00:56    <DIR>          .
01.08.2010  00:56    <DIR>          ..
01.08.2010  17:02    <DIR>          bin
29.07.2010  20:04    <DIR>          doc
...
01.08.2010  00:56    <DIR>          SMG.exe
29.07.2010  20:04    <DIR>          src
29.07.2010  20:04           227.729 uninstall.exe
...
C:\PCL64_161>dir SMG.exe
...
01.08.2010  00:56    <DIR>          .
01.08.2010  00:56    <DIR>          ..
25.07.2010  17:25           595.456 GradientComposite.exe
25.07.2010  17:25           637.440 GradientsToImage.exe
...

- download the attached script. You can run it directly from the script editor, or you can install it into PixInsight using the Script/Feature Scripts... menu entry.
- Allow PJSR to run external programs using PixInsight's Edit > Security Settings > Allow execution of commands from JavaScript scripts. This is necessary because the script calls the external solver to do the hard work...

Using the script (see attached screenshot):

- The script lets you add a number of views. The images need to be compatible in size and number of color channels.
- The areas of the images that are black are considered "don't care"/transparent.
- The Operations selector lets you select between 3 modes of image integration:
-- combine by using the average pixel value
-- use the maximum pixel value
-- use priority, i.e. the pixel from the first image that is not black.
- The Apply button starts the mosaic creation

Image integration currently is a slow process (15 minutes for 2 images of 10MPixel each), while the solver itself only requires a few seconds. During integration, the script outputs "." to indicate progress. So be patient.

There is still a lot of room for improvement, more on that in the next post.

Enjoy,

Georg

PS: Look at the nice detail I found in Andromeda  

[Nocturnal]

That guy is everywhere!

Nice job Georg!

[georg.viehoever]

Hi,

here some final thoughts on what should be done next. As far as I am concerned, this will not happen soon, as I plan to go on a long vacation soon. Even while I am away: feel free to improve on what we currently have.

- the UNIX port: Quite obvious, and should not be too difficult. Use the solver sources as a starting point. You may also be able to replace copies of libPng et al. with those that come with the OS.
- speedup: creating the integrated and label images is very slow. It currently is a pixel by pixel process implemented in JavaScript. I have the impression that it should be possible to do this much faster with some PixelMath expressions. I just have not yet had the time to look into this. If everything else fails: C++ of course would do this in seconds instead of minutes.
- accuracy: currently, everything is passed to/from the solver in 8bit/channel space. The limitation here is the solver that appears to accept/produce 8 bit images only. The internals of the solver clearly would work with 32/64 bit floats as well. Only someone has to implement it...
- accuracy also currently limits Merge Mosaic to be used as the final stage of the process. The result image is visually smooth, but does not allow for much additional processing. See the attached screenshot that shows the difference between the integrated (seams) and the  mosaic (smooth) image. You see how the "tensions" between the different image regions are smoothly distributed, but the steps are quite coarse.
- The solver may ultimately go into a PixInsight module, saving time and space for the intermediate images
- Of course, there are bugs and inefficiencies in the script - no doubt about that.  

I also have some ideas for other applications of this method. If you have some, please let us know...

Have fun,

Georg

[Juan Conejero]

Georg, I think you should check your optics well. You have bugs.

What a fantastic work. It's really exciting, the kind of things that wake up my insatiable hunger of programming  . Well done, I'm going to give this a thorough review!

[NKV]

Hi, Georg.

Code: [Select]

Merge Mosaic script started:Wed Aug 04 2010 12:10:05 GMT+0600
Version Info:PixInsight Core 01.06.01.0625 (x86),625,1,6,1
*** Error [001]: C:/PCL/src/scripts/MergeMosaic1.js, line 681: ReferenceError: data is not defined
BR,
Nikolay.

[georg.viehoever]

Hi Nikolay,

I will have a look this evening.

Georg

[georg.viehoever]

Nikolay,

Code: [Select]

Merge Mosaic script started:Wed Aug 04 2010 12:10:05 GMT+0600
Version Info:PixInsight Core 01.06.01.0625 (x86),625,1,6,1
*** Error [001]: C:/PCL/src/scripts/MergeMosaic1.js, line 681: ReferenceError: data is not defined

thanks for pointing out this problem. I wonder why it ever worked on my computer... Here the fixed version.

Georg

[NKV]

Hi, Georg.

Code: [Select]

** Warning [156]: C:/PCL/src/scripts/MergeMosaic2.js, line 503: assignment to undeclared variable i
Executing external command

!C:\PCL\SMG.exe\GradientComposite.exe --in D:\temp\PiTmpDir\control.txt --outX D:\temp\PiTmpDir\dx.half --outY D:\temp\PiTmpDir\dy.half --labels D:\temp\PiTmpDir\labels.png

Input Size: 2606 x 1372
Average: 7.902789 7.902789 7.902789
Executing external command done
Executing external command

!C:\PCL\SMG.exe\GradientsToImage.exe --inX D:\temp\PiTmpDir\dx.half --inY D:\temp\PiTmpDir\dy.half --out D:\temp\PiTmpDir\mosaic.png --width 2606 --height 1372 --average 7.787581501191199

Average: 0.127775 0.127775 0.127775
Solver Time:  6.63 seconds
Peak working set: 230 MB
I/O (Read/Write): 84 / 43 MB
Image Size: 2606 x 1372 = 3 MPixels
Executing external command done

Reading image:
D:/temp/PiTmpDir/mosaic.png
merging images done
And result in attachment.
It's OK?

[georg.viehoever]

Nikolay,

- I look at the "i" message tonight
- Integrated and labeled images look ok, but the mosaic is strange.
-- Do you have an STF active on the source images? You should apply this process on images that dont need an STF.
-- Could you try what happens if you change your source image into RGB images? I only tested with 3 channel images.
-- Would you share your source images with me?

Kind regards,
Georg

[NKV]

-- Do you have an STF active on the source images?

Yes.

You should apply this process on images that dont need an STF.

Thanks. Now is OK.

Only Line 503

Code: [Select]

for(i=0;i<image_p.numberOfChannels;++i){required reduction:

Code: [Select]

for(var i=0;i<image_p.numberOfChannels;++i){

[georg.viehoever]

Good to hear that it is working successful now. Can you share a screenshot?

The fix is going into the next version. For now, the current version works ok, only with a warning.

Thanks!
Georg

[NKV]

Can you share a screenshot?

Yes, please:

[georg.viehoever]

The resulting Mosaic still seems a little bit dark. Hmmm....