PixInsight Forum

Software Development => New Scripts and Modules => Topic started by: georg.viehoever on 2010 April 20 13:09:51

Title: Blind Solver
Post by: georg.viehoever on 2010 April 20 13:09:51
Hi,

as demonstrated in http://pixinsight.com/forum/index.php?topic=1853.0, I have created a script that automatically locates and labels astronomic images. The motivation for doing so was a session where I shot somewhere into the virgo cluster, but afterwards I was not able to determine what I had shot (I dont have GOTO, so finding the desired object often is quite a challenge...) . So I integrated the astrometry.net Blind Solver with PixInsight. The script still needs some polishing, but I wanted to share some results with you as soon as possible.

Below the original shot that got me started on this project. NGC4450 was on none of my maps...

Cheers,
Georg

Title: Re: Blind Solver
Post by: Carlos Milovic on 2010 April 20 13:14:53
Hey! That's great :)
Title: Re: Blind Solver
Post by: Silvercup on 2010 April 20 13:51:54
Great, very usefull.

Best, Silvercup
Title: Re: Blind Solver
Post by: Ioannis Ioannou on 2010 April 20 16:07:10
Extremely interesting !!
Just as I was downloading the sources from astrometry.net for similar reasons.
I'm very interested to give it a try  :)

John
Title: Re: Blind Solver
Post by: Ken Pendlebury on 2010 April 20 21:43:17
Wow... this is very cool.  Nice work!
Title: Re: Blind Solver
Post by: Juan Conejero on 2010 April 21 01:48:41
Will you post your script soon? We are getting impatient! ;D
Title: Re: Blind Solver
Post by: Niall Saunders on 2010 April 21 03:12:49
Will you post your script soon? We are getting impatient! ;D

Ha, ha, ha  ;D  ;D  ;D Juan,

Is this you "getting revenge" for everyone pleading with you to release v1.6 over the last few weeks !!!
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 April 21 11:25:06
Hi,

Will you post your script soon? We are getting impatient! ;D

I hope to be able to release the script within one week (that is: one week in Georg time scale  ;) ). But be warned: It needs you to install the astronomy.net software locally, and this requires you to compile the astrometry.net C-code. I did it on Fedora11-x64, but I think we should get it running on most UNIX like systems with almost zero effort - Windows probably requires more work. You also need some star database that you need to generate yourself, or a 4 GByte download. So this is not for fainthearts at the moment...

Attached is another example, this time from Harry's website http://www.harrysastroshed.com/Image%20html/ngc7331.html (used with kind permission by Harry).

Georg


Title: Re: Blind Solver
Post by: RBA on 2010 April 21 13:21:41
Very cool indeed. Re. fainthearts, if we cannot do the required steps, then we're just not worthy, end of story  ;D
Title: Re: Blind Solver
Post by: Harry page on 2010 April 21 13:48:03
Hi

Could have picked a better image  :P Not one of my best  ::)

Very nice work will have a go ASAP

Harry
Title: Re: Blind Solver
Post by: Juan Conejero on 2010 April 22 02:25:36
Quote
one week in Georg time scale

As long as it isn't in Juan time scale all will be fine  :cheesy:
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 April 22 10:44:08
Hi Juan,

Quote
one week in Georg time scale
As long as it isn't in Juan time scale all will be fine  :cheesy:

You seem to have a more accelerated life than I do, and according to Einstein that results in time dilation... http://en.wikipedia.org/wiki/Time_dilation ;D.

To shorten the waiting for those living in my "time zone": Here labeled version of my Orion wide field http://pixinsight.com/forum/index.php?topic=1678.0.

Georg
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 April 22 11:54:05
Hi,

let's get you started with the Blind Solver. In this article, I want to introduce you to the software that I use behind the scenes of the PI script (and that I did not write, I just use it in my script).

The Blind Solver is a product of a university project (of which I was not part), documented on http://astrometry.net/. If you are interested in how this software works, an article http://arxiv.org/PS_cache/arxiv/pdf/0910/0910.2233v1.pdf published by the project is probably the best introduction. Dustin Lang's thesis http://www.astro.princeton.edu/~dstn/ is an extended version of that. In essence, the Blind Solver works by creating a large database containing relative positions of groups of 4 stars (asterisms). When given an astronomical image, the Blind Solver identifies bright stars, and queries the database for the asterisms of those stars. Using the asterisms as fingerprints, it reduces possible positions/scales/rotations in the sky to a few candidates that can be validated using the other stars in the image. It can do so without knowing scale or rotations, because asterisms are invariant to these factors.

The result of the solver is center position, rotation, scale of the solved image, plus some more obscure information about the WCS coordinate system http://www.atnf.csiro.au/people/mcalabre/WCS/. This information can ultimately be used to label a star image, as I do in my application.

As far as I know, this capability is quite unique. Most of the astrometric tools I found on the web are only for "refining" already known star positions. Searching for more general tools, I found the PinPoint Astrometric engine http://pinpoint.dc3.com/ which is a commercial product. And I found the Elbrus Star Locator http://www.astrosurf.com/pulgar/elbrus/elbrusin.htm, which is open source. Both appear to require approximate information about image scale and rotation. When I tried Elbrus Star Locator, I could not convince it to solve my images. (I would be interested to hear if there are other programs solving this problem...)

Compiling the Blind Solver and installing the database requires some work. For those of us who want to give the Blind Solver a quick try, there are two options:
1. Register for the alpha version of the web service, as explained on http://astrometry.net/use.html, or
2. Upload your image to the astrometry group on Flickr http://www.flickr.com/groups/astrometry/ . The photostream of this group is screened by the astrometry.net Blind Solver web service, and annotations are added to your images automatically (sooner or later).

This, of course, is no option if you want to solve large amounts of images as I did for my Milky Way film http://pixinsight.com/forum/index.php?topic=1853.0, or if you want to use the Blind Solver to find the right position in the sky with your telescope. For that, you need the locally installed version of the Blind Solver, which usually solves an image within seconds. In the next forum article (maybe tommorow), I will give you some hints on how to install the Blind Solver software on Fedora 11 (which may also be helpful for other UNIXes).

Cheers,
Georg

Title: Re: Blind Solver
Post by: avastro on 2010 April 23 00:00:15
Hi Georg
Thanks for pointing me to the Blind Solver, It's a promising program hopefully a windows version will be available.

Quote
As far as I know, this capability is quite unique. Most of the astrometric tools I found on the web are only for "refining" already known star positions. Searching for more general tools, I found the PinPoint Astrometric engine http://pinpoint.dc3.com/ which is a commercial product. And I found the Elbrus Star Locator http://www.astrosurf.com/pulgar/elbrus/elbrusin.htm, which is open source. Both appear to require approximate information about image scale and rotation. When I tried Elbrus Star Locator, I could not convince it to solve my images. (I would be interested to hear if there are other programs solving this problem...)
About other programs for solving plates let's share my experience.
I utilize Pinpoint witch is part of ACP an observatory control program for astro-photography from the same company.
Pinpoint allow solving plates or searching for asteroids or supernovas, a light version is incorporated in Maxim program allowing to solve plate independently.
Using Pinpoint, John Winfield made a free plugin for Maxim SkySolve "It is an application which is supplied an image and with no prior knowledge of where in the sky it was taken, it uses a star pattern matching algorithm and reference database to locate the correct image coordinates."
http://winfij.homeip.net/development/SkySolve/index.html
Also Astrometrica http://www.astrometrica.at/ is a nice and cheap program I use it for minor planets search.
 "Astrometrica is a interactive software tool for scientific grade astrometric data reduction of CCD images."

Cheers

Antoine

Title: Re: Blind Solver
Post by: Ioannis Ioannou on 2010 April 23 00:34:50
It's a promising program hopefully a windows version will be available.

The code is based on U*ix flavor machines. Reorganizing it to work under Windows is , let say, complicated. And there are many dependencies on libraries that also have to be ported under Windows. The same , in smaller magnitude, should be true for porting (compiling) under Cygwin, a developers package that allows to port code from U*ix to Windows environments. See the file PORTABILITY inside the program's archive, it has some details.

For those that do not have experience with , eg, Linux, I suggest to download a program that can run virtual machines (eg VirtualBox) and also download a ready-made Linux image for this Virtual machine. Unfortunately such an image can not contain astrometry.net due to the license , or to be more precise, the program can be included since the license is GPL, but the indexes can not be redistributed, each individual should install them alone.

BTW, compiling on a Debian Linux (Lenny) box, with some of the standard developing tools already installed, was almost out of the the box, I just followed the instructions in README.

Just my 2 cents
Title: Re: Blind Solver
Post by: Juan Conejero on 2010 April 23 02:08:41
Another possibility would be writing a PixInsight module with the astronomy.net code (or the relevant part of it) embedded and adapted to the PCL. I don't know if their license terms would allow this, but of course the PI module would be published under GPL. If there is visible energy here to work in this direction, I can contact astronomy.net's authors asking for permission to start this project. Of course, something like this cannot be done without the direct blessings from the authors.

I don't know either how difficult this embedding task could be, but if the code is C and/or C++, it can be done for sure. The only thing that we need is a group of volunteer developers seriously interested. I would be part of that group very happily. My relativistic time line would limit my contributions though, but you could count with me for any advisory, documenting and helping tasks.

There's no need to explain the huge and rich functionality that the PixInsight platform would gain with this project, is it?
Title: Re: Blind Solver
Post by: Ioannis Ioannou on 2010 April 23 02:40:24
Hmm, 534 .c files, not all of them in the main program (some are libraries like cfitsio, others are demos etc). 278 header files. Not an easy walk, neither something that can not be done  - IF somebody has the time. The license of the code is GPL, so you can use it, as soon as you release your final product under GPL also.

I did not had the time to dig inside PCL, I'm new in the PI world (a fanatic though), and so far I was trying to find my way just on how to use it :  porting something like this to PCL means Javascript ?

John
Title: Re: Blind Solver
Post by: Juan Conejero on 2010 April 23 03:02:00
Hi John,

Welcome to PixInsight Forum by the way.

534 .c files... the PixInsight Core application has currently 479 .cpp files (not counting modules and external libraries) so this scares me but not too much :) If the astronomy.net application is reasonably self-contained, then the embedding work should not be really difficult IMO.

PCL is exclusively ISO C++ code; no other languages at all, including JavaScript. For documentation on the current version see:
http://pixinsight.com/developer/pcl/doc/html/
Title: Re: Blind Solver
Post by: Ioannis Ioannou on 2010 April 23 03:28:53
Thank you Juan for the welcome - I waited a few months before start posting.

Hmm, I think it will be more difficult to organize a port to be in the "PI (PCL) way" (OO oriented etc), than just to port the code :)

Thank you for the pointer. Although my time is also extremely limited, I'm intrigued and may took a look in this project. Any example of code available for newbies in PCL ? PCL's documentation is extended, but in a magnitude that can scare a hell of a newbie - no matter the professional experience behind :)

John
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 April 23 03:47:43
Wow, you guys are fast...

Since this licensing discussion has come up, a few words on this (written in my lunch break, more tonight):

- The Blind Solver software itself is published under GPL v2, see http://www.astrometry.net/use.html. As I understand GPL, it would not be ok to link this (or derived work) with a close sourced product. And PI is closed source... . According to GPL it would be ok to  package the (standalone) executables with PI, but I think even this should not be done without the author's blessing.

- Calling an external GPL program from some other program also is ok according to the GPL. And this is what my script currently does. I will not provide databases or Blind Solver executables. You will have to build and/or download these yourself.

- The pre-build databases provided by astrometry.net are based on a cleaned version of the USNO-B catalog. The astrometry team apparently gives non-commercial users access if you ask them. The use conditions are formulated in http://trac.astrometry.net/browser/trunk/src/astrometry/GETTING-INDICES. The key sentence here is:

Code: [Select]
NOT-FOR-COMMERCIAL-USE: Because the indexes represent a re-release of
36    data originating in the USNO-B Catalog, the indexes may not
37    be used for any commercial purpose whatsoever without prior written
38    consent from our team.  Our consent will only be granted to
39    commercial users for use in public, not-for-profit projects they
40    are engaged in or supporting.


As I understand this, it is unlikely that those databases can be redistributed with PI.

- Blind Solver also provides the tools to create databases from star catalogs, but this is something I have not done,  so I dont really know how complicated this is. The license conditions for such a database would probably depend on the license conditions of the original data.

- from the technical point of view: Blind Solver is a collection of executables that work with each other to do the job. These cannot go into one DLL or something. It is also likely be a lot of work to get Blind Solver going on non-UNIX systems (i.e. Windows), although this is not impossible.

Georg
Title: Re: Blind Solver
Post by: Ioannis Ioannou on 2010 April 23 04:29:25
Yap, Georg is absolutely right, we lost sight  :-[

Any "porting" can not be embedded , GPL is not like the BSD license.
But if it is in a form of stand-alone, shelf included program, and the sources are available, and the release license is still GPL, then it can be used from any program. But then, again Georg has a point: why to "embed" ? in this case, under Linux/Unix, it is just use the program as it is, maybe via a script. The only problem is for environments that does not exist (ie Windows).

Indexes are out of question, you can not redistribute them - either you agree with astrometry.net license or you built yours - period.

John
Title: Re: Blind Solver
Post by: vicent_peris on 2010 April 23 04:39:48
Hi,

this is an extremely interesting new project. As I understand, this is a similar case as GREYCstoration. In that case, there was no problem as PI has a modular architecture, so the code of the GREYC module was not directly linked to any other module nor the main app. Then it was suffice publishing the GREYC module code.

Could be done in the same way for astrometry.net?


V.
Title: Re: Blind Solver
Post by: Carlos Milovic on 2010 April 23 04:46:07
Hi John

Wellcome to PCL development ;)
Take a look at the source folder in PixInsight's one. There you'll find a lot of examples of code. Sandbox is the "easier" of them, providing a perfect example for beginners.


PS: If you need a hand developing it, here are mine. Although, I'm warning you that I have two left ones ;) VBG Seriously, I can help with the PCL style, but not that much with all that cross platform stuff, and advanced programing.
Title: Re: Blind Solver
Post by: Ioannis Ioannou on 2010 April 23 07:34:49
Thank you guys, even though I'm not sure if I'm the correct person to be welcomed to a developers team.
Too many years out of the development and too limited time  :(
OK, today the boss was not looking, so I upgraded my 4 years old Cygwin port with the newest, included gcc, make, automake and everything related to cairo, netpbm, libpng, libjpeg, python and numpy, as in astrometry.net's README and run make. It seems to compile up to the point that it is trying to make the extra "eye candy" stuff, ie cairo, netbpm etc. I guess I have to do it from the scratch, but use some trick like:

CC="gcc -I /usr/include/cairo -I /usr/include/netpbm" make

to pass - stay tuned. The issue is that I now have a working solve-field.exe which seems to work, but I have to test from home (I do not have the indexes with me and it is a pain to download again).

A bravo to both Cygwin and astrometry.net guys, they have done a great job  8)

So I guess, this is a fast solution for Windows users and it is up to Georg and his script ?

John
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 April 23 09:39:42
...
A bravo to both Cygwin and astrometry.net guys, they have done a great job  8)
...

Bravo John,

you are faster than I am in writing the articles and the script.

Regarding Windows: If you can get the blind solver compiled on Windows/Cygwin, I don't think it will be too difficult to get my script running there as well. A few "if(Windows)" statements in the JavaScript code should do the trick (path names, tmp directories, things like these). I currently also need a small PCL/C++ module http://pixinsight.com/forum/index.php?topic=1791.0 that should not be difficult to port to Windows, or that even can be replaced with pure JavaScript with slightly reduced functionality http://pixinsight.com/forum/index.php?topic=1791.msg10924#msg10924. If you want to have the labels (and not only the WCS coordinates), your probably need to get the "eye candy stuff" (netpbm etc.) working on Windows as well.

Attached a labeled version of M45 by Harry (http://www.harrysastroshed.com/Image%20html/m45eq120.html, used with kind permission).

Georg
Title: Re: Blind Solver
Post by: Niall Saunders on 2010 April 23 10:08:28
Actually guys,

It occurs to me tha, perhaps, the BlindSolver team might actually be quite 'keen' to see their hard work put to use as an add-on to the likes of PixInsight. In which case, an opening discussion with Juan, and maybe those of you who better understand what would be needed to implement the code (and satisfy all the licensing agreements), would be a great start.

Obviously, PixInsight - as, and of, itself - does not 'need' a BlindSolver add-in. And exactly the same could be said of BlindSolver itself - it doesn't 'need' the likes of PixInsight in order to survive.

However, I definitely see a 'kindred spirit' between the two applications. Both are born of a desire and are driven by passion. OK, so PixInsight (eventually) 'went commercial', but I very much doubt that Juan was in any great 'rush' to start earning money off his hard work, simply because the moment you 'set a price' you shift your own goal-posts - and generally make life an order of magnitude more complicated for yourself.

Let's see if an exchange of ideas CAN take place - and if nothing can be made to work out, there is no real loss (for either side). After all, PixInsight users can still upload their images to the Flickr site and have them BlindSolved 'on-line'.

Cheers,
Title: Re: Blind Solver
Post by: Ioannis Ioannou on 2010 April 23 10:46:44
I can not comment on Niall's post, I'm new here and all these are beyond me.

I managed to fully compile it it under Windows, something compliant with GPL, if anyone is interested read below.

My "porting" is via Cygwin, meaning that you will need both Cygwin's library (libraries ? not sure) and most probably all the standard shell tools, like bash, sed, awk etc that come with , plus the directory tree under c:\cygwin. The developers of astrometry.net warn for the need for shell tools, but I'm not sure if they mean during compilation or during operation of the program. For the same reasons I do not know if a binary distribution has any meaning - without Cygwin it may be unusable. But I may be wrong here , I have years to follow these things. And in any case all these are GPLed, meaning that they can not be included as part of a commercial product, only under the conditions GPL allows. Compiling with the minGW32 compiler may minimize the dependencies on Cygwin libraries, but again I think you will need bash etc., so......

Nevertheless it is fairly easy to have a Cygwin installation just for this. And that means that Georg's script needs the minimum of trimming, basically run the solver via a batch file that will run the program via Cygwin's bash first, or maybe just a "bash -c ...... "will do the trick.
 
Anyway, for those that are interested to compile it for their usage under Cygwin :

1) Download the setup from Cygwin project and start the installation - follow the instructions
2) During the setup, at the package selection be sure that you included the following:

      gcc, sed, gawk, make, automake, pkg-config, python, gzip, bzip2 and all packages related to
      cairo, netpbm, libpng, libjpeg, and numpy  (libraries etc)

I may miss some packages from the above list, since I had the installation already from the past and some packages have been there for years. You have to play it as it goes, and go back to the setup program later to install any missing package(s). WARNING do not trash the setup program after installation exactly for this reason.
The file README in the archive actually have most of the following steps. The command

   df -h

is your friend inside a Cygwin bash shell to help you understand how the different Windows drive letters map to Cygwin's unix-like filesystem.
You will need to remember your command line skills, and at least know how to issue commands like cd or ls.

3) open a Cygwin bash shell and untar the package ie, (assuming that you are in the directory that the astrometry.net's source archive is), run

   tar xjvf astrometry*tar*bz2
 
   cd astrometry*


4) edit the file util/makefile.netpbm (use either wordpad or an editor from inside the cygwin shell, eg vim or nano or mcedit) and change line 17 to be

NETPBM_INC := -I /usr/include/netpbm

4) from the top directory run

   make
   make install
   make install-indexes

(see README)

5) edit your Cygwin $HOME/.profile to add astrometry installation directory in your path.


6) Open a new bash shell, you should have a command solve-field.exe in your $PATH

Enjoy


A word about performance: it is slow comparing to native ports eg under Linux. The following image was solved in about 12 minutes under my Linux box and needed about 30 minutes under Cygwin

Enjoy
John

Title: Re: Blind Solver
Post by: Carlos Milovic on 2010 April 23 12:03:30
wow... I guess that I'll need a whole weekend for an image in my pentium 3 :D
Title: Re: Blind Solver
Post by: Niall Saunders on 2010 April 23 13:17:53
Having just read that post from Ioannis, I reckon that it could take me about 10 YEARS to get BlindSolver installed :'(

Cheers,
Title: Re: Blind Solver
Post by: Ioannis Ioannou on 2010 April 23 13:31:23
Naahh, it is not so difficult, do not let the detail I tried to put in the post to confuse you.
Actually it is more like a 1,2,3,4 procedure.
And - license permitting - can be a zip file that you unzip over a standard out-of-the box
Cygwin installation (just the code, again the data can not be redistributed, each person
should do them after)


John
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 April 23 13:59:16
Hi,
wow... I guess that I'll need a whole weekend for an image in my pentium 3 :D

I will not have the time to write about compilation of Blind Solver and the script tonight. Please be patient until tommorow. Just some hints to those already compiling the Blind Solver it:
- Fedora11 with Development profile had all tools required for compilation. No need to add additional packages.
- The netpbm fix in util/makefile.netpbm is also needed for Fedora11
- I got best results if I ran solve-field with the options
Code: [Select]
solve-field --downsample 2 -d 30 picture.jpg I did all of the examples and the small movie with these settings. See http://trac.astrometry.net/browser/trunk/src/astrometry/README for more options.
- Most images solve in well below 60 seconds on a Win7-x64 laptop running Fedora-x64 with 2 GBytes of RAM inside a VirtualBox virtual system.
- The limiting factor appears to be RAM during database access. I guess native Linux systems with more RAM should fly much faster than my setup (estimate: factor 3-10).

More tommorow,

Georg
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 April 24 03:59:30
Hello,

this article summarized what I learned during the installation of the Blind Solver. It does not have the goal to replace the directions you can find in Blind Solver's README files. It will also not (yet) explain how to install to PI module/script (that's my next article).

- Environment:
-- I did all of my experiments on a Fedora11-x64 system with 2 GB of RAM. This was running inside a VirtualBox environment running on a Window 7-x64 laptop with 4 GBytes of RAM. http://pixinsight.com/forum/index.php?topic=1052.msg10685#msg10685
-- If you want to try a similar setup: VirtualBox is free for private use http://www.virtualbox.org/. Installing Fedora on it is really not difficult if you remember to remove the install-CD after the first reboot. Don't forget to install the VirtualBox Guest Extensions in Fedora.
-- To exchange files between Linux and Windows, you can use the Virtual Box shared folders (there are also other options...)
---Create a permanent shared folder in the Devices Menue of your virtual machine.
---mkdir ~/host # the mount point
---su -
---mount -t vboxsf -o uid=500,gid=500 oeffentlich /home/georg/host
---you can add the mount line to /etc/rc.local so that it is executed on every boot
-- I chose the Development Profile when installing Fedora, and did not need to install additional packages for Blind Solver later. If you need to install packages: everything you need is available via System>Administration/Add-Remove Software.
-- As I see it, Blind Solver should be able to compile on most UNIX like platforms without much trouble, and even Windows seems possible http://pixinsight.com/forum/index.php?topic=1857.msg11350#msg11350.

- Blind Solver installation:
-- Your starting point is http://www.astrometry.net/use.html. Follow the instructions in the README file.
-- Currently, I am using the latest development version (Revision 14686) of Blind Solver, downloaded from the development repository as described in http://trac.astrometry.net/wiki/Subversion , using the http method (svn co -r 14686 http://astrometry.net/svn/trunk/src/astrometry astrometry) . The default 0.25 version available on http://www.astrometry.net/use.htm should work as well.
-- Compilation:
--- Patch util/makefile.netpbm to contain
Code: [Select]
NETPBM_INC ?= -I/usr/include/netpbm
NETPBM_LIB ?= -L. -lnetpbm
--- Patch the main Makefile to contain your desired installation directory, in my case
Code: [Select]
INSTALL_DIR ?= /home/georg/astrometry
--- compile with
Code: [Select]
make; make extra; make install--- add the PATH to the binaries to your PATH variable. In my case, I added this to ~/.bashrc:
Code: [Select]
# User specific aliases and functions
export ASTROMETRYDIR=$HOME/astrometry
export PATH=$ASTROMETRYDIR/bin:$PATH
--- "solve-field" should now answer with the help message.
-- Databases:
--- Blind solver uses "fingerprint" databases to identify stars. You can request access to them as described in http://trac.astrometry.net/browser/trunk/src/astrometry/GETTING-INDICES
--- The README http://trac.astrometry.net/browser/trunk/src/astrometry/README describes which databases make sense depending on the scale of your images, and how to install them (basically a copy to the data directory). In my case, I downloaded the databases including those in index-203*. In effect, all the images I tried solved with databases above and including index-205*, so I could have saved 3 GBytes of my 4 GBytes database download.
--- After downloading the databases into you source directory, it does not hurt to check their integrity using "md5sum -c md5sums.txt"
--- unpack them with with "make install-indexes"
-- All this may seem to be complicated. But compared to other Linux software, it is actually quite straight forward if you have some basic Linux experience.

- Use:
-- solve-field --downsample 2 -d 30 image.jpg usually works best for me. Downsampling increases the success-rate with JPEG images quite a lot.
-- in one example, I had to reduce the brightness of an image using the PI HistogramTransform by moving the midtones slider to the left. Blind Solver sometimes appears to have problems with (near) saturated stars.
-- it worked without problems on unprocessed images
-- using --sextractor as an alternative star extraction tool did not give additional benefits (This tool can be installed via System>Administration/Add-Remove Software, just as many other Linux astronomy tools)
-- runtime is usually <1 minute for solving. It can get larger when it has to go into the index-205* and below databases, since these no longer fits into my RAM file cache (BTW: The Blind Solver web service appears to be running on a 48 GBytes box, so effectively all databases are held in RAM).
--using the --scale-low parameter can reduce runtime a lot, for example if you know that you are solving a widefield shot.

- More information:
-- See the different README/FAQ files in the source tree
-- http://trac.astrometry.net/wiki
-- http://pixinsight.com/forum/index.php?topic=1857.msg11317#msg11317

Cheers,
Georg
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 April 24 04:54:43
Folks,

so finally here is the script. It is still not where I wanted it to be, but it is already a very useful tool. I used it (or previous versions) for all the images in this thread, and the Milky Way film.

- Installation:
-- Install the Blind Solver as described in http://pixinsight.com/forum/index.php?topic=1857.msg11360#msg11360
-- Install the SystemCommand module into PI as described in http://pixinsight.com/forum/index.php?topic=1791.msg11361#msg11361
-- Download the Script
-- Install it into PI using Script/Feature Scripts...
-- Done

- Use:
-- Apply it to the current image using the "Apply" button. The default settings work well for me.
-- The GUI (see attachment) is pretty self explaining, and has context help. I have added my most successful command line options to the "Additional Parameters" field, you may need different ones.
-- The "min. Field of View" setting is important to reduce runtime. But all my images also solved with the default setting.
-- "Load diagnostic images" displays the asterism use to solve the current image (plus matched stars during verification). For failed runs it displays the identified stars.
-- The "max compute time" parameter is important if Blind Solver is not successful. It terminates the solving process after the specified amount of seconds. The "Cancel" button does not work yet.
-- The script can be used as part of a process container, and with image containers. That's how I did the video (lazy me... :D )

- ToDo:
-- Cancel does not work. You'll just have to wait for maxTime seconds...
-- The text generated by "Location" sometimes does not have the right scale. Probably needs some fiddling with the text drawing code that I borrowed from the Draw Signature script
-- When used with ImageContainers, it seems to eat memory over time. After 178 images, my Linux systems became unresponsive due to PIs memory consumption. I think that is most likely a PI core problem, but I was not able to drill down to it.
-- The script -no doubt- has bugs...
-- I am sure there are many other creative uses of Blind Solver. One that immediately came to my mind was to help me navigate a non-goto telescope. This is probably something useful for the image aquisition module of PI 2.1.4 http://pixinsight.com/forum/index.php?topic=1809.msg11142#msg11142  ;) .
-- Creative Minds of PI: Unite!

Feel free to use, improve, modify the module as you would like. It is released under GPL 3.

Georg
Title: Re: Blind Solver
Post by: Ioannis Ioannou on 2010 April 24 06:53:15
Great job Georg, thank you.
I did not tested the script yet, I have to figure out the installation first, especially about the SystemCommand etc (remember I'm newbie in PI world).

Just a few information regarding the program itshelf:

You are right about the parameters and the time to solve - I did not used any parameters during my tests, thus the big times I reported. So, with the --downsample 2 -d 30 parameters I got 24secs on my Linux box (dualCore@1.6GHz+1GB RAM) instead of 12 minutes (!!!!). On the Cygwin port (DualCore@1.8GHz+3GB RAM,Vista) I got 14 minutes vs about 30 before. Both cases I have indexes from 203 to 219. Not sure why the Cygwin is so slow, and has nothing to do with the roles each machine has (the Linux box is a server running many-many things and has less RAM). I found some posts mentioning that Cygwin may be slow due to Antivirus/Firewall settings etc, but still I can not boost it more by disabling these services - anyway.

A tip about the VirtualBox+Linux image alternative: If somebody does not know/want to install the operating system in the box, you can find free ready Linux images for VirtualBox at http://virtualboxes.org/images/ . If the Cygwin compilation is indeed so slow, and it is not just my setup, then this is a better alternative under Windows.

John
Title: Re: Blind Solver
Post by: vicent_peris on 2010 April 24 08:53:02
-- Creative Minds of PI: Unite!

Hi Georg, very good work!

I think labeling is not the most important task this tool can do. If the blind solver is able to identify stars in the image, this can be the starting point of a bunch of new tools. Once we have identified all the catalogued stars in the image, we can output a star position list. And this list can be very useful; these are some applications that come to my mind now:

- Cross-identification of stars between Blind Solver and StarGenerator. StarAlignment cannot work properly if there are geometric distorsions between the SG synthetic map and the image. We can say SG to import a star list and force these stars to go to positions designed by the catalog and the defined projection. This can be a rigurous and, at the same time, automatic way to correct geometric distorsions in astronomical images.

- Photometric superflats. If we have brightness data of the stars of the image, we can compare their theoretical brightness with the one IN the image. In this way, we can correct multiplicative errors of flatfielding, or differential atmospheric extinction across the image (for wide fields). This could be a revolution in mosaicing techniques, because once you have corrected any multiplicative error, you really KNOW that any residual error is an additive one.

Other applications of blind solving can be for comets or asteroids. By solving a set of images you can know the center of each one. You can make a very precise track and stack of any comet or asteroid in the FOV. Interpolation algorithms are very good in PI, IMHO. And this could be a step forward repect to other software packages. Another possible use would be the process of an image set to alarm of any moving object in the FOV.


I'm an "idea guy"; now you must type code.  :P


Regards,
Vicent.
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 April 24 14:51:18
....Not sure why the Cygwin is so slow, and has nothing to do with the roles each machine has (the Linux box is a server running many-many things and has less RAM). I found some posts mentioning that Cygwin may be slow due to Antivirus/Firewall settings etc, but still I can not boost it more by disabling these services - anyway.

I heard that file operations in Cygwin are particularly slow. I did not measure that myself, and I dont know if it is related to Antivirus, just rumors I heard.

Georg
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 April 25 08:47:39
Hi Vincent,

-- Creative Minds of PI: Unite!
...
I'm an "idea guy"; now you must type code.  :P
...

these ideas sound interesting, although I am not always sure about the details behind them.  I am also not sure how much of catalog information one could/should use to improve images while still claiming they are real photos (ultimately, a BlindSolver++ could just download the best available Hubble shot of the identified region, elliminating problems with light polution and haze once and for all  ;) ). I already felt slightly uncomfortable with the star generator module helping me to stitch mosaics.

Lets see what ideas come up from the other creative minds....

Georg
Title: Re: Blind Solver
Post by: Carlos Milovic on 2010 April 25 10:58:49
[creative mode on]... ... beer ... [creative mode off]

:D

Seriously, first I wanna know what kind of information may be outputed by the blind solver. Then we can think on ways to use it. If we have star positions, identified (and their RA/Dec coordinates) there are lots of things we can do. From distortion maps, to intensity correlations. It may be a way to calibrate the flux in photometry calculations, using a few standard stars in the image.

From a less professional point of view, we might create a database of amateur shots, like the DSS, or create spherical charts like google sky... Or, create a tag system that contains all the objects in an image, so an online system may search for other pics with the one you are looking for. Other applications: create automatic crops of the object of interest (and rotations), etc. Draw a orientation compass on the image...
Title: Re: Blind Solver
Post by: Juan Conejero on 2010 April 27 02:48:59
Hi Georg, John et al,

- The Blind Solver software itself is published under GPL v2, see http://www.astrometry.net/use.html. As I understand GPL, it would not be ok to link this (or derived work) with a close sourced product. And PI is closed source...

I don't see this in the same way. I may be wrong —interpretation of the GPL is always a complex matter—, but let me explain how I understand open-source development on the PixInsight platform, in relation to GPL.

As you know, PixInsight is a modular system. The PixInsight Core application provides just the infrastructure to build an image-processing platform: three user interfaces (command-line, graphical and scripting interfaces) plus an application programming interface to communicate with external modules. The whole processing capabilities are implemented by installable modules in PixInsight. A PixInsight module is a dynamically linked shared object (a .so, .dylib or .dll file on UNIX/Linux, Mac OS X and Windows, respectively) that communicates with the PixInsight Core application through a free, publicly released and fully documented API: the PixInsight Class Library (PCL). A PixInsight module works following the client-server paradigm: the modules are clients and the PixInsight Core application is a server with a known set of resources, which are publicly accessible and can be used by its clients. The PixInsight Core application never works as a client of any PixInsight module.

You can release an open-source software product on the PixInsight platform, just as you can do the same on Windows, Mac OS X or commercial UNIX systems, which are —to greater or lesser extents— closed-source, proprietary operating systems. This is possible thanks to dynamic linkage.

What the GPL license prohibits explicitly is a closed-source software product utilizing a GPL'ed library to implement some part of its own functionality. In other words, a closed-source software product cannot depend on an open-source product released under the GPL. A software product P depends on a software product Q if by removing Q, P ceases to work as P has been defined initially (that is, before the existence of Q). This basically prohibits static linkage against a GPL'ed library, as in such case there is no way to demonstrate that the closed-source product could continue working exactly in the same way after suppression of the GPL'ed library. For example, linking astrometry.net code statically to the PixInsight Core application would force me to release the whole source code of PixInsight under the GPL, for that reason.

However, building a PixInsight module that follows a publicly released, fully documented API (PCL in this case), and releasing it under the GPL, is no problem at all IMO. This is because (i) the PixInsight module uses exclusively dynamic linkage to access the services offered by the PixInsight Core application, and (ii) the PixInsight Core application does not depend on the PixInsight module. You can uninstall a PixInsight module and PixInsight Core can continue working exactly in the same way. Of course, if you uninstall a module, the PixInsight platform, as a whole, ceases to offer the functionality implemented by the module in question, but this does not alter or limit the functionality inherent in PixInsight Core. PixInsight Core is an image processing application, with and without, before and after, the module in question.

From the user's perspective, the GPL says that the user must have the freedom to modify the source code of a GPL'ed software product, to adapt it to his/her own needs, or just to inspect it. The user must also have the freedom to replace a GPL'ed software product with his/her own version without limitations, in the same context as the original product. Both conditions are perfectly fulfilled with a PixInsight module released under the GPL. The question here is the definition of "product": the product covered by the GPL is the PixInsight module in this case, not the PixInsight Core application.

For open-source purists, the only problem could perhaps be static linkage to the PCL. Actually, there should be no legal problem here as the PCL does not depend on the GPL'ed code, and linking statically against the PCL does not change the fact that it is exclusively the GPL'ed module what is using PCL code to implement its own functionality. Anyway, I can release a new version of the PCL with an open-source part and a small proprietary part. The open-source part would be released under a dual GPL/LGPL license (in Qt's fashion), and would implement the whole low-level API for communication between PixInsight Core and PixInsight modules. In this way, open-source modules could be linked statically against an open-source PCL library. In fact, I plan on doing exactly this with PCL 1.1; this would only accelerate a bit what I already want to do.

Having said all this (sorry for the lenghty story), I must say that I would never start a project like this one without the author's blessings.

Quote
The pre-build databases provided by astrometry.net are based on a cleaned version of the USNO-B catalog. The astrometry team apparently gives non-commercial users access if you ask them. The use conditions are formulated in http://trac.astrometry.net/browser/trunk/src/astrometry/GETTING-INDICES. The key sentence here is: ...

As I understand this, it is unlikely that those databases can be redistributed with PI.

Indeed. A hypothetical astrometry.net PixInsight module could not distribute these databases, unless the authors provide explicit permission to do so.

In my opinion, everybody would benefit if some astrometry.net functionality were implemented as a PixInsight module. astrometry.net is the best astrometry engine, which could be used by all present and future PixInsight users. You and Vicent have pointed out some of the many and exciting possibilities that this integration could give rise to. I honestly think that we offer an excellent infrastructure, where astrometry.net would benefit from direct portability to Linux/UNIX, Mac OS X and Windows, a rich and modern user interface, and integration with a solid and evolving platform. Georg's excellent script clearly shows this.

A completely different thing is how difficult this integration could be, in strictly technical terms. It surely wouldn't be the easiest task but, well, we already have done quite a few difficult things with PixInsight...
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 April 27 10:39:52
Juan,

...
However, building a PixInsight module that follows a publicly released, fully documented API (PCL in this case), and releasing it under the GPL, is no problem at all IMO. This is because (i) the PixInsight module uses exclusively dynamic linkage to access the services offered by the PixInsight Core application, and (ii) the PixInsight Core application does not depend on the PixInsight module. You can uninstall a PixInsight module and PixInsight Core can continue working exactly in the same way. Of course, if you uninstall a module, the PixInsight platform, as a whole, ceases to offer the functionality implemented by the module in question, but this does not alter or limit the functionality inherent in PixInsight Core. PixInsight Core is an image processing application, with and without, before and after, the module in question.
...

as we are talking legaleese here, a disclaimer first: I am not a legal counsel, and I am not trained to interpret this kind of stuff correctly. So I may be entirely wrong  :(

I don't think that dynamic linking, the intermediate layer of the GPLed PCL module, the direction of calls or the level of dependency between the software pieces make any difference: There is a close interaction between the PI core and the module. And this appears to be the primary criterium here. See http://www.gnu.org/licenses/gpl-faq.html#NFUseGPLPlugins for a discussion of similar cases.

...
Having said all this (sorry for the lenghty story), I must say that I would never start a project like this one without the author's blessings.
...

I think that's the best way to go. The copyright holder of a software can give you any permission he likes, even if he distributes the software under GPL for "ordinary" users.

Georg

Title: Re: Blind Solver
Post by: georg.viehoever on 2010 April 27 11:41:38
Carlos,

...
Seriously, first I wanna know what kind of information may be outputed by the blind solver
....

See the attached screenshot for the type of information generated by the Blind Solver. This looks like a pretty complete set of WCS information. It should allow for the use of all those WCS tools (for example http://tdc-www.harvard.edu/wcstools or http://www.atnf.csiro.au/people/mcalabre/WCS/wcslib/index.html) that are available on the web (and pre-build in the Fedora software repository). I am not sure about the Blind Solver's accuracy, its main purpose seems to be to provide a rough solution. But for labeling image it is certainly sufficient. And you can always add a more accurate post-processor.

Georg

PS: My current script throws away most of this information, so there is room for a version 2...
Title: Re: Blind Solver
Post by: vicent_peris on 2010 April 29 09:18:38
Hi,

it would be great if the script outputs a text file with the equatorial coordinates of the center of the images. This would be, through an image container, a valuable tool to make a telescope pointing model.

At my Univ's observatory, we have an altazimuth telescope that can generate a pointing model from offsets measurements. Knowing the altazimuth coordinates of each of the pointing images, we can calculate the altazimuth offsets from the equatorial offsets. So, in case of ouputting a text file with the eq coordinates of each image, we only need an spreadsheet to transform those offsets.

OTOH, if we further develop this tool in the future to completely calculate the pointing model, would make this tool the first direct link between PI and observational instruments. IMO this is a very interesting new way to go.


Could be added, for the moment, the text file output?

Regards,
Vicent.
Title: Re: Blind Solver
Post by: Carlos Milovic on 2010 April 29 09:25:04
IMHO, going into the "instrument operation" field should be "off-limits" for the 1.x series... there is just too much work to do right now :D
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 April 29 09:42:11
...
it would be great if the script outputs a text file with the equatorial coordinates of the center of the images. This would be, through an image container, a valuable tool to make a telescope pointing model.
...

Yes, indeed an interesting idea. Only I dont know how I can generate a filename while processing an image container. The image view that the script sees has a more or less random name, if I remember correctly. Is there a way to find the original file name of a view while being called from an image container? Juan?

Georg
Title: Re: Blind Solver
Post by: Carlos Milovic on 2010 April 29 10:02:49
Don't have PI here, so I cannot check the PSJR classes... but take a look into something like ImageWindow. The PCL class has the "String  FilePath () const" function, that may do what you want.
Title: Re: Blind Solver
Post by: vicent_peris on 2010 April 29 10:52:07
...
it would be great if the script outputs a text file with the equatorial coordinates of the center of the images. This would be, through an image container, a valuable tool to make a telescope pointing model.
...

Yes, indeed an interesting idea. Only I dont know how I can generate a filename while processing an image container. The image view that the script sees has a more or less random name, if I remember correctly. Is there a way to find the original file name of a view while being called from an image container? Juan?

Georg


In fact, I think this can work without outputting the filenames, if you have stablished a clear naming order: you know that the second coordinate is for your second image.


V.
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 April 30 01:12:35
Vincent,

...
it would be great if the script outputs a text file with the equatorial coordinates of the center of the images. This would be, through an image container, a valuable tool to make a telescope pointing model.

At my Univ's observatory, we have an altazimuth telescope that can generate a pointing model from offsets measurements. Knowing the altazimuth coordinates of each of the pointing images, we can calculate the altazimuth offsets from the equatorial offsets. So, in case of ouputting a text file with the eq coordinates of each image, we only need an spreadsheet to transform those offsets.
...

Would you need equatorial coordinates (RA, DEC) or horizontal coordinates (Alt/Az) http://en.wikipedia.org/wiki/Celestial_coordinate_system ? RA/DEC is already computed by the Blind Solver (but not stored). Alt/Az of course is slightly more difficult, since this depends on your geographical location and time of day, but this could be done as well.

Georg
Title: Re: Blind Solver
Post by: vicent_peris on 2010 April 30 05:37:09
This scope has an altazimuth mount. So it generates the pointing model by analizing deviations in altitude and azimuth coordinates. For each image I acquire, I write the altaz coordinates of the telescope. What we were planning is to calculate the coordinates of each image, and then make a spreadsheet that make the transformation between alt&az deviation and AR&Dec deviation. This is done simply by knowing the altaz coordinates of each image: you can calculate the angle between the two coordinates systems in that altaz coordinates and then turn the altaz deviation.


Vicent.
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 April 30 05:41:49
This scope has an altazimuth mount. So it generates the pointing model by analizing deviations in altitude and azimuth coordinates. For each image I acquire, I write the altaz coordinates of the telescope. What we were planning is to calculate the coordinates of each image, and then make a spreadsheet that make the transformation between alt&az deviation and AR&Dec deviation. This is done simply by knowing the altaz coordinates of each image: you can calculate the angle between the two coordinates systems in that altaz coordinates and then turn the altaz deviation.
...

But alt-az coordinates change depending on the time of day, dont they? For example, during sunset the alt-coordinate of the sun is 0? So you must consider the tme at which you shot the image when you convert between alt/az and RA/Dec equatorial coordinates. Or am I getting something wrong here?

Georg
Title: Re: Blind Solver
Post by: vicent_peris on 2010 April 30 06:26:59
This scope has an altazimuth mount. So it generates the pointing model by analizing deviations in altitude and azimuth coordinates. For each image I acquire, I write the altaz coordinates of the telescope. What we were planning is to calculate the coordinates of each image, and then make a spreadsheet that make the transformation between alt&az deviation and AR&Dec deviation. This is done simply by knowing the altaz coordinates of each image: you can calculate the angle between the two coordinates systems in that altaz coordinates and then turn the altaz deviation.
...

But alt-az coordinates change depending on the time of day, dont they? For example, during sunset the alt-coordinate of the sun is 0? So you must consider the tme at which you shot the image when you convert between alt/az and RA/Dec equatorial coordinates. Or am I getting something wrong here?

Georg


No. For pointing model, you mast take the altaz coordinates of the mount exclusively. In my case, I wrote the altaz coordinates in my notebook at the time of acquiring each pointing model image. Then, all you need is geographic latitude and declination the telescope was pointing at. With these data, you can calculate the angle between declination and altitude at a known altaz coordinate, depending on the geographic latitude.


V.
Title: Re: Blind Solver
Post by: Nocturnal on 2010 April 30 14:03:31
Somehow I missed this whole firestorm :) This is terrific work Georg! I've been solving my images with astrometry.net for some time and thought about creating a better labeler myself as the am.net one is a bit limited. Now I don't have to write this anymore, excellent!

I'll be trying this out soon.
Title: Re: Blind Solver Version 0.2.0
Post by: georg.viehoever on 2010 May 16 04:29:20
Dear all,

attached is version 0.2.0 of the Blind Solver script. The main new feature is that it now supports writing a log file of the center coordinates and some other data, as requested by Vincent Peris in http://pixinsight.com/forum/index.php?topic=1857.msg11433#msg11433, to allow for creating pointing files. I am sure the community will find other good uses for this feature as well. The GUI allows to select the log file via a file dialog. If the file does not yet exist, Blind solver adds a header line. After this, each solver run appends to the file. Example:
Code: [Select]
filePath;viewId;parity;orientation;ra_center_hms;dec_center_dms;ra_center;dec_center;fieldw;fieldh;fieldunits
/home/georg/host/Pictures/2009_07_16_dachfenster_35mm/light (77).CR2;P7v2uLpcyj;1;171.35971;18:48:15;+1:32:10.5;282.062505381;1.53624216317;34.9;23.35;degrees
/home/georg/host/Pictures/2009_07_16_dachfenster_35mm/light (78).CR2;bIt7gqKidu;1;171.14811;18:48:54.5;+1:36:32.5;282.227113334;1.60903882256;35.38;23.43;degrees
/home/georg/host/Pictures/2009_07_16_dachfenster_35mm/light (79).CR2;ZWb8rFScfD;1;171.53073;18:50:17;+1:40:14.3;282.570871552;1.67064480014;34.95;23.61;degrees
/home/georg/host/Pictures/2009_07_16_dachfenster_35mm/light (80).CR2;dSrYMOCa7v;1;171.5969;18:51:23.2;+1:41:24.7;282.846575225;1.69018135215;34.71;23.61;degrees
/home/georg/host/Pictures/test/testleotripletprocessed3.fit;testleotripletprocessed3;1;33.150186;11:19:46.1;+13:15:25.2;169.941936086;13.2569995649;51.85;68.06;arcminutes

This CSV format can directly be read by M$ Excel -if your country uses "." as a decimal separator (as in the US or UK, Germany uses ","  >:( ). In Excel 2007, you can change the default in Excel-Options/Extended Settings.

The other changes are listed in the header of the file, copied below.

Georg

Release Notes for 0.2.0
         - BugFix: Include Parity in output
         - BugFix: get boolean parameters as booleans, not reals. May have affected
                   operation with image containers and stored instances
         - BugFix: Coordinates were not retrieved if none of the annotation options was active
         - BugFix: Grid-option for annotation did not work due to malformed command line
         - BugFix: Parameter bPlotConstellations was not properly saved in process instance (Image Container would not work)
         - Improvement: Some restructuring of sources, improved comments
         - Improvement: Output of coordinates into appended file (Vincent Peris request)
         - Improvement: avoid creating unnecessary plot files if no diagnostic files are requested to reduce run time
         - Improvement: report exceptions to console. Were silently caught previously.
Title: Re: Blind Solver 0.3.0
Post by: georg.viehoever on 2010 May 16 13:14:04
Hello,

another improvement, released as version 0.3.0: I have added a "SIMBAD" button that will open Firefox directly with the found coordinates in the SIMBAD database http://simbad.u-strasbg.fr/simbad/ . See screenshot below.

Enjoy!

Georg
Title: Re: Blind Solver 0.4.0
Post by: georg.viehoever on 2010 May 24 09:06:47
Hi,

here is a new version 0.4.0 of the Blind Solver script. It has a new option "FITS Keys" that will create a new image that has the WCS information embedded into FITS Keywords. This has the advantage of enabeling us to process it with other FITS+WCS aware astronomy tools, such as Aladin (http://aladin.u-strasbg.fr/) or ds9 (http://hea-www.harvard.edu/RD/ds9/, also available in the Fedora repositories). Currently, the image creation is a bit clumsy due to the FITS Keyword issues in PJSR (http://pixinsight.com/forum/index.php?topic=1908.0), but I hope to clean up the whole procedure when PI1.6.1 becomes available.

I used an early version of the script to create the annotated picture in http://pixinsight.com/forum/index.php?topic=1943.msg12248#msg12248. I plan to follow up with another example of how this can be used shortly.

Cheers,

Georg
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 May 24 09:39:35
Hi,

here is another example of how solved images can be used (see updated Blind Solver script in previous post). This is an annotated version of the image that originally triggered the whole Blind Solver Project, somewhere in the Virgo Cluster. I used Aladin to annotate the solved image, using the astronomical databases available to Aladin. In yellow are galaxies, in red other objects that have an entry in the Simbad database. Galaxies that have a measured magnitude have been annotated with names, and most of them are also visible in the image. All other galaxies are just ovals, and most of them are invisible in this shot.

The limiting magnitude of this image is around mag17 (determined by comparing visible stars to entries in the USNO-B1.0 catalog). The image also contains a visible quasar (the QSO right to IC796). I never thought I would catch one of these....

Enjoy,

Georg
Title: Re: Blind Solver
Post by: Nocturnal on 2010 May 24 14:38:03

Hi,

Georg asked me to give this new script a try which of course immediately failed as I tried this on a Windows box. A quick read of this thread explained why :)

I use astrometry.net all the time, in fact I routinely solve my images and post the resulting URL next to my images in my gallery. The other neat thing astrometry.net does is provide a Google Earth KMZ file that will show your solved image overlayed in space. As you take more and more images you can see the patches of sky you've imaged. Neato.

Anyway, solving images through astrometry.net is pretty easy so I don't miss a built in solver just yet. I already have Elbrus to add WCS data to my FITS files if I want. Clearly there are some interesting uses for having the image coordinates, angle and scale defined and accessible to PI. I used the view in Google Earth of 4 mosaic panels to rotate them into place and figure out the right mosaic order. PI could potentially do this automatically but frankly I think this is one of those 'high cost - low benefit' features. If GE can already show mosaic panels so users can help the PI mosaic feature that's fine for me.
Title: Re: Blind Solver on 1.61
Post by: georg.viehoever on 2010 August 02 16:32:38
Hi,

here is a new version of the Blind Solver script. Main change is that it no longer needs the SystemCommand module, and thus should run on PI 1.61 (Linux) where SystemCommand is not yet availble. Apart from that, I made only slight changes to error handling.

Instead of SystemCommand, the script is using a JavaScript replacement instead. This replacement requires that you allow JavaScript to execute external commands. This setting can be changed in "Edit > Security Settings > Allow execution of commands from JavaScript scripts".

Enjoy,
Georg
Title: Re: Blind Solver
Post by: Nocturnal on 2010 September 02 11:04:19
Hi,

just wanted to report I had some success building astrometry .34 using Cygwin. Unfortunately it doesn't solve my images yet but it does solve the examples so it's not completely dead. Working with Dustin to figure out what's going on.

One of the remaining problems is that plotxy reports netpbm isn't installed when in fact it is. Somehow the makefile doesn't detect netpbm, I'll have to figure out why. Without the graphics abilities the solver will still add WCS headers to FITS images which is very cool but I want the automated plotting as well.
Title: Re: Blind Solver
Post by: Niall Saunders on 2010 September 02 11:46:21
Georg,

I am trying to follow the development here, and when I see a reference to <Cygwin> my mind tells me that this suggests that BlindSolver may yet be capable of running on a Windoze machine - am I right, or am I just trying to be 'too clever' ???
Title: Re: Blind Solver
Post by: Nocturnal on 2010 September 02 11:50:25

A quick look at the cygwin site will tell you you are correct :)

That said it appears the astrometry install and build will not be trivial as it depends on lots of packages being installed to make the runtime work. This goes for both Linux and Cygwin installs. It is not clear to me that a pre-built cygwin astrometry.net distro can even be put together.

The future of this script may well be to integrate with astrometry.net rather than doing the solving locally.
Title: Re: Blind Solver
Post by: Juan Conejero on 2010 September 02 12:08:51
Quote
The future of this script may well be to integrate with astrometry.net

Good point, but depending on an online service is problematic and limits functionality. Another solution would be integrating astrometry.net's source code (or relevant portions of it) in an open-source PixInsight module. That would solve all portability problems. It would be a hard task for sure, but I think it is perfectly doable. Unfortunately the GPL license prevents this possibility.
Title: Re: Blind Solver
Post by: Nocturnal on 2010 September 02 12:13:46

Right, there are certainly drawbacks to depend on online services. So I keep Elbrus around if I'm desperate to solve an image :)
Title: Re: Blind Solver
Post by: Nocturnal on 2010 September 02 12:22:37
As far as GPL goes it's quite possible that the a.n team is willing to grant us a license if we write an open source PI solver based on their code. I haven't looked at the code in detail but it appears that most functionality is already organized in libraries so that would simplify things a bit. Clearly the drawback of having a large collection of executables glued together with python is that converting a set of these executables into a single dll could be tricky if there are data naming conflicts.

So I guess the question is "who would want to take on this project and do it for free" :)
Title: Re: Blind Solver
Post by: Nocturnal on 2010 September 05 14:58:31
Ok, with some makefile futsing I got astrometry.net to compile completely using cygwin. I'll be trying out the script and see if that can be made to work as well.
Title: Re: Blind Solver
Post by: Nocturnal on 2010 September 05 16:20:27
Hmm, that's not working. I didn't see any security settings to modify. Is this generally disabled for windows?

Code: [Select]
Writing file:
C:/Users/sander/AppData/Local/Temp/PixinsightTmpDir/orig.jpg
Compressing JPEG: 3 channel(s), 2994x1977 pixels: 100%
BlindSolverEngine.solveFile():commandLine=time solve-field -u arcminwidth -L 200 -H 250 --no-plots --downsample 2 -d 30 C:/Users/sander/AppData/Local/Temp/PixinsightTmpDir/orig.jpg
Executing external command
exception caught in solveFile:Error: Console.execute(): operation forbidden as per global security settings.
BlindSolverEngine::deleteDir(): dirName=C:/Users/sander/AppData/Local/Temp/PixinsightTmpDir
BlindSolverEngine::deleteDir(): names=C:/Users/sander/AppData/Local/Temp/PixinsightTmpDir/orig.jpg
BlindSolverEngine::deleteDir(): fileName=C:/Users/sander/AppData/Local/Temp/PixinsightTmpDir/orig.jpg
BlindSolver Script done
*** Error [000]: C:/PCL64/dist/BlindSolver5.js, line 836: Could not solve this image
Title: Re: Blind Solver
Post by: Juan Conejero on 2010 September 06 09:58:02
Hi Sander,

Quote
Error: Console.execute(): operation forbidden as per global security settings.

Select Edit > Security Settings and enable 'Allow execution of commands from JavaScript scripts'. An improved security system will be implemented in the next version, which is much more flexible and can operate on a per-module and per-script basis. For now, you have to enable a global security option to get this script running.
Title: Re: Blind Solver
Post by: Nocturnal on 2010 September 06 10:03:25
Doh! Never noticed that menu item before. It's interesting that that is one of the few traditional dialog boxes with 'apply' and 'cancel' buttons.
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 September 10 08:25:35
Hi Sander,

see also http://pixinsight.com/forum/index.php?topic=1857.msg11347#msg11347 where Ioannis tried a Cygwin port. Even when you get it going, I would expect that it is slow due to Cygwin's slow file operations. The Blind Solver is rather disk intense...

Cheers,
Georg
Title: Re: Blind Solver
Post by: Nocturnal on 2010 September 10 09:45:47
Hi Georg,

when I had it working it was fast enough to be useful. Probably because I have reasonably wide fields and constant resolution. I tried to solve my most recent image and that didn't work though. I've asked Dustin for help. Neither my Ubuntu nor Cygwin compiles can solve it but a.net solved it in seconds. I must be doing something wrong.
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 September 13 07:12:53
Sander,

in my examples, using the downsample option (I think something like -d 2) usually helped. Also, BlindSolver seems to be better at solving unprocesssed images that processed images. In some processed images, it helps to make them more dim (e.g. with moving the midtones slider in Histogram transform to the right). I also had a few images that I was not able to solve, such as Vicent's M51 (which is heavily processed), but I dont know if the web version would have solved these.

Georg
Title: Re: Blind Solver
Post by: Nocturnal on 2010 September 13 07:17:09
Hi Georg,

the web version never has any trouble solving my images as long as I don't screw up :) It'll happily solve in a few seconds in most cases. Unless they have some studdly mega-cloud computer sitting behind astrometry.net I have to think I can do equally fast on my PC. So far that's been elusive so I'm still trying to work this out. My goal is actually to add more objects to the NGC catalog as I think it's far to shallow right now. I've asked Dustin to add more catalogs to his lookup but got no response on this so far. Looking at the code it's clear why. The objects are hardcoded in C files, it doesn't use catalogs.
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 September 13 07:29:34
Sander,

I think the machine behind the Astronomy.net solver web server is an 48 GB multicore server. The key factor for its speed is that all catalogs fit into its RAM... at least thats what I believe. Blind Solver gets slow when going into the larger catalogs>1GB on my machine, which is plausible for a 2 GBytes machine.

I cannot help with catalog building, but I understand that all necessary tools are part of the astronomy.net distribution..

Georg
Title: Re: Blind Solver
Post by: Astrocava on 2010 December 23 08:26:20
Sander,

in my examples, using the downsample option (I think something like -d 2) usually helped. Also, BlindSolver seems to be better at solving unprocesssed images that processed images. In some processed images, it helps to make them more dim (e.g. with moving the midtones slider in Histogram transform to the right). I also had a few images that I was not able to solve, such as Vicent's M51 (which is heavily processed), but I dont know if the web version would have solved these.

Georg


Hi George,

Would be possible to solve a unprocessed image and then use the output into the processed one?

Have fun!

Sergio
Title: Re: Blind Solver
Post by: georg.viehoever on 2010 December 23 11:11:46
Hi George,

Would be possible to solve a unprocessed image and then use the output into the processed one?

Yes, that would be possible. Just copy the FITS keywords that are set in the solved image.

Georg
Title: Re: Blind Solver
Post by: pfile on 2011 March 02 23:05:31
hmm, on osx x86_64 i'm getting "exception caught in solveFile:TypeError: Console.execute is not a function"

sure enough in the javascript editor, i don't see a method called "execute" under console.

so is there an alternate way to execute an external program from javascript on osx, or is this just an osx x86_64 weirdness?
Title: Re: Blind Solver
Post by: georg.viehoever on 2011 March 03 11:15:03
hmm, on osx x86_64 i'm getting "exception caught in solveFile:TypeError: Console.execute is not a function"
...
Your observation is correct. I posted a bug report, see http://pixinsight.com/forum/index.php?topic=2903.0.

Georg
Title: Re: Blind Solver
Post by: pfile on 2011 March 03 11:49:08
cool, thanks.

didnt realize it was a 1.6.9 thing, figured it might have been os dependent.
Title: Re: Blind Solver
Post by: kwiechen on 2012 April 02 09:01:41
This is very interesting. Are there any plans to make a module? I do not understand why it is not possible to use the GPL here.

Kai

Title: Re: Blind Solver
Post by: georg.viehoever on 2012 April 02 09:06:23
No time to work on this. You may want to use the Solver/Annotation script that does something very similar http://pixinsight.com/forum/index.php?topic=3966.75 .
Georg
Title: Re: Blind Solver
Post by: kwiechen on 2013 January 27 12:26:00
I have looked again at the astrometry.net code. The solve-field binary calls other programs to separate the positions of stars and writes a xy-list sorted by signal descending. This is written to a FITS bintable with some information in the header (image size and flags) and submitted to the backend binary that does the job.

I have tried to extract the xy-list using code from the the SubFrameSelector script and this should work. The xy-list is converted to a FITS bintable using a modified xylist2fits binary from the astrometry.net suite and then the modified backend is called using ExternalProcess. The output is collected via stdout and written to the FITS header. The process of extracting star positions with StarAlignment is time consuming because there is no access to the internal data table. And there is some overhead because PI is not able (??) to write and read FITS bintable data and the resulting wcs header.

In the current state the resulting data is unfortunately not compatible with the ImageAnnotation script but OK as a starting point for the ImageSolver script. In addition the binaries are linux only but it should be possible to use the cygwin dlls (the relevant C source files were tranfered to a cmake build tree - the project compiles with cygwin, but the backend is still not usable in a windows environment because of problems with wildcard expansion). I will do some cleanup and then post the files.

Kai
Title: Re: Blind Solver
Post by: Juan Conejero on 2013 January 27 12:37:56
I plan on releasing a StarDetector process, which will fill this gap nicely. It will be available soon (perhaps it will be part of the final 1.8.0 release in February). This process is actually rather easy to write because all the necessary pieces already exist in StarAlignment and DynamicPSF. Just a bit of patience! :)
Title: Re: Blind Solver
Post by: netwolf on 2013 September 19 19:39:52
Georg, Does your blind sovler script work with the Astrometry.net beta web portal? Or does it only work with a local copy?

I am not sure on the legal side of these things but would it not be possible to have ImageSolver have a link to nova.astrometry.net/upload  the beta web API for astrometry.net? I know sequencegenerator pro has this in there software so I am guessing it is ok to use this service in such a way.

Title: Re: Blind Solver
Post by: rleisenz on 2013 September 19 22:03:23
i don't know if somebody mentioned it, there is a solution
called astrotortilla, which is also based on astrometry.net.
works like a charm on my computer.
http://sourceforge.net/p/astrotortilla/home/Home/

Robert
Title: Re: Blind Solver
Post by: georg.viehoever on 2013 September 19 23:48:07
Hi Robert, Netwolf,
my script used a local copy of the Astrometry.net software - but this was long ago. Today PI has the ImageSolver script that will find a plate solution as long as you roughly know the coordinates of your image (kind of half blind). Plus it will handle distortions and more.

I know about AstroTortilla, but never tried it.
Georg
Title: Re: Blind Solver
Post by: netwolf on 2013 September 20 01:08:02
Robert, yes I am aware of Astrotortila i did try to install it some time ago but had problems with it. Might give it another try.
I have Elburs installed for use with SGPro but that does not do blind solve.

Georg, yes I have tried the Imagesolver and can see it does link in with extenral websites , just wondering if this could be modified to use the nova.astrometry.net web portal, I think there is meant to be an API around this web portal to allow remote use.


Regards
Fahim