PixInsight Forum

PixInsight => General => Topic started by: georg.viehoever on 2014 June 08 15:15:17

Title: Using Amazon EC2 with PixInsight
Post by: georg.viehoever on 2014 June 08 15:15:17
This message describes how to run PI on a Windows system rented at Amazon’s Elastic Cloud 2 (EC2) service that is avaiable as part of Amazon Web Services (AWS). At 0.30 EUR/hours this is a realistic alternative to replacing an outdated laptop.

Motivation
=======

So far, I did all of my PI work on my Toshiba Windows laptop. Win7x64 home edition, 4GB RAM, Intel Core 2 Duo T7300@2GHz. The new Drizzle functionality basically means that RAM is no longer sufficient, and general speed-especially for Batch Preprocessing- also was no longer really good.
This text investigates option 2.

Overview
=======
To get an instance (=a computer in EC2) running, you need to perform the following steps that are discussed in more detail below. I am trying to explain Amazon speak as we go along.

Once you have some experience with this, it will take you maybe 1 minute to re-launch a new instance, and maybe 10 minutes to completely configure an instance for PI. The first try may take 60 minutes or so.

Setup
====

The starting point of this description is http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/IntroWindowsUserGuide.html . The documentation there gives a lot of detail that is not necessary to get started with PI.

1. Create AWS account and identity
Create account and log in on http://aws.amazon.com. Following http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/get-set-up-for-amazon-ec2-windows.html here. Note that this is the “super user” of anything you do in AWS.
2. Security
Define security settings. Create a key pair for identity "georgvAdmin". The key pair is used to communicate secrets (such as passwords) to you. Also set a Security Group (network permissions) for your instances. In the AWS console, select the EC2 tool.

2. Creating an Instance
An instance is a computer/system in Amazon speak. You need to create it using EC2 the EC2 tool.  Following http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/EC2Win_GetStarted.html here:

3. First Login
In this first login, we will configure additional users and new passwords:

Normal login
=========
This is the normal login procedure (plus PI installation).

Using your Instance
==============
You can now use the system. Some hints:

Some Tips
=======


Costs
====
Costs: Study the Amazon pages http://aws.amazon.com/ec2/pricing/ . Here only a short summary:

Some performance data in the next post.
Georg

Edit: Making the creation of new identity optional. Adding overview section.
Title: Re: Using Amazon EC2 with PixInsight
Post by: georg.viehoever on 2014 June 08 15:32:21
Some performance data
=================

Using the PI Benchmark
Instance typePi Vers.Run Serial numberRemark$/hTotalCPUSWAPTransfer [MiB/sec]Total time
olde Laptop01.08.01.1092-only 4GB RAM, swap on C: system harddisk-1742906612.0072703.5
M3.large01.08.01.1092L19X57R9K4MUX4TM1P6C99I89Q288K8Mswap on C: (EBS)0.266618117920937.66761.18
M3.large01.08.01.1092S9BPN02252O9KC24EA1XB0F9JPB50C11swap on Z: (SSD)0.26611361178990178.694413.97
C3.2xlarge01.08.01.1092GY4H434TDG43R988XLN8P0I3GY0Y39WSStart of stopped instance with new instance type. PI did not automatically pick up new SSD.0.752943448122239.994498.98
C3.2xlarge01.08.01.10922CUG40K63WW5SEHSX2XVJTJIT69NPQB6Swap on Z: (SSD). 2nd SSD invisible0.752323744421528275.833145.34
C3.2xlarge01.08.01.1092RS8DTB6J233S5JD9VS17G11Q92IP83HQNew Instance. Swap on  Y:+Z: (SSD)0.752404245492770500.158116.38
C3.8xlarge01.08.01.109206E7FZN18YTHUNKCYN2186XJVBHVZAFQSwap on  Y:+Z: (SSD)3.008681493723211579.72069.03

With version 01.08.01.1092:

Georg

Edit: Data for c3.8xlarge, and some informal performance tests with laptop
Edit: Added Laptop Benchmark data
Edit: Added PI Version Info
Title: Re: Using Amazon EC2 with PixInsight
Post by: NGC7789 on 2014 June 08 15:33:06
I'm going to have to make sure my wife doesn't see this, depending on the outcome, given that I just built a new Hackintosh justified by the demands of PI.  :moneyinmouth:

EDIT: Just saw those numbers. If I read them right, that's not very impressive.
Title: Re: Using Amazon EC2 with PixInsight
Post by: georg.viehoever on 2014 June 08 15:44:04
I would say the performance numbers are medium range. But then: The instances I tested are by no means the fastest ones. An they just cost me $2 for the half day that I did that work.
The best Windows system registered on http://www.pixinsight.com/benchmark/ so far on is Total=9363, CPU=8905 ,SWAP=11922 ,Transfer=2152.55, total time=50.24, so a  C3.2xlarge instance  is at about 40% of this...

Georg
Title: Re: Using Amazon EC2 with PixInsight
Post by: NGC7789 on 2014 June 09 13:12:39
This is starting to look very interesting. The c3.8xlarge config beats my hackintosh on the benchmarks (at least for OS X). How would your rate the hassle of getting your data up and back? Although I guess such things can be done while sleeping. Could it be that my hackintosh is the last computer I ever buy for performance? Might this be the true end of the desktop? Ipad for surfing, Cloud for crunching.
Title: Re: Using Amazon EC2 with PixInsight
Post by: georg.viehoever on 2014 June 09 13:38:32
The transfer of (raw) data certainly is a bottleneck. I did it via Windows Explorer which claimed a speed of 350 kBits/sec, and I calculated something around 0.11 MBytes/sec. Dont know if Windows does any compression. I did not try ftp, rsanc or other transfer methods (for instance via AWS S3 service) - just using the Windows tools is so easy...

The data transfer indeed can happen in the background/during night, and you can do it with one of the dirty cheap (and slow) instance of type t1.micro @ 2cents/hour. Worked flawlessly.

Costs: I am now at $7 after playing for 2 days with several instance types, with 2 cents for data transfer....

Georg
Title: Re: Using Amazon EC2 with PixInsight
Post by: Astrocava on 2014 June 13 11:19:11
I've been using profitbricks (http://www.profitbricks.com/). It works fine using desktop remote access. The possibility of increasing some resources (cores or RAM) in a easy way is remarkably. As Georg said, the worst thing with this cloud machines is the transfer of files between your computer and the cloud, more if you have a slow connection as mine. Uploading hundreds of BIAS was a nightmare. I used a windows server (some cents more per hour) with MS Skydrive: First I upload the files to skydrive and then I took them from there to start processing.

After 3 or 4 objects, I've ended bored. I have about 20 objects from last year unprocessed.  ::)

I think I need to replace my Intel Core 2 6400 and 3Gb RAM with XP to something more new...   :laugh:

Sergio
Title: Re: Using Amazon EC2 with PixInsight
Post by: NGC7789 on 2014 June 27 11:51:16
the worst thing with this cloud machines is the transfer of files between your computer and the cloud

I which there was a service that you could bring a hard drive to a store front and they would do the upload for you. They could have some super-fast business connection and you could pickup up your drive the next day or something. Maybe I will ask next time I'm in a copy shop. Seems like something they could do.