Author Topic: Using Amazon EC2 with PixInsight  (Read 3571 times)

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2131
    • View Profile
Using Amazon EC2 with PixInsight
« on: 2014 June 08 15:15:17 »
This message describes how to run PI on a Windows system rented at Amazon’s Elastic Cloud 2 (EC2) service that is avaiable as part of Amazon Web Services (AWS). At 0.30 EUR/hours this is a realistic alternative to replacing an outdated laptop.

Motivation
=======

So far, I did all of my PI work on my Toshiba Windows laptop. Win7x64 home edition, 4GB RAM, Intel Core 2 Duo T7300@2GHz. The new Drizzle functionality basically means that RAM is no longer sufficient, and general speed-especially for Batch Preprocessing- also was no longer really good.
  • Option 1: By new system, sets me back 800-2000 EUR
  • Option 2: Rent system at >=0.30 EUR/h at Amazon EC, if speed is needed.
This text investigates option 2.

Overview
=======
To get an instance (=a computer in EC2) running, you need to perform the following steps that are discussed in more detail below. I am trying to explain Amazon speak as we go along.
  • You need to setup an account with AWS. Optionally, you can also set up identities (AWS user ids managed by you) using Identity and Access management (IAM)
  • You create a cryptographic key pair that helps you with secure communication, and describe the internet access permission (~firewall) for your instances by defining a security group.
  • You boot your instance by selecting an instance type (=hardware) and and Amazon Image (AIM, operating system software)
  • After booting, you connect to this instance, configure users and install software
  • Now you remote system is ready for use, allowing GUI access and copy between your local and remote drives.
  • If necessary, you can switch the instance type to something stronger, shutdown/reboot the system, ... . You only (mostly) pay for what you use.
  • I am including a couple of hints for optimal use and expected performance.

Once you have some experience with this, it will take you maybe 1 minute to re-launch a new instance, and maybe 10 minutes to completely configure an instance for PI. The first try may take 60 minutes or so.

Setup
====

The starting point of this description is http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/IntroWindowsUserGuide.html . The documentation there gives a lot of detail that is not necessary to get started with PI.

1. Create AWS account and identity
Create account and log in on http://aws.amazon.com. Following http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/get-set-up-for-amazon-ec2-windows.html here. Note that this is the “super user” of anything you do in AWS.
  • Make sure you select the right region (the computer center you use) on the top right of the AWS management console. In my case, that’s Ireland.
  • In the following steps, you optionally can create identities (AWS users that are managed by you). You can manage their permissions (such as creating, launching, stopping instances) with a high degree of detail. As an alternative, just work with your AWS login instead of the identity georgAdmin that we create below.
    • Create new identity (=user account that belongs to your main AWS account) in the IAM tool. You can also use your AWS account directly, but Amazon recommends doing this.
    • Create new group of users “Administrators” with Administrators access rights.
    • Create new  identity (“georgAdmin”)
    • We need not download credentials here, since we use AWS management console
    • In the  IAM tool, click on “users” on the newly created identity, and use” Security Credentials/Manage Password” to set a password for this new user.
    • Take note of the AWS account ID (123456, can be seen for  instance on your AWS account), and log onto AWS console with  https://12345.signin.aws.amazon.com/console/ using your newly created identity and password
    • You may want to create similar new identities for other family members or roles. But note that any action that is made in this account will be charged to your main AWS account.
2. Security
Define security settings. Create a key pair for identity "georgvAdmin". The key pair is used to communicate secrets (such as passwords) to you. Also set a Security Group (network permissions) for your instances. In the AWS console, select the EC2 tool.
  • Using the new console, select the EC2 tool.
  • Create a key pair, and save it in a save place on your local system.
  • Create a Security Group (network permissions) that allows Windows Terminal Server Access (RDP) to your instance. Here: Name RDPWindows, no VPC, Allow Inbound RDP from Anywhere

2. Creating an Instance
An instance is a computer/system in Amazon speak. You need to create it using EC2 the EC2 tool.  Following http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/EC2Win_GetStarted.html here:
  • Open EC2 tool
  • Select Launch Instance.
  • Select the AMI  (=base software setup) to be Microsoft Windows Server 2012 R2 Base (currently ami-27539050). This is similar to Windows 8…
  • Choose Instance type (=hardware of system). I am using m3.large here because it is the cheapest instance with >6GB RAM ($0.266 per Hour according to http://aws.amazon.com/ec2/pricing/)
  • Select  Review and Launch.
  • Be sure to select the RDPWindows Security Group
  • You may want to change the size of your root EBS device (30 GB default, roughly 12 GB free).
  • Click “View Instances”. When your instance is running, you can log in. It may take a minute or so until you can actually log in, even if the console shows “running” as status.

3. First Login
In this first login, we will configure additional users and new passwords:
  • Take note of the public DNS of your instance as displayed in the EC2 console, something like "ec2-54-217-186-177.eu-west-1.compute.amazonaws.com". You need this to talk to the system later.
  • In EC console, press “Connect”. Download and launch the RDP file. This launches Windows Remote Desktop and connects to your instance. You’ll see a login screen.
  • in EC2 consolse "Decode Password" (using your key pair).
  • Using Remote Desktop and your password, Log in as "Adminstrator". Ignore the certificate error. You now should have a Windows desktop in front of you.
  • Change your Administrator Password using Control Panel/User Accounts/Manage Accounts. You need to go to “Remove Accounts” to be able to change passwords and add new users !
  • Create an ordinary user ("georg") in using Control Panel/User Accounts/Manage Accounts. You will be using this account for everything else.
  • in Control Panel/System and Security/Allow Remote Access/Remote/Select Users you need to add the new user georg. Otherwise georg does not have remote access!
  • Sign out

Normal login
=========
This is the normal login procedure (plus PI installation).
  • Start "Remote Desktop Connection" from the Start Menu of the local system. Do not use the Connect button from EC2, it will not allow you to share disks.
  • Enter the Public DNS name you took down above in General tab as host name.
  • In Local Resources tab, share your disks.
  • After this, you can copy files between the local system and the EC2 instance using Explorer.
  • log in as the normal user (georg, not Administrator) using the name and password you set above
  • Install PI as usual. Install your license file into your user directory as described in http://pixinsight.com/faq/index.html point 2.7. Launch and Update…
  • If possible, configure the local disks (instance stores/ephemeral disks, Y/Z) as swap devices for PI (Edit/Global Preferences/Directories and Network).  See performance section below. The root directories do not work, use something like Z:\SwapPi and Y:\SwapPi.

Using your Instance
==============
You can now use the system. Some hints:
  • Work with this system. You’ll be charged ~0.30 EUR while it is running.
  • When you are done, stop your instance in the EC2 console. This is similar to shutting down your personal computer.
    • You can stop the system. You will no longer be charged for the instance.
    • You will be charged for the storage on your C:-drive (EBS device) . With the default setup of m3.large this is 30GB (at $0.055 GB/month)
    • When you relaunch your stopped instance, all your software/data/setttings on C: will be there.
    • The data on the instance store (also called ephemeral storage, Z: 30GB SSD) will be lost on relaunch.
    • Note that the DNS name of a relaunched instance will change.
    • A stopped instance can also be launched on a difference instance type, for instance if you need faster hardware. Change it in EC2 console/Instance/Actions/Change Instance type. Note that you may not be able to use all instance storage in this case.
  • You can also terminate the system. This is similar to throwing away your system. After removing your EBS store, all your data/user accounts, installed software, data will be lost. There are no costs because you don’t use anything. You’ll need to start over with your very first login as Administrator. (Note: You can preserve the EBS storage, but this is similar to stop/start).

Some Tips
=======
  • Make sure you use the local disk (instance storage/ephemeral storage, for instance Z: but not C:) as your PI swap device. It makes a lot of difference – see Performance section below.
  • Be sure to delete a Disk (EBS volume) that you no longer need when you terminate (instead of stop) an instance.
  • When launching with a new instance type, sometimes the additional instance stores are not found (see performance section). I needed to create a new instance to get them with C3.2xlarge.


Costs
====
Costs: Study the Amazon pages http://aws.amazon.com/ec2/pricing/ . Here only a short summary:
  • The instance costs (depends on instance type, charged per launch in units of one hour) and data transfer into Amazon (0 cents/GB) are simple to calculate.
  • Amazon charges for data stored on their storage systems (EBS) even when the instance is not running, something like $0.055GB/month. They also charge for data transfer from/to those systems ($0.055 per 1 million I/O requests)  - which is kind of difficult to estimate, but never was a relevant factor for me.
  • Amazon charged for data transfer out of Amazon, <=$0.12/GB
  • Please add VAT to all prices.

Some performance data in the next post.
Georg

Edit: Making the creation of new identity optional. Adding overview section.
« Last Edit: 2014 June 09 05:21:30 by georg.viehoever »
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2131
    • View Profile
Re: Using Amazon EC2 with PixInsight
« Reply #1 on: 2014 June 08 15:32:21 »
Some performance data
=================

Using the PI Benchmark
Instance typePi Vers.Run Serial numberRemark$/hTotalCPUSWAPTransfer [MiB/sec]Total time
olde Laptop01.08.01.1092-only 4GB RAM, swap on C: system harddisk-1742906612.0072703.5
M3.large01.08.01.1092L19X57R9K4MUX4TM1P6C99I89Q288K8Mswap on C: (EBS)0.266618117920937.66761.18
M3.large01.08.01.1092S9BPN02252O9KC24EA1XB0F9JPB50C11swap on Z: (SSD)0.26611361178990178.694413.97
C3.2xlarge01.08.01.1092GY4H434TDG43R988XLN8P0I3GY0Y39WSStart of stopped instance with new instance type. PI did not automatically pick up new SSD.0.752943448122239.994498.98
C3.2xlarge01.08.01.10922CUG40K63WW5SEHSX2XVJTJIT69NPQB6Swap on Z: (SSD). 2nd SSD invisible0.752323744421528275.833145.34
C3.2xlarge01.08.01.1092RS8DTB6J233S5JD9VS17G11Q92IP83HQNew Instance. Swap on  Y:+Z: (SSD)0.752404245492770500.158116.38
C3.8xlarge01.08.01.109206E7FZN18YTHUNKCYN2186XJVBHVZAFQSwap on  Y:+Z: (SSD)3.008681493723211579.72069.03

With version 01.08.01.1092:
  • Copy of 128 MB from PC disk to Amazon Z-drive (16 MBit/s DSL connection with 1 MBit/s upload speed) per Explorer copy function: 18min25sec=1120 second=0.11 MBytes/sec .
  • Histogram Transform of 3 channel float 3908x2602 image: Laptop 1.182 secs, C3.2xlarge 0.125 secs
  • TGVDenoise 3 channel float 300x300 image: Laptop 5.72 secs, C3.2xlarge 1.95 secs
  • TGVDenoise of image generated by PixInsight Benchmark: Laptop 101.3 secs, c3.8xlarge 8.594 secs

Georg

Edit: Data for c3.8xlarge, and some informal performance tests with laptop
Edit: Added Laptop Benchmark data
Edit: Added PI Version Info
« Last Edit: 2014 June 25 11:21:50 by georg.viehoever »
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline NGC7789

  • PixInsight Old Hand
  • ****
  • Posts: 391
    • View Profile
Re: Using Amazon EC2 with PixInsight
« Reply #2 on: 2014 June 08 15:33:06 »
I'm going to have to make sure my wife doesn't see this, depending on the outcome, given that I just built a new Hackintosh justified by the demands of PI.  :moneyinmouth:

EDIT: Just saw those numbers. If I read them right, that's not very impressive.

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2131
    • View Profile
Re: Using Amazon EC2 with PixInsight
« Reply #3 on: 2014 June 08 15:44:04 »
I would say the performance numbers are medium range. But then: The instances I tested are by no means the fastest ones. An they just cost me $2 for the half day that I did that work.
The best Windows system registered on http://www.pixinsight.com/benchmark/ so far on is Total=9363, CPU=8905 ,SWAP=11922 ,Transfer=2152.55, total time=50.24, so a  C3.2xlarge instance  is at about 40% of this...

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline NGC7789

  • PixInsight Old Hand
  • ****
  • Posts: 391
    • View Profile
Re: Using Amazon EC2 with PixInsight
« Reply #4 on: 2014 June 09 13:12:39 »
This is starting to look very interesting. The c3.8xlarge config beats my hackintosh on the benchmarks (at least for OS X). How would your rate the hassle of getting your data up and back? Although I guess such things can be done while sleeping. Could it be that my hackintosh is the last computer I ever buy for performance? Might this be the true end of the desktop? Ipad for surfing, Cloud for crunching.

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2131
    • View Profile
Re: Using Amazon EC2 with PixInsight
« Reply #5 on: 2014 June 09 13:38:32 »
The transfer of (raw) data certainly is a bottleneck. I did it via Windows Explorer which claimed a speed of 350 kBits/sec, and I calculated something around 0.11 MBytes/sec. Dont know if Windows does any compression. I did not try ftp, rsanc or other transfer methods (for instance via AWS S3 service) - just using the Windows tools is so easy...

The data transfer indeed can happen in the background/during night, and you can do it with one of the dirty cheap (and slow) instance of type t1.micro @ 2cents/hour. Worked flawlessly.

Costs: I am now at $7 after playing for 2 days with several instance types, with 2 cents for data transfer....

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Astrocava

  • PixInsight Addict
  • ***
  • Posts: 180
    • View Profile
    • Astrocava.com
Re: Using Amazon EC2 with PixInsight
« Reply #6 on: 2014 June 13 11:19:11 »
I've been using profitbricks. It works fine using desktop remote access. The possibility of increasing some resources (cores or RAM) in a easy way is remarkably. As Georg said, the worst thing with this cloud machines is the transfer of files between your computer and the cloud, more if you have a slow connection as mine. Uploading hundreds of BIAS was a nightmare. I used a windows server (some cents more per hour) with MS Skydrive: First I upload the files to skydrive and then I took them from there to start processing.

After 3 or 4 objects, I've ended bored. I have about 20 objects from last year unprocessed.  ::)

I think I need to replace my Intel Core 2 6400 and 3Gb RAM with XP to something more new...   :laugh:

Sergio
Moonfish ED80 over a Meade LX200GPS 8"

Offline NGC7789

  • PixInsight Old Hand
  • ****
  • Posts: 391
    • View Profile
Re: Using Amazon EC2 with PixInsight
« Reply #7 on: 2014 June 27 11:51:16 »
the worst thing with this cloud machines is the transfer of files between your computer and the cloud

I which there was a service that you could bring a hard drive to a store front and they would do the upload for you. They could have some super-fast business connection and you could pickup up your drive the next day or something. Maybe I will ask next time I'm in a copy shop. Seems like something they could do.