A primer for running data science workloads on cloud resources.
Slide Deck for DS6051.
Watch a walkthrough:
A UVA-managed catalog of prepackaged solutions and services. Provides self-service provisioning of cloud resources.
Sign in here using your UVA credentials:
Using a terminal either create or us an existing SSH keypair. To create a new pair:
ssh-keygen -t rsaYou can specify a name if you like, or accept the defaults. Note the public key and private key. (The latter must be chmod 600 for security purposes.)
Do not share your private key with anyone. Rotate keys at least 1x a year.
COPY and paste your public key into the AWS Service Catalog when you provision a Keypair.
Once your EC2 instance has been created (following the steps below) you will use your SSH keypair as a means of authentication. Cloud instances tend not to have system usernames/passwords, as SSH keys are more secure.
To connect to your instance, you need to (1) have the private key associated with the instance; have (2) the default username to connect as, and (3) the public IP address of your new instance. Together with your private key you can establish a connection:
ssh -i ~/.ssh/my_key [email protected]
Once you have signed into the AWS Access Portal above, complete the following steps to create a new EC2 instance:
- Find the AWS-SDS-Agarwal account and expand the box.
- Click into the
ServiceCatalogReadOnlyaccount, which will open up the AWS Console home page. - In the upper-right corner, be sure you are in the "United States (N. Virginia) region.
- Using the search box at the top, enter "Service Catalog" and click into it. You can also click the star next to the service name to bookmark it in the top of your browser.
- Within the Service Catalog, select "Products" under the "Provisioning" section in the left hand navigation.
Provision an SSH Keypair
- First select the Keypair and then launch the product.
- We suggest you provision the product name as "uva-xxxx-keypair" where you replace
xxxxwith your computing ID. For the keyname, we suggestuva-xxxxwhere you replacexxxxwith your computing ID. - In the
PublicKeyfield, paste in the single line contents of your SSH public key. - Finally, click "Launch product" to complete the task.
Provision an EC2 Instance
- From the Service Catalog list of "Products" select the "Ec2 Ml" product, then Launch Product.
- For Provisioned Product Name, we suggest naming it
xxxx-ds6051-instancewhere you replacexxxxwith your computing ID. - Leave the Instance Type as
t1.micro. - Select your keypair from the list of available keys.
- Select either subnet in the
SubnetIdfield. - Under "Manage Tags" we suggest you add a custom tag. Click the "Add new item" button, and create a field with a key
Nameand a value of your computing ID (i.e.mst3k,nem2p, etc.) - Finally, click "Launch Product". Your new instance will be provisioned in 1-2 minutes. You can refresh the page using the refresh (circular arrow) button. Once complete, near the bottom click into the "Outputs" tab to see your instance attributes. The most important attribute is the
PublicIpwhich you will need to connect.
Assuming you just created your instance, and you are still on your provisioned product details page, note the "Actions" dropdown in the upper-right corner.
To stop your instance, select:
- Actions -->
- Service Actions -->
- EC2-Stop
To start or restart your instance, select:
- Actions -->
- Service Actions -->
- EC2-Start / EC2-Restart
For cost management purposes it is important that you turn off your instance whenever you are not using it.
To resize or re-type your instance, i.e. extend or reduce its CPU/MEM/GPU resources, do the following:
- Stop your instance. Confirm this action.
- Next, select the "Update" option from the Actions menu.
- Change the InstanceType as necessary.
- Click Update and wait for the change to propagate.
- Finally, start your instance again from the Actions menu.
BE SURE if you resize up to a trn1.2xlarge instance type to resize your instance back down to a t1.micro after you are done.
sudoapthtopcrondf -hdu -sh