Deploylib

#Deploylib

Deploylib is a scala tool for setting up and interacting with a cluster of compute nodes, either privately owned or on Amazon EC2. It is also integrates with Mesos, the cluster management platform, so it is capable of launching and managing Mesos masters, slaves, and Mesos applications. In particular, there is a framework for running and managing Java or Scala services (jars) and web services (wars via Jetty).

Setting up deploylib

Prerequisites

Install SCADS (see the "How to install and build SCADS" section of the wiki Home). The recommended installation is via SBT.

Environment Variables

In addition to the prerequisites, you must set the following environment variables:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_KEY_NAME
• AWS_KEY_PATH
• EC2_PRIVATE_KEY
• EC2_CERT

Using deploylib

Once you have set up deploylib, you should be able to use it by doing the following:

1. Type bin/mvnconsole (warning: this is misleadingly named mvnconsole because it used to use Maven (i.e. mvn), however it not longer uses MVN. Thus the name may change to console soon). This should drop you into a scala console. Remember, tab completion should work here.
2. Type :load deploylib/setup.scala. This should execute several import statements, such as logging, mesos, etc. The output should look as follows:

  scala> :load deploylib/setup.scala
  Loading deploylib/setup.scala...
  import net.lag.logging.Logger
  import deploylib._
  import deploylib.ec2._
  import deploylib.mesos._
  import deploylib.rcluster._
  import net.lag.configgy._
  toFile: (str: String)java.io.File
  debug: Unit
  updateScads: Unit
  stopAllInstances: Unit

1. Start ec2 instances, if the aren't running, and setup services on the cluster by running cluster.setup().
2. Look at the javadoc API documentation to get a feeling for the Deploylib API.
3. To get a list of all EC2 instances you have running, type: EC2Instance.activeInstances. It should return something like the following:

scala> EC2Instance.activeInstances
INF [20110107-18:12:32.375] ec2: Updated EC2 instances state
res0: List[deploylib.ec2.EC2Instance] = List(<EC2Instance i-c9b64ba5>, <EC2Instance i-25ad5049>, <EC2Instance i-27ad504b>)

Using deploylib with Mesos

You can use deploy lib to start and manage a Mesos cluster running on EC2. To familiarize yourself with Mesos, check out the Mesos homepage.

Setup a Mesos Cluster

Explore and learn the deploylib API by viewing the deploylib javadoc (if these scala/java-doc are out of date, email andyk "a+" berkeley "d0+" edu and nag him to update them).

Too lazy to click through to the scala/java-doc? Here are a some of the most important functions you'll be using:

• cluster.setupZookeeper - start machines to participate in a zookeeper quorum:
• cluster.setupMesosMaster - start a Mesos master type:
• cluster.addSlaves(<num_slaves_to_add>) - add a Mesos slave (which will spin up a new EC2 instance)
  • Note: You can call halt() on any EC2Instance to turn it off.
• cluster.updateDeploylib - pushes the newest deploylib jars to the slaves from S3.