Deploylib

#Deploylib

Deploylib is a scala tool for setting up and interacting with a cluster compute nodes, either privately owned cluster or an Amazon EC2 cluster. It is also integrates with Mesos, the cluster management platform, so it is capable of launching and managing Mesos masters, slaves, and Mesos applications. In particular, there is a framework for running and managing Java or Scala services (jars) and web services (wars via Jetty).

Setting up deploylib

Prerequisites

Install SCADS (see the "How to install and build SCADS" section of the wiki Home). The recommended installation is via SBT.

Environment Variables

In addition to the prerequisites, you must set the following environment variables:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_KEY_NAME
• AWS_KEY_PATH
• EC2_PRIVATE_KEY
• EC2_CERT

Using deploylib

Once you have set up deploylib, you should be able to use it by doing the following:

1. Type bin/mvnconsole (warning: this is misleadingly named mvnconsole because it used to use Maven (i.e. mvn), however it not longer uses MVN. Thus the name may change to console soon). This should drop you into a scala console. Remember, tab completion should work here.
2. Type :load deploylib/setup.scala. This should execute several import statements, such as logging, mesos, etc. The output should look as follows:

  scala> :load deploylib/setup.scala
  Loading deploylib/setup.scala...
  import net.lag.logging.Logger
  import deploylib._
  import deploylib.ec2._
  import deploylib.mesos._
  import deploylib.rcluster._
  import net.lag.configgy._
  toFile: (str: String)java.io.File
  debug: Unit
  updateScads: Unit
  stopAllInstances: Unit

1. To get a list of all EC2 instances you have running, type: EC2Instance.activeInstances. It should return something like the following:

scala> EC2Instance.activeInstances
INF [20110107-18:12:32.375] ec2: Updated EC2 instances state
res0: List[deploylib.ec2.EC2Instance] = List(<EC2Instance i-c9b64ba5>, <EC2Instance i-25ad5049>, <EC2Instance i-27ad504b>)

Using deploylib with Mesos

You can use deploy lib to start and manage a Mesos cluster running on EC2. To familiarize yourself with Mesos, check out the Mesos homepage.

Setup a Mesos Cluster

• To start a Mesos master type: MesosEC2.startMaster
• To add a Mesos slave (which will spin up a new EC2 instance) type: MesosEC2.addSlaves(<num_slaves_to_add>)
• You can call halt() on any object of type Class (or descendent of class, such as MesosEC2.master.halt())

Running a Web Application

Deploylib can launch a webapp that has been bundled into a Web ARchive (WAR) on the cluster via a Mesos "Framework" (see The Mesos Framework Dev Guide for more about how Mesos Frameworks work.

1. Build your webapp into a war.
2. Deploy deploylib to the node you want to run the framework on (typically the master) and all of the Mesos slaves. (Note: If you are using EC2, you cannot
3. Deploy the
4. TODOAdd the step here about using jrun to launch the framework, have it connect to tphe

Building a new Mesos Framework inside of deploylib

Here are some notes I took while we were writing a new Web App Mesos Framework Scheduler in deploylib:

Note: I assume that you are using the default JavaExecutor.scala for your Mesos executor, which can run any arbitrary Jar or War (in a Jetty instance) for you.

1. Make changes to your Scheduler (mine is
2. Build the webframework
3. Push your jars to the Mesos master