create-release.html.md.erb

---
title: Creating a Release
---

A release contains one or more pieces of software that work together.
For example, you could create a release of a service with three pieces:
two MySQL nodes and a dashboard app.

There are four fundamental elements in a release:

* **Jobs** describe pieces of the service or application you are releasing
* **Packages** provide source code and dependencies to jobs
* **Source** provides packages the non-binary files they need
* **Blobs** provide packages the binaries they need, other than binaries that
are checked into a source code repository

The following instructions use an example release that includes two jobs:
a web UI and a background worker.
The two jobs split up the functionality provided by single Ruby app,
`ardo_app` (you can use simple [gist](https://gist.github.com/antonsoroko/974924e0692aa2171229dafa5f2561b2) as app).

---
## <a id="prep"></a>Preparation ##

This section needs to be completed once.
Next, you iterate through Steps 1 through 6 until your dev release is
satisfactory.
Then you can do a final release.

### <a id="release-dir"></a>Create the release directory ###

To create the release directory, navigate into the workspace where you want the
release to be, and run:

  `bosh init release <release_name>`

Also you can add `--git` option to initialize git repository.
Use dashes in the release name.
Use underscores for all other filenames in the release.

View the release with `tree`:

<pre class="terminal">
$ tree .
.
├── blobs
├── config
│   └── blobs.yml
├── jobs
├── packages
└── src

5 directories, 1 file
</pre>

When deploying your release, BOSH places compiled code and other resources
in the `/var/vcap/` directory tree, which BOSH creates on the job VMs.
The four directories you just created, `jobs`, `packages`, `src`, and `blobs`,
appear on job VMs as `/var/vcap/jobs`, `/var/vcap/packages`, `/var/vcap/src`,
and `/var/vcap/blobs`, respectively.

### <a id="source"></a> Populate the src directory ###

Copy your source code into the `src` directory.
Alternatively, link your source code to the directory using a mechanism such as
a Git submodule or a Mercurial repo.

### <a id="strategy"></a> Choose a work strategy ###

Choose whether you want to work one step at a time or one job at a time.
For releases with just a few jobs, going one step at a time is probably easiest.
If you have a larger number of jobs, going one job at a time may be more efficient.

---
## <a id="job-skel"></a> Step 1: Create Job Skeletons ##

Navigate into the release directory.

For each job, create a job skeleton:

  `bosh generate job <job_name>`

In our example, we run `bosh generate job` twice, once for the `web_ui` job,
and once for the `bg_worker` job.

View the job skeletons with `tree`:

<pre class="terminal">
$ tree .
.
├── blobs
├── config
│   └── blobs.yml
├── jobs
│   ├── bg_worker
│   │   ├── monit
│   │   ├── spec
│   │   └── templates
│   └── web_ui
│       ├── monit
│       ├── spec
│       └── templates
├── packages
└── src

9 directories, 5 files
</pre>

### <a id="control"></a> Create control scripts  ###

Every job needs a way to start and stop.
You provide that by writing a control script and updating the `monit` file.

The control script:

* Includes a start command and a stop command.
* Is an ERB template stored in the `templates` directory for the relevant job.

For each job, create a control script that configures the job to store logs in `/var/vcap/sys/log/JOB_NAME`. Save this script as `ctl.erb` in the `templates` directory for its job.

The control script for the `web_ui` job looks like this:

<pre type="bash">
#!/bin/bash

RUN_DIR=/var/vcap/sys/run/web_ui
LOG_DIR=/var/vcap/sys/log/web_ui
PIDFILE=${RUN_DIR}/pid

case $1 in

  start)
    mkdir -p $RUN_DIR $LOG_DIR
    chown -R vcap:vcap $RUN_DIR $LOG_DIR

    echo $$ > $PIDFILE

    cd /var/vcap/packages/ardo_app

    export PATH=/var/vcap/packages/ruby_1.9.3/bin:$PATH

    exec /var/vcap/packages/ruby_1.9.3/bin/bundle exec \
      rackup -p <%%= properties.web_ui.port %> \
      >>  $LOG_DIR/web_ui.stdout.log \
      2>> $LOG_DIR/web_ui.stderr.log

    ;;

  stop)
    kill -9 `cat $PIDFILE`
    rm -f $PIDFILE

    ;;

  *)
    echo "Usage: ctl {start|stop}" ;;

esac
</pre>

If your release needs templates other than the control script, create them now.

### <a id="monit"></a> Update monit files  ###

The `monit` file:

* Specifies the process ID (pid) file for the job
* References each command provided by the templates for the job
* Specifies that the job belongs to the `vcap` group

On a deployed release, a BOSH Agent runs on each job VM.
BOSH communicates with the Agent, which in turn executes commands in the
control script.
The Agent does this using open source process monitoring software called
[Monit](http://mmonit.com/monit/).

The `monit` file for the `web_ui` job looks like this:

<pre type="terminal">
check process web_ui
  with pidfile /var/vcap/sys/run/web_ui/pid
  start program "/var/vcap/jobs/web_ui/bin/ctl start"
  stop program "/var/vcap/jobs/web_ui/bin/ctl stop"
  group vcap
</pre>

Update the `monit` file for each of your jobs.
Use `/var/vcap` paths as shown in the example.

<p class="note"><strong>Note</strong>: BOSH requires a <code>monit</code> file for each job in a release. When developing a release, you can use an empty <code>monit</code> file to meet this requirement without having to first create a control script.</p>

### <a id="job-specs"></a> Update job specs  ###

At compile time, BOSH transforms templates into files, which it then replicates
on the job VMs.

The template names and file paths are among the metadata for each job that
resides in the job `spec` file.

In the job `spec` file, the `templates` block contains key/value pairs where:

* Each key is template name
* Each value is the path to the corresponding file on a job VM

The file paths that you provide for templates are relative to
the `/var/vcap/jobs/<job_name>` directory on the VM.
For example, `bin/ctl` becomes `/var/vcap/jobs/<job_name>/bin/ctl` on the job VM.
Using `bin` as the directory where these files go is a convention.

The `templates` block of the updated `spec` files for the example jobs look
like this:

~~~yaml
templates:
  ctl.erb: bin/ctl
~~~

For each job, update the `spec` file with template names.

### <a id="commit-one"></a> Commit ###

You have now created one or more job skeletons; this is a good time to commit.

If you used the `--git` option with `bosh init release` (as recommended), the
correct `.gitignore` file has been automatically created for you.

---
## <a id="graph"></a> Step 2: Make Dependency Graphs ##

There are two kinds of dependencies in a BOSH release:

* The **runtime dependency**, where a job depends on a package at runtime.
For example, the `web_ui` job depends on Ruby.
* The **compile-time dependency**, where a package depends on another package at
compile time.
For example, Ruby depends on the YAML library.

Three rules govern these dependencies:

* Jobs never depend on other jobs.
* Jobs can depend on packages.
* Packages can depend on other packages.

### <a id="build-graph"> </a>Building the Dependency Graph ###

Create a dependency graph to clarify your understanding of the
dependencies between the jobs and packages in your release.

#### <a id="runtime"> </a>Identify runtime dependencies ####

Whenever a control script or other template cites a package name, the job
that the template belongs to depends on the cited package at runtime.

For each job, find all the cases where your control scripts cite packages.
Add these runtime dependencies to your dependency graph.

In our example, this line in both of our `ctl.erb` scripts cites `ardo_app`:

~~~
cd /var/vcap/packages/ardo_app
~~~

This line cites Ruby:

~~~
exec /var/vcap/packages/ruby_1.9.3/bin/bundle exec
~~~

This means that both the `web-ui` and `bg_worker` jobs have runtime
dependencies on both the `ardo_app` and `ruby_1.9.3` packages.

We add these four runtime dependencies to our example dependency graph.

#### <a id="compile-time"> </a>Identify compile-time dependencies ####

Use your knowledge about the runtime dependencies you have already noted.
Consider the packages you have identified as dependencies.
Do any of them depend on other packages in turn?

Whenever a package depends on another package, that is a compile-time
dependency.

For each job, add the compile-time dependencies to your dependency graph.
If you miss a dependency, BOSH lets you know later, when you try to deploy.

In our example, we already noted a runtime dependency on Ruby 1.9.3.
We now ask ourselves whether Ruby 1.9.3 itself has any dependencies.
The answer is yes, it depends on libyaml 0.1.4.

We add this compile-time dependency to our example dependency graph.

#### <a id="dep-graph-example"> </a>The complete example dependency graph ####

The complete dependency graph for `ardo-release` looks like this:

<%= image_tag("./images/dep-graph.png") %>

For a large or complicated release, consider making more than one dependency
graph.

## <a id="pkg-skeletons"></a> Step 3: Create Package Skeletons ##

Packages give BOSH the information needed to prepare the binaries and
dependencies for your jobs.

Create package skeletons starting from the bottom of your dependency graph.

  `bosh generate package <dependency_name>`

In our example, we run this command three times.
Starting from the bottom of the dependency graph,
we run it for `libyaml_0.1.4`, `ruby_1.9.3`, and `ardo_app`.

View the package skeletons with `tree`:

<pre class="terminal">
$ tree packages
packages
├── ardo_app
│   ├── packaging
│   ├── pre_packaging
│   └── spec
├── libyaml_0.1.4
│   ├── packaging
│   ├── pre_packaging
│   └── spec
└── ruby_1.9.3
    ├── packaging
    ├── pre_packaging
    └── spec

3 directories, 9 files
</pre>

Putting each dependency in a separate package provides maximum reusability
along with a clear, modular structure. This is not mandatory; what packages
to create is a matter of preference. You could even opt to put all the
dependencies together in a single package, though that is not recommended.

<p class="note"><strong>Note</strong>: Use of the <code>pre_packaging</code> file is not recommended, and is not discussed in this tutorial.</p>

Without using `pre_packaging` for our `ardo_app` we need to pack gems manually for further usage:

<pre class="terminal">
$ cd src/ardo_app/
$ bundle package
</pre>

### <a id="update-pkging-specs"></a> Update packaging specs ###

Within each package directory, there is a `spec` file which states:

* The package name
* The package's dependencies
* The location where BOSH can find the binaries and other files that the package needs at compile time

Use your dependency graph to determine which dependencies belong in each spec.
Developer preferences and style play a role here.
Consider our example: the spec for Ruby lists `rubygems` and `bundler` as dependencies along
with Ruby itself.
Some Ruby developers would do it this way; others would not.

To maximize portability of your release across different versions of stemcells,
never depend on the presence of libraries or other software on stemcells.

To describe binary locations in the `files` block of the spec:

* Find the official site for the binary in question.
For example, Ruby might be at `http://cache.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p484.tar.gz`.

* Download the binary from the official location and make sure the file hash matches.

* Record the binary name including version number, with a slash and the binary
filename concatenated to it.
It's a good idea to cite the official URL in a comment, in the same line.

BOSH interprets the locations you record in the `files` section as being
either in the `src` directory or in the `blobs` directory.
(BOSH looks in `src` first.)
When you add the actual blobs to a blobstore (see the next section),
BOSH populates the `blobs` directory with the correct information.

For packages that depend on their own source code, use the globbing pattern
`<package_name>/**/*` to deep-traverse the directory in `src` where
the source code should reside.

Update the spec for each package.
Refer to the example specs below for guidance.

#### <a id="pkg-spec-libyaml"></a> Example libyaml package spec ####

~~~yaml
---
name: libyaml_0.1.4

dependencies: []

files:
- libyaml_0.1.4/yaml-0.1.4.tar.gz # From http://pyyaml.org/download/libyaml/yaml-0.1.4.tar.gz
~~~

#### <a id="pkg-spec-ruby"></a> Example Ruby package spec ####

~~~yaml
---
name: ruby_1.9.3

dependencies:
- libyaml_0.1.4

files:
- ruby_1.9.3/ruby-1.9.3-p484.tar.gz # http://cache.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p484.tar.gz
- ruby_1.9.3/rubygems-1.8.24.tgz    # http://production.cf.rubygems.org/rubygems/rubygems-1.8.24.tgz
- ruby_1.9.3/bundler-1.2.1.gem      # https://rubygems.org/downloads/bundler-1.2.1.gem
~~~

#### <a id="pkg-spec-ardo"></a> Example ardo_app package spec ####

~~~yaml
---
name: ardo_app

dependencies:
- ruby_1.9.3

files:
- ardo_app/**/*
~~~

### <a id="pkg-scripts"></a> Create packaging scripts ###

At compile time, BOSH takes the source files referenced in the package specs,
 and renders them into the executable binaries and scripts that your deployed
jobs need.

You write packaging scripts to instruct BOSH how to do this.
The instructions may involve some combination of copying, compilation, and
related procedures.
For example:

* For a Ruby app like `ardo_app`, BOSH must copy source files and install Ruby
gems.

* For Ruby itself, BOSH must compile source code into a binary.

* For a Python app, BOSH must copy source files and install Python eggs.

BOSH relies on you to write packaging scripts that perform the correct operation.

Adhere to these principles when writing packaging scripts:

* Use your dependency graph to determine which dependencies belong in each
packaging script.

* Begin each script with a `set -e -x` line.
This aids debugging at compile time by causing the script to exit immediately
if a command exits with a non-zero exit code.

* Ensure that any copying, installing or compiling delivers resulting code to
 the install target directory (represented as the `BOSH_INSTALL_TARGET`
environment variable). For `make` commands, use `configure` or its equivalent
to accomplish this.

* Be aware that BOSH ensures that dependencies cited in the `dependencies`
block of package `spec` files are available to the deployed binary.
For example, in the `spec` file for the Ruby package, we cite libyaml as a
dependency.
This ensures that on the compilation VMs, the packaging script for Ruby has
access to the compiled libyaml package.

If the instructions you provide in the packaging scripts fail to deliver compiled
code to `BOSH_INSTALL_TARGET`, the job cannot function because the VM has no
way to find the code to run.
This failure scenario can happen if, for example,
you use a `make` command that delivers compiled code to some standard location
by default.
You can fix the problem by configuring `make` to compile into
`BOSH_INSTALL_TARGET`.
See how this is done in the example packaging scripts.

Like control scripts, writing packaging scripts is one of the heavier tasks
entailed in creating a release.
Write your packaging scripts now.
Refer to the examples below for guidance.

#### <a id="pkg-script-libyaml"></a> Example libyaml packaging script ####

~~~
set -e -x

tar xzf libyaml_0.1.4/yaml-0.1.4.tar.gz
pushd yaml-0.1.4
  ./configure --prefix=${BOSH_INSTALL_TARGET}

  make
  make install
popd
~~~

#### <a id="pkg-script-ruby"></a> Example Ruby packaging script ####

~~~
set -e -x

tar xzf ruby_1.9.3/ruby-1.9.3-p484.tar.gz
pushd ruby-1.9.3-p484
  ./configure \
    --prefix=${BOSH_INSTALL_TARGET} \
    --disable-install-doc \
    --with-opt-dir=/var/vcap/packages/libyaml_0.1.4

  make
  make install
popd

tar zxvf ruby_1.9.3/rubygems-1.8.24.tgz
pushd rubygems-1.8.24
  ${BOSH_INSTALL_TARGET}/bin/ruby setup.rb
popd

${BOSH_INSTALL_TARGET}/bin/gem install ruby_1.9.3/bundler-1.2.1.gem --no-ri --no-rdoc
~~~

#### <a id="pkg-script-ardo"></a> Example ardo_app packaging script ####

~~~
set -e -x

cp -a ardo_app/* ${BOSH_INSTALL_TARGET}

cd ${BOSH_INSTALL_TARGET}

/var/vcap/packages/ruby_1.9.3/bin/bundle install \
  --local \
  --deployment \
  --without development test
~~~

### <a id="update-job-specs-with-deps"></a> Update job specs with dependencies ###

The dependency graph reveals runtime dependencies that
need to be added to the `packages` block of the job spec.

Edit the job specs to include these dependencies.

In our example, the dependency graph shows that `web_ui` job depends on
`ardo_app` and `ruby_1.9.3`:

~~~yaml
packages:
- ardo_app
- ruby_1.9.3
~~~

---
## <a id="blobs"></a> Step 4: Add Blobs ##

When creating a release, you likely use a source code repository.
But releases often use tar files or other binaries, also known as blobs.
Checking blobs into a repository is problematic if your repository
unsuited to dealing with large binaries (as is true of Git, for example).

BOSH lets you avoid checking blobs into a repository by doing the following:

* For dev releases, use local copies of blobs.

* For a final release, upload blobs to a blobstore, and direct BOSH to obtain the blobs from there.

### <a id="config-blobstore"></a> Configure a blobstore  ###

In the `config` directory, you record the information BOSH needs about the
blobstore:

* The `final.yml` file names the blobstore and declares its type, which is either `local`
or one of several other types that specify blobstore providers.

* The `private.yml` file specifies the blobstore path, along with a secret.

`private.yml` contains keys for accessing the blobstore and should not be
checked into a repository.
(If you used the `--git` option when running `bosh init release` at the beginning
of this tutorial, `private.yml` is automatically `gitignored`.)

The `config` directory also contains two files whose content is automatically
generated: the `blobs.yml` file and the `dev.yml` file.

Adapt the examples below to fit the specifics of your release.
Our example release uses the `local` type blobstore because otherwise it would
be necessary to explain how to configure a public blobstore such as
Amazon S3, which is too large a topic for this context.

The `local` type blobstore is suitable for learning but the resulting release
cannot be shared.
For that reason, you should configure a non-local, publicly available blobstore
for releases that you intend to share.
Normally, the blobstore you choose when you begin working on a release is used
for all subsequent versions of the release.
Changing the blobstore that a release uses is beyond the scope of this tutorial.

Example `final.yml`:

~~~yaml
---
blobstore:
  provider: local
  options:
    blobstore_path: /tmp/ardo-blobs
final_name: ardo_app
~~~

Example `private.yml`:

~~~yaml
---
blobstore_secret: 'does-not-matter'
blobstore:
  local:
    blobstore_path: /tmp/ardo-blobs

~~~

If you have a `private.yml` file:

* **Required**: Include the `blobstore_path` in the `private.yml` file.
* **Optional**: Include the `blobstore_path` in the `final.yml` file. Doing so allows you to `gitignore` the `private.yml` file but still allow the release to be downloaded and used on other systems.

<p class="note"><strong>Note</strong>: The <code>blobstore_secret</code> is required for the <code>local</code> type blobstore.
This is true even though the <code>blobstore_secret</code> line is deprecated and its
content does not matter.
There is never <code>blobstore_secret</code> line for blobstores of types other than
<code>local</code>.</p>

### <a id="inform"></a> Inform BOSH where blobs are ###

In the package `spec` file, the `files` block lists any binaries you downloaded,
along with the URLs from which you downloaded them.
(This assumes that you followed the directions in the [Update package specs](#update-pkging-specs) section.)

Those files are blobs, and now you need the paths to the downloaded blobs on
your local system.

In our example, the `spec` file for the `libyaml_0.1.4` package includes the line:

~~~yaml
files:
- libyaml_0.1.4/yaml-0.1.4.tar.gz # From http://pyyaml.org/download/libyaml/yaml-0.1.4.tar.gz
~~~

If you downloaded the blob, its local path might be:

`~/Downloads/yaml-0.1.4.tar.gz`

Go through all your packages and make a list of local paths to the blobs you downloaded.
Now you are ready to inform BOSH about these blobs.

For each blob, run:

`bosh add blob <path_to_blob_on_local_system> <package_name>`

e.g.

`bosh add blob ~/Downloads/yaml-0.1.4.tar.gz libyaml_0.1.4`

The `bosh add blob` command adds a local blob to the collection your release
recognizes as BOSH blobs.

The usage shown above is a blend of requirement and convention.
It works like this:

* For the first argument, you provide the path to the blob on your local system
* For the second argument, you provide a destination within the `blobs` directory
in your release
* BOSH goes into the `blobs` directory and creates a subdirectory with
the name of the package that the local blob represents
* In the new subdirectory, BOSH creates a symbolic link to a copy of the blob
which BOSH makes in a hidden directory

Using the package name as the second argument of the `bosh add blobs` command
is recommended because it produces a cleanly-organized blobs directory.

Later, when you upload blobs for a final release, BOSH uses the hidden directory
as a staging area.

### <a id="no-upload"></a> Do not upload blobs for a dev release ###

Once you have uploaded blobs to a non-local blobstore, those blobs may become
essential to some other developer.
For this reason, uploading a blob and then removing it is considered poor practice.

When creating dev releases, do not run `bosh upload blobs`.
(You only run it when you do a final release.)

---
## <a id="create-props"></a> Step 5: Create Job Properties  ##

If your service needs to be configurable at deployment time,
you create the desired inputs or controls and specify them in the release.
Each input is a _property_ that belongs to a particular job.

Creating properties requires three steps:

1. Define properties and defaults in the `properties` block of
the job spec.

1. Use the property lookup helper `p()` to reference properties in
relevant templates.

For example, a start command can take a property as an argument,
using the property lookup helper:

       <%%= p('<job_name>.<property_name>') %>

1. Specify the property in the [deployment manifest](./deployment-manifest.html#properties).

Adapt the example below to create any properties your release needs now.

In our example, we want the port that the web UI listens on to be a
configurable property.

We edit the spec for the web UI job to look like this:

~~~yaml
properties:
   web_ui.port:
     description: Port that web_ui app listens on
     default: 80
~~~

---
## <a id="dev-release"></a> Step 6: Create a Dev Release  ##

All the elements needed to create a dev release should now be in place.

### <a id="dev-release-release"></a> Release  ###

For the dev release, use the `--force` option with the `bosh create release`
command.
This forces BOSH to use the local copies of our blobs.

Without the `--force` option, BOSH requires blobs to be uploaded before you
run `bosh create release`.
For a final release, we upload blobs, but not for a dev release.

Create the dev release:

`bosh create release --force`

BOSH prompts for a release name, and assigns a dot-number version to the release.

### <a id="dev-release-deploy"></a> Deploy the Dev Release  ###

Deploying the release requires three or more steps, depending on whether
BOSH is targeting the desired Director, and whether BOSH is already pointing
to a release.

See what director BOSH is targeting:

  `bosh target`

Target a director:

  `bosh target <director_url>`

See what releases are available:

  `bosh releases`

If BOSH is already pointing to a release, edit the BOSH deployment manifest.
Otherwise, create a manifest. See [BOSH Deployment Manifest](./deployment-manifest.html) for more information.
Simple manifest for `ardo_app` can be found [here](https://gist.github.com/antonsoroko/3be4c70b38f846b1d79eca7192a5ab58) (OpenStack) or [here](https://gist.github.com/uzzz/9ad9cad105032fecdbeb223798607a87) (AWS).

Upload the new dev release.

   `bosh upload release`

Assuming you are in the release directory, no path is needed with the above command.

Deploy:

   `bosh deploy`

See [Deploying Distributed Software with BOSH](./deploy-with-bosh.html) for more
about deployment.

### <a id="dev-release-test"></a> Test the Dev Release  ###

What tests to run depends on the software you are releasing.

Start by opening a separate terminal, logging in on the job VM, and observing
logging output as you test your release.

If your release fails tests, follow this pattern.

* Fix the code.
* Do a new dev release.
* Run `bosh deploy` to see whether the new release deploys successfully.

Using `bosh deploy --recreate` can provide a clearer picture because with that option,
BOSH deploys all the VMs from scratch.

---
## <a id="final-release"></a> Create a Final Release  ##

Only proceed to this step if your latest dev release passes all tests.

### <a id="upload-blobs"></a> Upload blobs ###

When you use the `bosh create release --force` command to create them, dev
releases depend on locally-stored blobs.
To do a final release, you must upload blobs first.

If files that you need to keep private are uploaded to a public blobstore,
there is no satisfactory way to undo the mistake.
To avoid this situation, complete the following steps immediately before
you upload blobs:

1. Run `bosh blobs` to see the list of blobs BOSH is prepared to upload

1. Proofread the list of blobs displayed by the command

1. The list should include only the blobs you need for the final release

1. If the list includes any files that should not be uploaded, find and delete
the symbolic links to them in the `blobs` directory

To upload your blobs, run:

  `bosh upload blobs`

### <a id="final-commit-two"></a> Commit ###

The `bosh upload blobs` command has now populated the `blobs.yml` file
in the `config` directory with metadata for uploaded blobs.

This is a good reason to commit.

### <a id="final-release-release"></a> Release ###

Run:

  `bosh create release --final`

BOSH prompts you for a release name, and assigns a whole-number version to the release.

This is a good time to push your code to a shared repository to give others access to
your final release.

### <a id="final-release-commit"></a> Commit ###

Do one more commit before you deploy!

### <a id="final-release-deploy"></a> Deploy the Final Release  ###

Run:

`bosh deploy`