Skip to content

Commit dbcb8e7

Browse files
marcparadisemarcparadise
committed
Add Code of Conduct doc. Add contrib doc. MOve old contrib to building.md
Signed-off-by: Marc A. Paradise <[email protected]>
1 parent 58e7091 commit dbcb8e7

File tree

3 files changed

+291
-287
lines changed

3 files changed

+291
-287
lines changed

‎BUILDING.md

+288
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,288 @@
1+
# Building the ChefDK
2+
3+
To build the chef-dk, we use the omnibus system. Go to the
4+
[omnibus README](omnibus/README.md) to find out how to build!
5+
6+
# Installation and Testing
7+
8+
A standard `bundle install` in the root directory and in the `omnibus/`
9+
directory will get you started. You can run unit tests via:
10+
11+
## Running `chef` Locally
12+
13+
```
14+
bundle exec chef
15+
```
16+
17+
## Unit Testing
18+
19+
```
20+
bundle exec rspec spec/
21+
```
22+
23+
## Chef Verify
24+
25+
You can run the [chef verify command](README.md#chef-verify) on your local dev
26+
environment if you have the ChefDK binary installed. Note that you will likely
27+
need the current version of ChefDK, not the last released version. You can get
28+
the current version [here](http://artifactory.chef.co/simple/omnibus-current-local/com/getchef/chefdk/).
29+
30+
Once installed, you can run the `chef verify` command by pointing at the location
31+
of the root the latest current version on your local disk by running:
32+
33+
```
34+
bin/chef verify git --omnibus-dir <path_to_root_of_current_version>
35+
```
36+
37+
If you used the defaults while installing, `<path_to_root_of_current_version>` is
38+
simply `/opt/chefdk`.
39+
40+
# Updating Dependencies
41+
42+
If you want to change our constraints (change which packages and
43+
versions we accept in the chef-dk), there are several places to do so:
44+
45+
* To add or remove a package from the chef-dk, or update its version,
46+
edit [Gemfile](Gemfile).
47+
* To change the version of binary packages, edit
48+
[omnibus_overrides.rb](omnibus_overrides.rb).
49+
* To add new packages to the chef-dk, edit
50+
[omnibus/config/projects/chefdk.rb](omnibus/config/projects/chefdk.rb).
51+
52+
Once you've made any changes you want, you have to update the lockfiles
53+
that actually drive the build:
54+
55+
* To add any new gems, and update the chef-dk's dependencies to the very latest versions
56+
available, run `rake dependencies:force_update`.
57+
* To update the chef-dk's dependencies (changing as
58+
little as possible), run `rake dependencies:update`.
59+
the rake task handles both the windows and non-windows lockfiles and
60+
updates them in sync.
61+
* To update the gems used by omnibus (generally when changing the
62+
definitions in `omnibus-software`), run `rake dependencies:update_omnibus_gemfile_lock`.
63+
64+
# How the ChefDK Builds and Versions
65+
66+
The ChefDK is an amalgam of many components. These components update all
67+
the time, necessitating new builds. This is an overview of the process
68+
of versioning, building and releasing the ChefDK.
69+
70+
## ChefDK Packages
71+
72+
The ChefDK is distributed as packages for Debian, RHEL, Ubuntu, Windows
73+
and macOS. It includes a large number of components from various sources,
74+
and these are versioned and maintained separately from the chef-dk
75+
project, which bundles them all together conveniently for the user.
76+
77+
These packages go through several milestones:
78+
- `master`: When code is checked in to master, the patch version of
79+
chef-dk is bumped (e.g. 0.9.10 -> 0.9.11) and a build is kicked off
80+
automatically to create and test the packages in Chef's Jenkins cluster.
81+
- `unstable`: When a package is built, it enters the unstable channel.
82+
When all packages for all OS's have successfully built, the test phase
83+
is kicked off in Jenkins across all supported OS.
84+
- `current`: If the packages pass all the tests on all supported OS's,
85+
it is promoted as a unit to `current`, and is available to install
86+
from the current channel on https://downloads.chef.io/ or by using
87+
`mixlib-install`.
88+
- `stable`: Once a month, Chef will mark a release as stable.
89+
When this happens, it is manually promoted to stable
90+
and an announcement is sent to the list. It can be reached at
91+
https://downloads.chef.io or installed using `mixlib-install`
92+
93+
## Continuous Integration
94+
95+
Whenever a change is merged to either the `master` or `chefdk-1`
96+
branches, the `expeditor` bot updates the changelog, and increments the
97+
patch version of the branch. Expeditor is [documented thoroughly](http://expeditor-docs.es.chef.io/),
98+
and is used by many Chef projects.
99+
100+
Expeditor is controlled by a number of Pull Request labels:
101+
* `Expeditor: Exclude From Changelog`: This PR is not a software change
102+
and should not be included in the ChangeLog
103+
* `Expeditor: Skip Build`: This PR doesn't warrant a new build of
104+
ChefDK.
105+
* `Expeditor: Skip Version Bump`: This PR doesn't change any behaviour
106+
of ChefDK, and so does not warrant a new patch version. Typically,
107+
skipping a version bump will also imply one should skip running a build, too.
108+
* `Expeditor: Skip All`: All of the above.
109+
110+
## Component Versions
111+
112+
The chef-dk has two sorts of component: ruby components like `berkshelf`
113+
and `test-kitchen`, and binary components like `openssl` and even `ruby`
114+
itself.
115+
116+
In general, you can find all chef-dk desired versions in the
117+
[Gemfile](Gemfile) and [omnibus_overrides.rb](omnibus_overrides.rb) files. The
118+
[Gemfile.lock](Gemfile.lock) is the locked version of the Gemfile.
119+
omnibus. [build](omnibus/Gemfile) and [test](acceptance/Gemfile)
120+
Gemfiles and [Berksfile](omnibus/Berksfile) version the toolset we use
121+
to build and test.
122+
123+
### Binary Components
124+
125+
The versions of binary components (as well as rubygems and bundler,
126+
which can't be versioned in a Gemfile) are stored in
127+
[omnibus_overrides](omnibus_overrides.rb). These are updated as
128+
required.
129+
130+
These have software definitions either in
131+
[omnibus/config/software](omnibus/config/software) or, more often, in
132+
the
133+
[omnibus-software](https://github.com/chef/omnibus-software/tree/master/config/software)
134+
project.
135+
136+
### Rubygems Components
137+
138+
Most of the actual front-facing software in the chef-dk is composed of
139+
ruby projects. berkshelf, test-kitchen and even chef itself are made of
140+
ruby gems. Chef uses the typical ruby way of controlling rubygems
141+
versions, the `Gemfile`. Specifically, the `Gemfile` at the top of the
142+
chef-dk repository governs the version of every single gem we install
143+
into the chef-dk package. It's a one-stop shop.
144+
145+
See [Updating Dependencies](#updating-dependencies) for further
146+
information.
147+
148+
### Build Tooling Versions
149+
150+
Of special mention is the software we use to build omnibus itself. There
151+
are two distinct bits of code that control the versions of compilers,
152+
make, git, and other tools we use to build.
153+
154+
First, the Jenkins machines that run the build are configured entirely
155+
by the [opscode-ci cookbook](https://github.com/chef-cookbooks/opscode-ci)
156+
cookbook. They install most of the tools we use via `build-essentials`,
157+
and standardize the build environment so we can tear down and bring up
158+
builders at will. These machines are kept alive long-running, are
159+
periodically updated by Chef to the latest opscode-ci, omnibus and
160+
build-essentials cookbooks.
161+
162+
Second, the version of omnibus we use to build the chef-dk is governed
163+
by `omnibus/Gemfile`. When software definitions or the omnibus framework
164+
is updated, this is the file that drives whether we pick it up.
165+
166+
The omnibus tooling versions are locked down with
167+
`omnibus/Gemfile.lock`, and can be updated by running `rake
168+
dependencies`.
169+
170+
### Test Versions
171+
172+
chef-dk is tested by the [chef-acceptance framework](https://github.com/chef/chef-acceptance),
173+
which contains suites that are run on the Jenkins test machines. The
174+
definitions of the tests are in the `acceptance` directory. The version
175+
of chef-acceptance and test-kitchen, are governed by
176+
`acceptance/Gemfile`.
177+
178+
The test tooling versions are locked down with
179+
`acceptance/Gemfile.lock`, which can be updated by running `rake
180+
dependencies`.
181+
182+
## The Build Process
183+
184+
The actual ChefDK build process is done with Omnibus, and has several
185+
general steps:
186+
187+
1. `bundle install` from `chef-dk/Gemfile.lock`
188+
2. Reinstall any gems that came from git or path using `rake install`
189+
3. appbundle chef, chef-dk, test-kitchen and berkshelf
190+
4. Put miscellaneous powershell scripts and cleanup
191+
192+
### Kicking Off The Build
193+
194+
The build is kicked off in Jenkins by running this on the machine (which
195+
is already the correct OS and already has the correct dependencies,
196+
loaded by the `omnibus` cookbook):
197+
198+
```
199+
load-omnibus-toolchain.bat
200+
cd chef-dk/omnibus
201+
bundle install
202+
bundle exec omnibus build chefdk
203+
```
204+
205+
This causes the [chefdk project definition](omnibus/config/projects/chefdk.rb)
206+
to load, which runs the
207+
[chef-dk-complete](omnibus/config/software/chef-dk-complete.rb) software
208+
definition, the primary software definition driving the whole build
209+
process. The reason we embed it all in a software definiton instead of
210+
the project is to take advantage of omnibus caching: omnibus will
211+
invalidate the entire project (and recompile ruby, openssl, and
212+
everything else) if you change anything at all in the project file. Not
213+
so with a software definition.
214+
215+
### Installing the Gems
216+
217+
The primary build definition that installs the many ChefDK rubygems is
218+
[`software/chef-dk.rb`](omnibus/software/chef-dk.rb). This has
219+
dependencies on any binary libraries, ruby, rubygems and bundler. It has
220+
a lot of steps, so it uses a
221+
[library](omnibus/files/chef-dk/build-chef-dk.rb) to help reuse code and
222+
make it manageable to look at.
223+
224+
What it does:
225+
226+
1. Depends on software defs for pre-cached gems (see "Gems and Caching"
227+
below).
228+
2. Installs all gems from the bundle:
229+
- Sets up a `.bundle/config`
230+
([code](omnibus/files/chef-dk/build-chef-dk.rb#L17-L39)) with
231+
--retries=4, --jobs=1, --without=development,no_<platform>, and
232+
`build.config.nokogiri` to pass.
233+
- Sets up a common environment, standardizing the compilers and flags
234+
we use, in [`env`](omnibus/files/chef-dk-gem/build-chef-dk-gem.rb#L32-L54).
235+
- [Runs](omnibus/config/software/chef-dk.rb#L68) `bundle install --verbose`
236+
3. Reinstalls any gems that were installed via path:
237+
- [Runs](omnibus/files/chef-dk/build-chef-dk.rb#L80) `bundle list --paths`
238+
to get the installed directories of all gems.
239+
- For each gem not installed in the main gem dir,
240+
[runs](omnibus/files/chef-dk/build-chef-dk.rb#L89) `rake install`
241+
from the installed gem directory.
242+
- [Deletes](omnibus/files/chef-dk/build-chef-dk.rb#L139-L143) the
243+
bundler git cache and path- and git-installed gems from the build.
244+
4. [Creates](omnibus/files/chef-dk/build-chef-dk.rb#L102-L152)
245+
`/opt/chefdk/Gemfile` and `/opt/chefdk/Gemfile.lock` with the gems that
246+
were installed in the build.
247+
248+
#### Gems and Caching
249+
250+
Some gems take a super long time to install (particularly
251+
native-compiled ones such as nokogiri and dep-selector-libgecode) and do
252+
not change version very often. In order to avoid doing this work every
253+
time, we take advantage of omnibus caching by separating out these gems
254+
into their own software definitions.
255+
[chef-dk-gem-dep-selector-libgecode](omnibus/config/software/chef-dk-gem-dep-selector-libgecode.rb)
256+
for example.
257+
258+
Each of these gems uses the `config/files/chef-dk-gem/build-chef-dk-gem`
259+
library to define itself. The name of the software definition itself
260+
indicates the .
261+
262+
We only create software definitions for long-running gems. Everything
263+
else is just installed in the
264+
[chef-dk](omnibus/config/software/chef-dk.rb) software definition in a
265+
big `bundle install` catchall.
266+
267+
Most gems we just install in the single `chef-dk` software definition.
268+
269+
The first thing
270+
271+
### Appbundling
272+
273+
After the gems are installed, we *appbundle* them in
274+
[chef-dk-appbundle](omnibus/config/software/chef-dk-appbundle.rb). This
275+
creates binstubs that use the bundle to pin the software .
276+
277+
During the process of appbundling, we update the gem's `Gemfile` to
278+
include the locks in the top level `/opt/chefdk/Gemfile.lock`, so we can
279+
guarantee they will never pick up things outside the build. We then run
280+
`bundle lock` to update the gem's `Gemfile.lock`, and `bundle check` to
281+
ensure all the gems are actually installed. The appbundler then uses
282+
these pins.
283+
284+
### Other Cleanup
285+
286+
Finally, the chef-dk does several more steps including installing
287+
powershell scripts and shortcuts, and removing extra documentation to
288+
keep the build slim.

‎CODE_OF_CONDUCT.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Please refer to the Chef Community Code of Conduct at https://www.chef.io/code-of-conduct/

0 commit comments

Comments
 (0)