Testing

awless has been from the start a Test Driven Development project.

It has allowed the team to move quickly (multiple refactoring, re-design, code reduction, etc.) and safely, ensuring user features could be implemented while keeping the complete integrity of the existing product.

Basically, we do not want to break the user!

It is paramount to ensure that new written code validates multiple layer of tests. All those test should be ran before a release and a failure is a show stopper.

Below is a taxonomy of the test suite for awless and how to run the different tests.

Unit/Integration/Acceptance tests

Also it mixes unit/integration/acceptance tests, the are run in isolation with the simple go test contract. This allow to get rapid feedback while having a comprehensive test suite (even more rapid with the test caching introduced in Golang 1.10)

$ cd awless
$ go test ./... -race 

Note that -race flag is not mandatory on each run but it is important before release of pushed to the master (-race will instrument your code to find concurrency flaws, so it takes more time to run).

Parsing and compiling repo of real templates

The awless-templates project is a repository of standard templates examples.

Cloning the repo and running its tests will exercise your current version of awless against all the repo's templates. This makes it a very good test suite to check if you have not broken anything in term of parsing, and running templates.

You need to have the awless project locally up to date. Then run it with:

$ go get -u github.com/wallix/awless   # if necessary
$ git clone [email protected]:wallix/awless-templates.git
$ cd awless-templates
$ go test

The output will be verbose enough for you to understand what has been achieved.

Real life tests (costs money!)

Since those tests are run against the cloud hence are charged by AWS, the core team usually will run those tests against new contributions/PR.

Smoke test

This test compile awless and run it against the cloud doing happy path scenarios and finishing by asserting that everything went well. This is a test you want to run before pushing a feature or a release.

The test lasts ~2 mins, run it with:

$ ./smoke_tests/smoke_test.sh

Mastodonte test

This test will run almost all possible awless one-liners command against a already prepared cloud region.

The test last ~30 mins, run it with:

$ ./smoke_tests/test-all-drivers.sh