e2e testing

End to end tests are tests that simulate a user going through your application or site. This allows high level tests which closely reflect the expected experience a real user will have.

However because of the complexity and time involved in running these tests, e2e tests should not be expected to make unit tests obsolete. E2e tests should serve as a final guard making sure a few pieces of key functionality are always working in the system when all the parts are working together like on the production server.

The testing team at Google has written a good, although somewhat harsh article on the often broken promises of e2e testing. Their conclusion is that e2e tests should make up around 10% of your total tests.

Running tests locally

When writing code and therefore breaking tests, it's useful (necessary) to be able to run the tests locally. For this purpose a shell script (bash) was created. Opening a terminal in your product root folder and executing bash tests/run-e2e.sh performs the same steps as what happens on travis. When running this, make sure you are able to connect to the integration database as defined in db.ini. What actually happens in this script and on travis in general is explained below.

Running tests

The environment requirements to execute a test are pretty simple: The tests require CasperJS and PhantomJS. CasperJS can be installed using npm i -g casperjs. Locally you can just run this in any old terminal window. For Travis CI it's best to leave this in a before_install block in your travis.yml config. NPM and PhantomJS should already be installed on Travis CI. If PhantomJS is not yet available it will be installed as a dependency of CasperJS.

Executing the tests is done by calling casperjs test tests/e2e/, either from a terminal in the root of your app or in the script block in your Travis CI config. This will execute all the scripts in the /tests/e2e folder of your app.

Running an app to test

Most of the time you will want to test the code that was added most recently. This means you will need to run a full version of the app to test based on the latest commit. For garp projects this comes down to initializing the database, reading in snippets, compiling assets and booting a web server. Luckily this can all be done in your Travis CI configuration.

As a database we choose to use the integration version of the database. The connection info is simply entered in the integration section of garp's db.ini This database should ideally be set up to contain data that closely resembles the actual data on the production data. A possible strategy would be having a cronjob keeping the integration database synced with the production database. For now just sync the data between production and integration manually, using Navicat or something. Because we do want to test the most recent version of the code, including possible changes to the data model, a garp spawn should always be run before running e2e test s.

Example Travis CI before_script:

# Initialize database (not required when using an external db)
- mysql -e "create database IF NOT EXISTS schoolwijzer_i;" -uroot
- mysql -e "create user 'garp-integration'@'localhost' identified by 'NEUNvBUjA4tdVCZfh$RJGG:=BXvmRatsPyyF-G1P;jjZTQYpWD';" -uroot
- mysql -e "grant all privileges on *.* to 'garp-integration'@'localhost';" -uroot

# Update data model and snippets
- php garp/scripts/garp.php spawn -b --e=integration
- php garp/scripts/garp.php snippet create --e=integration

# Boot php server at 127.0.0.1:8027 and the correct ENV
- APPLICATION_ENV=integration php -S 127.0.0.1:8027 -t public/ > /dev/null 2>&1 &

Please note that using php's built in web server requires a version of php 5.4 or later. You can define this in the Travis CI config. Also make sure the base href and asset urls for the integration configs are pointing to the local versions at the same address as where you run the web server (e.g. 127.0.0.1:8027).

Writing tests:

Although many high level testing languages and frameworks exist, CasperJS should be enough for now. It allows you to control a headless browser directly and intuitively and define tests using a handful of assert functions. Their documentation is also pretty decent.

There are a couple of caveats to take into account when writing e2e tests:

Watch out for single pagey stuff. Make sure to wait for the creation and appearance of the expected dom elements before asserting anything on them. CasperJS can wait automatically for classic browser page requests but by itself knows little of asynchronous data being filled in by javascript. You'll have to manually define what you are waiting for using CasperJS's waitForX functions.

Watch out for media queries. By default CasperJS sets the viewport to 400x300, this means a mobile version for most sites. When testing visual aspects this would mean mobile styling. With Angular or other javascript involved, this might even result in different DOM from the desktop version. The viewport can be defined easily using CasperJS:

casper.options.viewportSize = {
  width: 1024,
  height: 600
};

A challenge when writing tests is to avoid strong coupling between your tests and the technical implementation in your app. Because of the time involved in writing and fixing tests, tests should be made as general as possible. You should not have to rewrite the tests every time you rewrite the code of the app. Ideally your e2e tests are so general they last for years. Testing details which are closely tied to the technical approach taken within the code should be left to unit tests.