Continuous Integration for Docker & Vagrant builds

[pitkley]

Given both the rapid growth of this project and general convenience, I think a Continuous Integration solution is in order. It would help everyone by:

1. showing the end-user in the README if the current master builds (which it should!),
2. checking every PR if it builds correctly, and informing the requester if it didn't,
3. not allowing a merge of a failed PR (configurable).

Further this should take at least some burden off of @danielquinn's shoulders since a baseline check will already have been performed on the submitted changes.

From what I've seen, Travis CI seems to be the de facto standard on GitHub for automatically building open-source projects for free. There are some alternatives (like drone.io) but they usually don't allow us to install our requirements (like ImageMagick) or have other limitations. While a system like Jenkins could definitely cover all use cases, and I use it personally with great success, it requires a dedicated system to run on.

Travis CI is controlled by a .travis.yml file which has to be located in the root of the repository. This file is a specification for Travis on what steps to perform and what commands to execute in order to build and test an application. The build-environment can be a full VM based on Ubuntu 12.04 or Ubuntu 14.04 or it can be container-based.

I have created a simple .travis.yml to test if and how it works, see feature/travis-ci. I have managed to build Paperless successfully. This build was based on my Docker implementation feature/dockerfile and tested both the building of the Docker image and running of the tests under both Python 3.4 and 3.5.

Building and testing the application directly in the Travis VM without using Docker is of course possible, and Travis allows to test multiple Python-versions automatically. But (a) from what I've seen combining the Travis Python tests with the Docker tests is a hassle and (b) the environment we build up within the Docker container is identical to the one we would build up in the VM.

Ignoring the Dockerfile altogether is of course an option. While this would simplify installing and testing directly in the Travis VM, it would not guarantee that a pull request doesn't break the Dockerfile, which in my eyes should be covered by CI! (Even testing Vagrant seems to be an option.)

---

Summarized, those seem to be the points for discussion:

• Should a CI system be introduced?
• What exactly are the requirements for the CI?
• Is Travis CI a decent choice? Are there better alternatives?
• Do we want to cover the Dockerfile and Vagrantfile?

[danielquinn]

Should a CI system be introduced?

Absolutely. I was looking into Travis CI just this morning actually.

What exactly are the requirements for the CI?

I'd like it to (at least) run a complete unit test suite, but testing the Dockerfile would also be nice. So long as I can develop and run my unit tests without having to use Docker, I'm cool with it.

Is Travis CI a decent choice? Are there better alternatives?

I've used Travis on other projects and found the GitHub integration to be excellent. I can't speak for alternatives though.

Do we want to cover the Dockerfile and the Vagrantfile?

The stickler in me says that if we're going to develop something and document it, it should be tested in the CI. I say this however knowing full well that there are currently exactly two tests for the whole application, so obviously there's a lot of work to be done here.

To be honest, I don't have a lot of experience writing tests for a Django app (yes, I'm a bad, bad man). I've written lots of tests for Python projects, but for higher-level stuff using a database and externalities like Tesseract and ImageMagick, I'm somewhat sketchy on what can be done for appropriate test coverage. If you, or anyone else have advice, examples, or even pull requests to fill this gap, it would be appreciated.

I'll leave this issue open for a few days to see if anyone else wants to offer insight. If I don't hear anything, I'll setup Travis CI and tell it to execute ./manage.py test.

[pitkley]

[...] testing the Dockerfile would also be nice. So long as I can develop and run my unit tests without having to use Docker, I'm cool with it.

This is definitely an important point: We should add Docker as another way to do things, but not to replace the existing ways and local workflows.

The stickler in me says that if we're going to develop something and document it, it should be tested in the CI. I say this however knowing full well that there are currently exactly two tests for the whole application, so obviously there's a lot of work to be done here.

This is definitely a point to expand on, but introducing CI which not only covers the unittests themselves, but also the two additional runtime-environments we support seems like a good way to move forward.

I'll leave this issue open for a few days to see if anyone else wants to offer insight. If I don't hear anything, I'll setup Travis CI and tell it to execute ./manage.py test.

Sounds reasonable. When you have added and configured Travis, I'll update the Docker PR #39 to update the .travis.yml file similar to how I have already tested it.

[danielquinn]

I know it's late, but I just added a .travis.yml file and setup Travis CI as a webhook. I had to fiddle with the tox configuration a bit because of where I've placed directories etc, but I think it's ok. If you have a cleaner/better way, feel free to issue a PR.

[stgarf]

Looks like #509 may at least satisfy the Docker part of this three year old issue once it is merged.