Testing and automated releases

[smebberson]

At present releases work as follows:

• Develop a fix/feature.
• Test it locally.
• Update image README, VERSIONS and project README.
• Version bump that image specifically.
• Tag the Git repository with the image name and version (i.e. alpine-redis-v1.0.0).
• Manually build the latest tag on Docker Hub.
• Manually create and build the 1.0.0 (or whatever release it is) tag on Docker Hub.

Note: I've turned of "automatic builds when a push happens" in Docker Hub. This is because when I'm pushing for say the alpine-redis image, the alpine-nodejs image also builds. This unnecessarily builds containers I don't intend on getting built and it also updates the timestamp on the :latest tag for other builds which is really frustrating.

Another note: because of the close inheritance of some of the images (which I think is a good thing), you then need to go through and test, bump version numbers, tag and release on Docker Hub downstream containers to take advantage of any updates. If you update alpine-base it's a big job! I haven't been doing this in one commit either so that I can keep the code in master as close as possible to the :latest release of any container. This makes things time consuming and hard to pull in great PRs such as #29.

It's not the greatest process at all.

I've thought of creating a build server and using Docker Hubs web hooks to automate the process a little more, but I just haven't had the time to implement anything and work through those issues above.

Also, I haven't seen much in the way of how to test and verify the state of a Docker image. It would be great to have them running through Travis CI. Especially when updating alpine-base for example, you could see at an instance what you've broken down-stream.

I'd like to create a discussion around this to see if we can streamline these processes a little.

[smebberson]

I've found some good leads for this:

• https://github.com/aelsabbahy/goss (my favourite so far) (related: http://www.unixdaemon.net/tools/testing-with-goss/)
• https://github.com/sstephenson/bats (my next favourite)
• http://serverspec.org/ (most complete, but my least favourite)

@matthewvalimaki, do you have any other sources of information or potential routes to achieve this?

[matthewvalimaki]

@smebberson the first one looks awesome. I wonder though if we could use http://doc.gitlab.com/ce/ci/docker/using_docker_images.html with our own scripts? On that page I see bundle exec rake spec but no code. I think it's just a shell script returning proper exit code.

On that same note I wonder if we could use Gitlab runners feature to build & push more selectively than what hooks allow? For example query Docker Hub for existing releases and only build if tags/versions have changed in Github?

I have not used any of these features but they could provide a way to avoid any custom self hosted solutions.