Automated Testing MVP

There's now automated testing for every variant of every image.
Each image has its own test file, allowing for a high level of granularity (tests are very bare-bones right now).
A high-level explanation of the testing setup has been added to the README.

.circleci/config.yml

@@ -162,9 +162,8 @@ jobs:
             fi
             export NEW_ORG=${NEW_ORG:-$COMPUTED_ORG}
 
-            export GOSS_FILES_PATH=~/circleci-bundles/shared/goss
+            export GOSS_FILES_PATH=~/circleci-bundles/$PLATFORM
             export GOSS_FILES_STRATEGY=cp
-            export GOSS_OPTS="--color --format documentation --retry-timeout 120s"
             export GOSS_SLEEP=5s
             make -j $PLATFORM/publish_images
 

README.md

@@ -124,6 +124,22 @@ Don't do it.
 If you have an Ultrabook laptop, it won't be happy.
 If you really want to do it, look at the `Makefile`.
 
+### Testing
+There is automated testing for every variant of every image!
+
+Tests run with [dgoss](https://github.com/aelsabbahy/goss/tree/master/extras/dgoss).
+
+Tests are platform-specific (e.g., PHP and Android have their own distinct sets of tests)—see the `goss.yaml` file in each image directory.
+
+The testing logic is currently in `shared/images/build.sh`, as it is nestled between the existing automated `docker build` and `docker push` functionality. In short, for a particular variant of an image, we make a copy of its Dockerfile, append some Dockerfile syntax to add a custom entrypoint that allows us to execute tests against the running container, build a special, temporary version of the image (added time is insignificant, as most of the Dockerfile steps are cached), and run the tests.
+
+A particular variant is pushed to Docker Hub only if tests pass; if not, the same process restarts for the next variant, etc.
+
+#### Remaining work
+
+- Tests are very bare-bones right now and could use image-specific additions
+- It is possible to convert `goss` test output to JUnit XML—it would be great to aggregate all this output and store it via CircleCI's Test Summary section
+- The testing code is spread across the repository and is a bit confusing; some refactoring would help
 
 ## Limitations
 * The template language is WIP - it only supports `{{BASE_IMAGE}}` template.  We should extend this.

buildpack-deps/goss.yaml

@@ -0,0 +1,13 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true
+
+  # ensure basic directory structure created
+  /home/circleci:
+    exists: true
+
+user:
+  # ensure circleci user exists
+  circleci:
+    exists: true

clojure/goss.yaml

@@ -0,0 +1,13 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true
+
+  # ensure basic directory structure created
+  /home/circleci:
+    exists: true
+
+user:
+  # ensure circleci user exists
+  circleci:
+    exists: true

dynamodb/goss.yaml

@@ -0,0 +1,4 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true

elixir/goss.yaml

@@ -0,0 +1,13 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true
+
+  # ensure basic directory structure created
+  /home/circleci:
+    exists: true
+
+user:
+  # ensure circleci user exists
+  circleci:
+    exists: true

golang/goss.yaml

@@ -0,0 +1,13 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true
+
+  # ensure basic directory structure created
+  /home/circleci:
+    exists: true
+
+user:
+  # ensure circleci user exists
+  circleci:
+    exists: true

jruby/goss.yaml

@@ -0,0 +1,13 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true
+
+  # ensure basic directory structure created
+  /home/circleci:
+    exists: true
+
+user:
+  # ensure circleci user exists
+  circleci:
+    exists: true

mariadb/goss.yaml

@@ -0,0 +1,4 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true

mongo/goss.yaml

@@ -0,0 +1,4 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true

mysql/goss.yaml

@@ -0,0 +1,4 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true

node/goss.yaml

@@ -0,0 +1,13 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true
+
+  # ensure basic directory structure created
+  /home/circleci:
+    exists: true
+
+user:
+  # ensure circleci user exists
+  circleci:
+    exists: true

openjdk/goss.yaml

@@ -0,0 +1,13 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true
+
+  # ensure basic directory structure created
+  /home/circleci:
+    exists: true
+
+user:
+  # ensure circleci user exists
+  circleci:
+    exists: true

php/goss.yaml

@@ -0,0 +1,13 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true
+
+  # ensure basic directory structure created
+  /home/circleci:
+    exists: true
+
+user:
+  # ensure circleci user exists
+  circleci:
+    exists: true

postgres/goss.yaml

@@ -0,0 +1,4 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true

python/goss.yaml

@@ -0,0 +1,13 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true
+
+  # ensure basic directory structure created
+  /home/circleci:
+    exists: true
+
+user:
+  # ensure circleci user exists
+  circleci:
+    exists: true

redis/goss.yaml

@@ -0,0 +1,4 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true

ruby/goss.yaml

@@ -0,0 +1,13 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true
+
+  # ensure basic directory structure created
+  /home/circleci:
+    exists: true
+
+user:
+  # ensure circleci user exists
+  circleci:
+    exists: true

rust/goss.yaml

@@ -0,0 +1,13 @@
+file:
+  # ensure a basic shell exists
+  /bin/sh:
+    exists: true
+
+  # ensure basic directory structure created
+  /home/circleci:
+    exists: true
+
+user:
+  # ensure circleci user exists
+  circleci:
+    exists: true

shared/images/build.sh

@@ -57,36 +57,37 @@ function run_goss_tests() {
     mkdir $GOSS_DOCKERFILE_PATH
 
     echo "----------------------------------------------------------------------------------------------------"
-    echo "copying Dockerfile to $GOSS_DOCKERFILE_PATH for testing modifications"
+    echo "copying Dockerfile to $GOSS_DOCKERFILE_PATH for testing modifications..."
     cp ~/circleci-bundles/$DOCKERFILE_PATH $GOSS_DOCKERFILE_PATH
 
     # cat our additions onto the Dockerfile copy
     echo "----------------------------------------------------------------------------------------------------"
-    echo "adding the following modifications to copied Dockerfile:"
+    echo "adding the following modifications to copied Dockerfile..."
     echo "----------------------------------------------------------------------------------------------------"
     cat ~/circleci-bundles/shared/goss/goss-add.Dockerfile
     cat ~/circleci-bundles/shared/goss/goss-add.Dockerfile >> $GOSS_DOCKERFILE_PATH/Dockerfile
 
     echo "----------------------------------------------------------------------------------------------------"
-    echo "copying custom entrypoint for testing:"
+    echo "copying custom entrypoint for testing..."
     echo "----------------------------------------------------------------------------------------------------"
     cat ~/circleci-bundles/shared/goss/goss-entrypoint.sh
     cp ~/circleci-bundles/shared/goss/goss-entrypoint.sh $GOSS_DOCKERFILE_PATH
 
     # build our test image
     echo "----------------------------------------------------------------------------------------------------"
-    echo "building modified test image: $IMAGE_NAME-goss"
+    echo "building modified test image: $IMAGE_NAME-goss..."
     echo "----------------------------------------------------------------------------------------------------"
     docker build -t $IMAGE_NAME-goss $GOSS_DOCKERFILE_PATH || (sleep 2; echo "retry building $IMAGE_NAME-goss"; docker build -t $IMAGE_NAME-goss $GOSS_DOCKERFILE_PATH)
 
     # run goss tests
     echo "----------------------------------------------------------------------------------------------------"
-    echo "running goss tests on $IMAGE_NAME-goss"
+    echo "running goss tests on $IMAGE_NAME-goss..."
     echo "----------------------------------------------------------------------------------------------------"
     dgoss run $IMAGE_NAME-goss
 
     echo "----------------------------------------------------------------------------------------------------"
-    echo "removing goss variant"
+    echo "removing goss variant..."
+    echo "----------------------------------------------------------------------------------------------------"
     docker image rm $IMAGE_NAME-goss
     echo "----------------------------------------------------------------------------------------------------"
 }