Testing Docker images with Autotest

For awhile now, I’ve had a docker image which blindly gets rebuilt via dockerhub every time the upstream image changes. As the docker image I’m building just pulls bits and pieces from other sources, I didn’t really have anything to unit test as part of the CI process. Since it was working fine at the time, I just left it.

Fast-forward a few years and sure enough, one of the things I bundle into the image shipped a broken version. The docker build still succeeded, but the final image didn’t work as expected. Rather than let that happen again, I thought I should find a testing solution.

Docker’s Autotest looked like a good fit. It can run tests on new commits, PRs and crucially on upstream image changes. However, I was a bit confused how I could use it without bloating my image with tests embedded inside. Turns out it’s really simple using a multi-stage build.

Suppose we have the following trivial docker image, where we simploy add a dummy example-app to the alpine image.

FROM alpine:latest
COPY app/ /app
WORKDIR /app
RUN ln -s /app/example-app /usr/local/bin/example-app

We can turn it into a multi-stage image with a testing layer.

# Create release layer and label it
FROM alpine:latest as release
COPY app/ /app
WORKDIR /app
RUN ln -s /app/example-app /usr/local/bin/example-app

# Create testing layer
FROM alpine:latest as test
COPY --from=release / /
COPY test/ /test
WORKDIR /test
ENTRYPOINT ["/bin/sh", "-c"]
CMD ["./tests.sh"]

# Output release as the final image
FROM release

First we build the regular image and label it release. We then build a testing layer using release as a base and add in our tests. Here we just copy in a directory containing a bash script of arbitrary tests. The only requirement is that it exits with a 0 when tests pass, or anything else when they fail.

Finally we call FROM release to ensure that on a regular build, we only build the app without the testing layer.

For Autotest to work, we need to create a docker-compose.test.yml file with a sut service.

version: "3.8"
services:
  sut:
    build:
      context: .
      target: "test"

We set the target to test so that it builds and runs the test layer from our image. We can test it locally (with a deliberate test failure):

$ docker-compose -f docker-compose.test.yml up sut
...
sut_1  | Executing tests...
sut_1  | OK - example-app should run fine
sut_1  | FAIL - example-app should run fine
sut_1  | PASSED: 1 FAILED: 1
docker-autotest-example_sut_1 exited with code 1

Now that it works locally we just need to enable Autotest in the Dockerhub settings.

Full example repository: Autotest-example.