Image testing in CI

A CI job that builds an image but never tests it produces an artifact with unverified properties. Every tag pushed without a test is a guess about what the image can do. Part LIII’s non-negotiable is that the tag only lands after four kinds of test have passed: vulnerability, size, structure, and behaviour.

The four test categories

Each category asks a different question:

• Vulnerability. Does any package in the image carry a CVE that exceeds the team’s threshold? Scanners (Docker Scout, Trivy, Grype, Snyk) compare package lists against CVE feeds and assign severities.
• Size. Is the image under the team’s size budget? Budgets are usually expressed as absolute bytes (image < 200 MB) rather than deltas (+10 MB over last build) because storage and pull costs are absolute.
• Structure. Does the image have the layout the deployment assumes? Tools like dive enumerate layers and their size; tools like crane query the OCI config for User, Entrypoint, Env, WorkingDir.
• Behaviour. Does the entrypoint start and answer the liveness contract? Behaviour tests run the image and probe its healthcheck or its management port.

flowchart LR
    A["Built image"] --> B["Vulnerability scan"]
    A --> C["Size budget"]
    A --> D["Structure (OCI config)"]
    A --> E["Behaviour smoke"]
    B --> F{"All green?"}
    C --> F
    D --> F
    E --> F
    F -->|yes| G["Tag and push"]
    F -->|no| H["Block, notify"]

Each category catches a distinct class of failure. A vulnerability scan does not catch a 600 MB image size regression. A size budget does not catch a USER root regression. A structure test does not catch an exec failed at runtime. A behaviour test does not catch a CVE.

Vulnerability scanning

Docker Scout ships with Docker Desktop and integrates with the Docker Hub CVE feed. It can scan a local image directly:

docker/scout cves --image app:$COMMIT_SHA --exit-code --severity high,critical

The --exit-code flag makes the command exit non-zero on any match, which is what CI needs to fail the job. The --severity filter scopes the gate to high and critical, leaving medium and low to a periodic review rather than a per-build block.

A typical CI step:

- name: Scan image
  uses: docker/scout-action@v1
  with:
    command: cves
    image: app:$COMMIT_SHA
    severity: high,critical
    exit-code: "true"

The choice of scanner matters less than the choice of threshold. Different scanners disagree on package lists and on CVE feeds; pick one, document its false-positive rate, and use the same threshold across the team.

Size budgets

A size regression is one of the cheapest CI failures to detect and one of the most expensive to debug in production. A 50 MB image becomes a 600 MB image because a developer added apt-get install vim to a debug step and forgot to remove it.

The CI step that gates on size:

SIZE=$(docker image inspect app:$COMMIT_SHA --format '{.Size}')
LIMIT=$((200 * 1024 * 1024))
if [ "$SIZE" -gt "$LIMIT" ]; then
  echo "image exceeds 200MB budget: $SIZE bytes"
  exit 1
fi

The threshold is a team decision. 50 MB for a static Go binary, 500 MB for a Python service with ML models, 2 GB for a model serving workload. The discipline that matters is that the budget exists and is enforced.

Structure tests

A structure test asserts the OCI image configuration matches what the deployment assumes:

USER=$(docker image inspect app:$COMMIT_SHA --format '{.Config.User}')
if [ "$USER" != "nonroot:nonroot" ]; then
  echo "image must run as nonroot:nonroot, got: $USER"
  exit 1
fi

The same pattern applies to:

• Config.Entrypoint and Config.Cmd matching expectations.
• Config.WorkingDir matching /app or another known absolute path.
• Config.Env not carrying secrets.
• RootFS.DiffIDs matching expected layer count.

A library such as goss, container-structure-test, or python:testinfra codifies these as declarative specs.

Behaviour tests

The cheapest, highest-value image test is the behaviour smoke: run the image, exercise the entrypoint, confirm it responds. The mechanism is HEALTHCHECK in the Dockerfile plus a CI step that runs the container and probes it.

HEALTHCHECK --interval=30s --timeout=3s \
    CMD curl -f http://localhost:8080/healthz || exit 1

The CI test:

docker run --detach --name app-under-test \
    --health-cmd "curl -f http://localhost:8080/healthz" \
    app:$COMMIT_SHA
# wait for healthy
for i in 1 2 3 4 5 6 7 8 9 10; do
  STATUS=$(docker inspect --format='{.State.Health.Status}' app-under-test)
  [ "$STATUS" = "healthy" ] && break
  sleep 2
done
[ "$STATUS" = "healthy" ] || { echo "image not healthy"; exit 1; }
docker rm -f app-under-test

The behaviour test catches distroless-without-runtime mistakes (LIII-03’s example), CMD typos, missing libraries, and start loops. It is the only test category that exercises the image the way production will.

Production discipline

• Test before tagging, not after. An image tagged before the test passes is an image that exists by digest in the registry even if the test would have failed. This is the registry-cache vulnerability: a tagged digest is observable to deploys even after the test fails. Tag after tests.
• Use the same threshold in CI as on the security dashboard. A scan that fails the build but does not appear in the security team’s dashboard creates two views of the truth.
• Treat size and structure regressions as bugs, not advisories. They are usually caused by a specific commit; revert or fix forward, do not raise the threshold.
• Run a behaviour test for every entrypoint change. A Dockerfile diff that changes ENTRYPOINT has not been deployed until a real container has run it.

Cross-course references

• Containerisation for Production Sysadmins - Part X (image debugging) covers the runtime equivalents of these tests and why they belong in CI rather than at deploy time.
• Container Security for Production Sysadmins - Part V (scanning pipelines) covers the policy side: which CVEs are blocking, which are advisory, and how those policies map to the gates above.

Quiz