CI/CD failure modes — the green pipeline that ships a broken image

A pipeline that fails is a nuisance. A pipeline that passes while shipping the wrong thing is an incident with a paper trail that says everything was fine, which is much harder to unpick at 2am.

Every failure in this lesson has the same shape: the pipeline’s checks and the pipeline’s output are talking about different artefacts. Once you look for that shape, they are all the same bug.

The invariant

docker buildx build --metadata-file is how you get the digest out of the build step and into everything that follows:

docker buildx build \
--tag registry.example.com/myorg/myapp:1.4.2 \
--provenance=mode=max \
--sbom=true \
--metadata-file build-metadata.json \
--push .

DIGEST=$(jq -r '."containerimage.digest"' build-metadata.json)
echo "built registry.example.com/myorg/myapp@${DIGEST}"

From here on, every stage uses myorg/myapp@$DIGEST. The tag exists for humans.

1. The tag that moved

$ docker buildx imagetools inspect registry.example.com/myorg/myapp:1.4.2 --format '{{.Manifest.Digest}}'

sha256:5781759b3d27734d4d548fcbaf60b1180dbf4290e708f01f292faa6ae764c5e6

Record that digest in the change ticket. It is the only unambiguous statement of what was deployed.

2. Jobs that race

In GitHub Actions, GitLab CI and most other systems, jobs run in parallel unless you declare a dependency. A push job that does not declare needs: test starts at the same time as test, and on a fast build it finishes first.

jobs:
  build:
    runs-on: ubuntu-latest
    # ...
  test:
    needs: build
    runs-on: ubuntu-latest
    # ...
  push:
    needs: [build, test]     # without this line, push races test
    runs-on: ubuntu-latest

The symptom is not a red pipeline. It is an image in the registry that was pushed thirty seconds before the test that would have failed it. If the deploy is triggered by the push rather than by the pipeline completing, the untested image is already live when the test job goes red.

The rule: the registry is a deployment surface, not a scratch space. If a stage needs somewhere to put an intermediate image, use a distinct repository that nothing deploys from, or keep it in the runner with --load instead of --push.

3. Testing a different build

# Two builds. Two different images. The tested one is thrown away.
docker build -t myapp:test .
docker run --rm myapp:test pytest
docker build -t myapp:1.4.2 --push .

A Dockerfile with RUN apt-get install -y curl or RUN pip install -r requirements.txt without full pinning produces a different image on every build. The second build can pick up a package published in the two minutes between them. You tested one artefact and shipped another.

Build once, then reference the result:

docker buildx build --load --tag myapp:ci --iidfile image-id.txt .
IMAGE_ID=$(cat image-id.txt)
docker run --rm "$IMAGE_ID" python -m pytest tests/

4. Scanning the wrong thing

The mirror image of the previous bug:

trivy image --severity HIGH,CRITICAL --exit-code 1 myapp:test
docker push registry.example.com/myorg/myapp:1.4.2

The scanner examined myapp:test. The push sent myapp:1.4.2. Whether those are the same bits depends on whether anything retagged them, and nothing in the pipeline asserts it.

The other half of this failure is quieter: a scanner invoked without --exit-code 1, or with a trailing || true added months ago to unblock a release, reports findings into a log nobody reads and exits 0 forever after. Grep your pipeline definitions for || true and continue-on-error and justify every occurrence.

5. Build arguments are not secrets

$ docker history --no-trunc --format '{{.CreatedBy}}' myapp:1.4.2 | grep -iE 'token|password|secret|apikey'

|1 NPM_TOKEN=REDACTED /bin/sh -c npm ci --production

If that returns anything, the credential is compromised in every copy of the image that has ever been pulled. Rotate it; do not try to rewrite the image.

6. Shared runner caches

--cache-from and --cache-to pointed at a shared registry cache make builds dramatically faster and quietly widen the trust boundary. Anything that can write to the cache can influence the layers of everything that reads from it, and on many CI systems that includes builds triggered by pull requests from forks.

The same applies to self-hosted runners with a persistent Docker daemon: jobs share the layer cache, the image store, and the build cache. A job can read what a previous job built, including layers containing another team’s credentials.

Scope caches per branch or per repository, do not let untrusted builds write to a cache trusted builds read, and prefer ephemeral runners for anything that handles release credentials.

7. The deploy that never happened

EXPECTED="$DIGEST"
ACTUAL=$(ssh deploy@appserver01.example.com \
  "docker inspect --format '{{.Config.Image}}' web")

if [ "$ACTUAL" != "registry.example.com/myorg/myapp@${EXPECTED}" ]; then
  echo "deploy assertion failed: running ${ACTUAL}, expected ${EXPECTED}" >&2
  exit 1
fi

A skipped deploy, a failed pull, a host that was down during the roll, and a docker compose up that silently kept the old container all produce the same mismatch, and the assertion catches all four.

Rollback needs the old digest to still exist

Deploying by digest only helps if the digest is still resolvable. Registry retention policies — “keep the last 10 tags”, “delete untagged manifests older than 7 days” — routinely garbage-collect exactly the manifest you want to roll back to, because the previous release’s digest became untagged the moment the tag moved.

Before you rely on a rollback path, confirm the previous digest is still there:

docker buildx imagetools inspect \
"registry.example.com/myorg/myapp@sha256:REPLACE_ME_WITH_PREVIOUS_DIGEST"

Knowledge check