Capstone stage 2 — building and pinning the three images · Docker & Containers

Stage 1 left you with a host that is ready and empty. Stage 2 produces the three images the stack runs: web, api and worker.

The stage gate is not “the image builds”. It is that each image is non-root by default, carries an SBOM and a provenance attestation, passes a vulnerability threshold, and is referenced in the deployment by a digest that cannot change under you.

The shared Dockerfile shape

All three services in the capstone are built from one Go module with three entry points, which is why one Dockerfile with three build targets is the right shape here:

# syntax=docker/dockerfile:1
ARG GO_VERSION=1.23

FROM golang:${GO_VERSION}-bookworm AS build
WORKDIR /src
# Dependencies first: this layer is cached unless go.mod or go.sum change.
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
    go mod download
COPY . .
ARG TARGETOS TARGETARCH
ARG SOURCE_DATE_EPOCH
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
    go build -trimpath -ldflags '-s -w -buildid=' -o /out/ ./cmd/...

FROM gcr.io/distroless/static-debian12:nonroot AS api
COPY --from=build /out/api /app/api
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/app/api"]

FROM gcr.io/distroless/static-debian12:nonroot AS worker
COPY --from=build /out/worker /app/worker
USER 65532:65532
ENTRYPOINT ["/app/worker"]

FROM gcr.io/distroless/static-debian12:nonroot AS web
COPY --from=build /out/web /app/web
COPY --chown=65532:65532 ./public /app/public
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/app/web"]

Four decisions in there are worth stating explicitly, because each one is graded later:

distroless/static-debian12:nonroot has no shell, no package manager and no libc. An attacker who achieves code execution has nothing to pivot with, and the CVE surface is the application plus CA certificates. The :nonroot variant sets USER 65532 for you; the explicit USER line above is redundant and kept deliberately so that a base-image change cannot silently make the image root.
CGO_ENABLED=0 produces a static binary, which is what allows the static distroless base rather than the larger base variant.
-trimpath and -buildid= remove build-path and build-id variability, which is most of what stands between you and a reproducible build.
Cache mounts rather than a copied module cache: the cache never becomes a layer, so it cannot leak into the published image.

A distroless image has no shell, and that changes your health check

HEALTHCHECK CMD curl -f http://localhost:8080/health does not work here. There is no curl, no wget, and no /bin/sh for the CMD form to invoke.

Three workable answers, in order of preference:

Let the orchestration layer probe over HTTP. In the capstone the Compose healthcheck for web and api can use the CMD exec form against a tiny health-check binary you built alongside the service.
Ship a health sub-command in the same binary: ENTRYPOINT ["/app/api"] with HEALTHCHECK CMD ["/app/api", "healthcheck"]. One binary, no extra surface, and it can check internal state a HTTP probe cannot.
Use the debug distroless tag in non-production only, which adds a busybox shell. Never in the image you deploy.

Option 2 is what the capstone uses.

Building with attestations

Configuration changebuild all three with SBOM and provenance— --sbom and --provenance attach in-toto attestations to the manifest. mode=max records the full build environment.

export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
for svc in api worker web; do
  docker buildx build \
    --target "$svc" \
    --tag "registry.example.com/$svc:1.4.0" \
    --build-arg SOURCE_DATE_EPOCH="$SOURCE_DATE_EPOCH" \
    --label org.opencontainers.image.source=https://git.example.com/org/app \
    --label org.opencontainers.image.revision="$(git rev-parse HEAD)" \
    --label org.opencontainers.image.version=1.4.0 \
    --sbom=true \
    --provenance=mode=max \
    --push .
done

--sbom=true and --provenance=mode=max are shorthands for --attest=type=sbom and --attest=type=provenance,mode=max. Both require a registry output — attestations are separate manifests in an OCI index, and --load into the local Docker image store cannot represent that. A build that produced no attestation is almost always a build that used --load instead of --push.

Read-only / Safeverify the attestations exist— Reads the pushed index back from the registry and confirms the attestation manifests are attached.

docker buildx imagetools inspect registry.example.com/api:1.4.0 --raw \
  | jq -r '.manifests[] | "\(.platform.os)/\(.platform.architecture) \(.annotations["vnd.docker.reference.type"] // "image")"'

linux/amd64 image
unknown/unknown attestation-manifest

Illustrative output

An index with only the image line and no attestation-manifest means the attestations were not attached. That is a gate failure, not a warning.

The scan gate

# Fails the pipeline on any CRITICAL, or on a HIGH that has a fix available.
IMG=registry.example.com/api:1.4.0
trivy image --severity CRITICAL --exit-code 1 --ignore-unfixed "$IMG"
trivy image --severity HIGH --exit-code 1 --ignore-unfixed "$IMG"

--ignore-unfixed is the setting that decides whether the gate is useful or ignored. A distroless Go image typically has a handful of findings against packages with no upstream fix; blocking on those teaches the team to pass --exit-code 0, after which the gate protects nothing. Block on what can be fixed; track the rest with a review date.

Scan the digest, not the tag

By the time the scan runs, the tag has been pushed and could in principle be repointed. Capture the digest from the build and scan that:

DIGEST=$(docker buildx imagetools inspect registry.example.com/api:1.4.0 \
  --format '{{.Manifest.Digest}}')
trivy image --severity CRITICAL --exit-code 1 "registry.example.com/api@$DIGEST"

The same digest then flows into the signature and into the deployment, so the thing scanned, the thing signed and the thing running are provably identical.

Signing and pinning

DIGEST=$(docker buildx imagetools inspect registry.example.com/api:1.4.0 \
  --format '{{.Manifest.Digest}}')
cosign sign --yes "registry.example.com/api@$DIGEST"
cosign verify \
  --certificate-identity-regexp 'https://git\.example\.com/org/app/.*' \
  --certificate-oidc-issuer https://token.actions.example.com \
  "registry.example.com/api@$DIGEST"

The stage deliverable is a file the deployment reads, generated by the pipeline rather than typed by a person:

Read-only / Safeproduce the digest manifest— One line per service, committed next to compose.yml. This is what stage 3 consumes.

for svc in api worker web; do
  D=$(docker buildx imagetools inspect "registry.example.com/$svc:1.4.0" \
        --format '{{.Manifest.Digest}}')
  echo "${svc}_DIGEST=registry.example.com/$svc@$D"
done | tee digests.env

api_DIGEST=registry.example.com/api@sha256:4f2ac19d3b8e7a10c5d9f2b47e6a08c31d5b9e7c2a4f60d81b3e5c7a9d0f2b46
worker_DIGEST=registry.example.com/worker@sha256:81e077b3a2c46d95f108b3e7c5a9d2f460b8e1c37a5d9f02b4e6c8a1d3f5b709
web_DIGEST=registry.example.com/web@sha256:9b7102af5c3e8d16b4a0f2c79e5d38b1a6c04e9f2d7b5a8c1e3f60d92b4a7c85

Illustrative output

# compose.yml (stage 3 consumes digests.env through an env_file or a template)
services:
  api:
    image: ${api_DIGEST}

Reproducibility as a gate

Read-only / Safebuild twice, compare digests— Two builds from the same commit with the same SOURCE_DATE_EPOCH should produce the same image digest.

export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
A=$(docker buildx build --target api --build-arg SOURCE_DATE_EPOCH="$SOURCE_DATE_EPOCH" \
      --output type=image,name=localhost/api:r1,push=false,rewrite-timestamp=true \
      --metadata-file /tmp/a.json . >/dev/null 2>&1; jq -r '.["containerimage.digest"]' /tmp/a.json)
B=$(docker buildx build --target api --build-arg SOURCE_DATE_EPOCH="$SOURCE_DATE_EPOCH" \
      --output type=image,name=localhost/api:r2,push=false,rewrite-timestamp=true \
      --metadata-file /tmp/b.json . >/dev/null 2>&1; jq -r '.["containerimage.digest"]' /tmp/b.json)
[ "$A" = "$B" ] && echo "OK reproducible: $A" || echo "DIFFER: $A vs $B"

OK reproducible: sha256:4f2ac19d3b8e7a10c5d9f2b47e6a08c31d5b9e7c2a4f60d81b3e5c7a9d0f2b46

Illustrative output

Treat a mismatch as informational rather than as a hard gate on a first capstone. Reproducibility is achievable for a static Go binary and much harder for an interpreted stack with a package manager in the build; what matters is that you measured it and know which category you are in.

What provenance actually records, and what it is worth

A provenance attestation is an in-toto statement, signed and stored as a separate manifest in the image index, describing how the image was produced: the builder identity, the source repository and commit, the build parameters, the base images by digest, and with mode=max the full set of build arguments and the materials consumed.

What that buys you is a specific and narrow thing: given an image, you can determine which commit and which builder produced it, without trusting a build log that lives in a system somebody could edit.

What it does not buy you:

It does not say the source is safe. It says which source was used.
It does not say the builder was not compromised. It says which builder claimed the build.
It is only as trustworthy as the signature over it. An unsigned attestation is documentation, not evidence.

The value shows up during an incident: a CVE lands in a library, and provenance turns “which of our forty images include it” from an archaeology project into a query over the SBOMs, with the commit for each one already attached.

What happens if you deploy by tag and the pipeline re-pushes it?

Nothing visible, until a container restarts.

Docker resolves a tag when it pulls. A running container holds its image by digest, so it keeps running the image it started with. The divergence appears the moment anything triggers a pull-and-restart — a host reboot, an OOM kill with restart: unless-stopped, a docker compose up to change one unrelated service.

At that point one replica is on the audited image and the other is on whatever 1.4.0 points at now. They serve different code, the behaviour is intermittent and load-balancer dependent, and every docker inspect shows the same Config.Image string on both.

The tell is in .Image rather than .Config.Image — the two replicas have different digests. The prevention is stage 2’s deliverable: deploy the digest.

Build all three targets from one Dockerfile with cache mounts and no secrets in any layer.
Attach SBOM and provenance with --sbom=true --provenance=mode=max, pushing to a registry rather than loading locally.
Scan the digest and fail on fixable CRITICAL and HIGH findings.
Sign the digest and verify the signature in the same pipeline run.
**Emit digests.env** and commit it with the Compose file.
Gate: for each of the three images, an attestation manifest exists, the scan exits 0, cosign verify exits 0, and the digest in digests.env matches the signed digest.

The shared Dockerfile shape

Building with attestations

The scan gate

Signing and pinning

Reproducibility as a gate

Sanity check