Reproducibility — pinned bases, deterministic layers · Docker & Containers

A reproducible build produces the same output bytes given the same inputs. Reproducibility is the foundation of supply chain security: if you cannot reproduce a build, you cannot audit it, you cannot sign it, and you cannot compare it to what is running.

It is also frequently oversold. Most of what is written about container reproducibility describes SOURCE_DATE_EPOCH and stops, which leaves the impression that setting one environment variable gets you byte-identical images. It does not, and the gap between what it fixes and what it does not is the useful content of this lesson.

Two different goals, often confused

	What it claims	What it costs	When you need it
Rebuildable	The same source and pinned inputs produce a functionally identical image	Pinning discipline	Always
Reproducible	The same source and pinned inputs produce the same digest	Pinning, plus timestamp normalisation, plus eliminating every non-deterministic tool in the chain	Attested supply chains, regulated builds, independent verification

Almost every estate needs the first and treats the second as aspirational. That is a defensible position, provided it is a decision rather than an accident. Say which one you are claiming, in writing, because “our builds are reproducible” is an assertion an auditor will test.

What non-determinism looks like

A non-reproducible build produces different bytes each time, even with identical inputs. Common causes:

Timestamps (date in commands).
Random values (UUIDs, temp files in /tmp).
Filesystem order (ls outputs differently on different filesystems).
Package manager state (apt’s package cache, npm’s installed-by).
Compression artefacts (gzip header includes mtime).
Order of files in tar archives.

Pinning inputs

Pin everything that affects the output:

FROM ubuntu:24.04@sha256:REPLACE_WITH_REAL_DIGEST

Pin the base image by digest. Pin package versions:

RUN apt-get update \
 && apt-get install -y --no-install-recommends nginx=1.24.0-2ubuntu7.4 \
 && rm -rf /var/lib/apt/lists/*

Pin toolchain versions:

FROM golang:1.24.5@sha256:REPLACE_WITH_REAL_DIGEST

Pin the Dockerfile frontend too, if you use the # syntax directive — # syntax=docker/dockerfile:1 is a moving tag, and a frontend change can alter how instructions are lowered into the build graph.

`SOURCE_DATE_EPOCH`: what it does and does not touch

SOURCE_DATE_EPOCH is a Reproducible Builds convention: a Unix timestamp, normally the commit time of the source, that build tools use in place of “now”.

BuildKit understands it, and understands it in a more limited way than most people assume.

Which timestamps BuildKit rewrites, and which it leaves alone

BuildKit’s reproducibility documentation is precise about the propagation: “The buildctl CLI (>= v0.13) and Docker Buildx (>= 0.10) automatically propagate the $SOURCE_DATE_EPOCH environment value from the client host to the SOURCE_DATE_EPOCH build arg.”

So you set it as an environment variable on the client and pass nothing on the command line.

What the value is then applied to:

“the created timestamp in the OCI Image Config”
“the created timestamp in the history objects in the OCI Image Config”
“the org.opencontainers.image.created annotation in the OCI Image Index”
“the timestamp of the files exported with the local exporter”
“the timestamp of the files exported with the tar exporter”

Read that list again for what is missing. The mtimes of files inside the image layers are not in it. By default you get a normalised image config and index, sitting on top of layers whose file timestamps are still whatever the build happened to produce. Two builds an hour apart therefore have identical configs and different layer digests — which means a different image digest, which means the reproducibility claim fails at exactly the level people check it.

Rewriting the in-layer timestamps is a separate, explicit exporter option, available since BuildKit v0.13:

docker buildx build \
  --output type=image,name=registry.example.com/myorg/app:1.0.0,push=true,rewrite-timestamp=true \
  .

One more consequence, from Docker’s cache-invalidation reference: “Changing SOURCE_DATE_EPOCH between builds invalidates the cache for WORKDIR and all subsequent instructions.” Wire a per-build date +%s into it and you have not made the build reproducible, you have made it uncacheable.

We set SOURCE_DATE_EPOCH and the digests still differ

The pipeline exports SOURCE_DATE_EPOCH from the commit time. The two builds are on identical runners, from an identical commit, with a digest-pinned base. The image digests differ anyway.

The instinct is to hunt for a rogue date call in the Dockerfile. Usually there is none.

Two much more likely causes, in this order:

In-layer file timestamps were never normalised, because rewrite-timestamp=true was not set on the exporter. The config is identical; the layers are not.
Provenance attestations. Provenance is attached at mode=min by default and records the build’s own metadata, including timing and builder identity. That changes the index digest between runs even when the image manifest is stable. When you are testing reproducibility, build with --provenance=false and compare containerimage.digest; add provenance back for the release build, and make your reproducibility claim about the image manifest rather than the index.

The diagnosis that wastes the most time is assuming the difference is in your application. Compare at the right level first: config, then layers, then index.

Determinism in build steps

# Bad — uses `date` which changes every build
RUN echo "Built at $(date)" > /build-info.txt

# Better — use the SOURCE_DATE_EPOCH build arg
ARG SOURCE_DATE_EPOCH
RUN echo "Built at $(date -u -d "@$SOURCE_DATE_EPOCH" --iso-8601=seconds)" > /build-info.txt

Declaring ARG SOURCE_DATE_EPOCH with no default is what makes the propagated build arg visible inside the stage. With a default of 0, a missing value silently becomes 1970 rather than failing.

# Bad — depends on filesystem enumeration order
RUN find /usr/share -type f | xargs strip

# Better — deterministic ordering, and safe against odd filenames
RUN find /usr/share -type f -print0 | sort -z | xargs -0 -r -n1 strip

For Go specifically:

RUN CGO_ENABLED=0 GOOS=linux \
    go build -trimpath -ldflags="-s -w -buildid=" -o /out/app

-trimpath removes local paths from the binary — without it the build directory is embedded and a build under /home/alice differs from one under /build. -buildid= clears the action ID Go otherwise records. Go does not read SOURCE_DATE_EPOCH; do not add it to the go build line expecting an effect.

For tar archives:

ARG SOURCE_DATE_EPOCH
RUN tar --sort=name --mtime="@${SOURCE_DATE_EPOCH}" --owner=0 --group=0 \
        --numeric-owner -cf /tmp/app.tar -C /src .

Sort by name, fixed mtime, fixed ownership.

The honest ceiling

Timestamp normalisation is the easy half. These are the things it cannot reach, and they are why full bit-for-bit reproducibility is a project rather than a flag:

Package managers resolve. apt-get install nginx picks whatever the archive serves today. Pinning the exact version string helps until the version is superseded and the .deb is deleted from the pool — at which point the build stops working rather than producing different bytes, which is arguably the better failure. npm ci and pip install -r with hashes are genuinely deterministic; npm install and unpinned pip are not.
Compilers embed things. Build paths, build IDs, debug sections, parallelism-dependent symbol ordering. Every toolchain has its own set of flags for this and its own remaining gaps.
Compression is not part of the content. Layer blobs are compressed, and identical uncompressed content can compress to different bytes under a different zlib build or compression level. That changes the layer digest and therefore the image digest, with no difference in what the layer contains.
Cache mounts hide inputs. A RUN --mount=type=cache step can consume an artefact left by an earlier, unrelated build. Its contents are deliberately excluded from the cache key, so nothing records what was used.
The base image is somebody else’s build. You can pin its digest; you cannot make it reproducible.

Verifying reproducibility

Comparing exported tarballs does not answer the question — a filesystem tar includes mtimes and ordering that have nothing to do with the image. Compare the thing that is actually addressed: the image digest.

Read-only / Safedigest comparison— Build twice from a cold cache with provenance off, and compare the recorded image digests. Non-zero exit means the build is not reproducible.

export SOURCE_DATE_EPOCH=1767225600

docker buildx build --no-cache --provenance=false \
--metadata-file /tmp/build-a.json --output type=image,rewrite-timestamp=true .
docker buildx build --no-cache --provenance=false \
--metadata-file /tmp/build-b.json --output type=image,rewrite-timestamp=true .

A=$(jq -r '."containerimage.digest"' /tmp/build-a.json)
B=$(jq -r '."containerimage.digest"' /tmp/build-b.json)
printf 'A=%s\nB=%s\n' "$A" "$B"
test "$A" = "$B" || { echo 'FAIL: build is not reproducible'; exit 1; }
echo 'reproducible'

Read-only / Saferesult— What a failing reproducibility check looks like. The two digests differ, so the build is not reproducible — and nothing else on the page would have told you.

$ bash ./check-reproducible.sh

A=sha256:9f2b1c4a7d3e5081bb6c2f9a4e7d1c8305a6f4b2e9d7c1a3
B=sha256:41c8e0a95b7d2f36ca9e4b1d80f27a6c3e5b9d84f1a02c67
FAIL: build is not reproducible

Illustrative output

--no-cache matters: a second build that reuses every layer will of course produce the same digest, and proves nothing at all. The check is only meaningful from a cold cache, which is also why it belongs in a nightly job rather than on every commit.

When it fails, narrow it down:

Read-only / Safenarrow the difference— Compare configs first, then layer digests. The level at which they diverge tells you which class of non-determinism you are dealing with.

docker buildx imagetools inspect --raw registry.example.com/myorg/app:a > /tmp/a-manifest.json
docker buildx imagetools inspect --raw registry.example.com/myorg/app:b > /tmp/b-manifest.json
diff -u /tmp/a-manifest.json /tmp/b-manifest.json

If only the config digest differs, it is metadata — timestamps, history, labels. If layer digests differ, it is content or compression, and dive or an unpacked docker save will show you which paths.

Why this matters for security

A reproducible build can be reproduced by an auditor. If the auditor’s build differs from yours, your build is suspicious.

Signatures apply to reproducible builds. A signature on a non-reproducible build is fragile: a bit-flipped timestamp invalidates the signature.

Supply chain attacks (like SolarWinds) succeed in part because the affected products were not reproducible; the auditors could not independently verify what was shipped.

What if a pinned base image digest disappears from the registry?

FROM ubuntu:24.04@sha256:... fails to pull, and every build that references it stops — usually all of them at once, on a day nobody changed anything. Registries do garbage-collect, mirrors do expire, and an upstream project does occasionally delete a repository.

This is the cost of pinning, and it is a real one. Mitigate it the same way you mitigate any external dependency: mirror the base images you depend on into a registry you control, and pin against that. A pull-through cache is the low-effort version; an explicit docker buildx imagetools create copy into an internal repository is the version that survives the upstream deleting things.

Recovering without a mirror means finding an equivalent digest, which is not the same image and therefore breaks the very property you were pinning for.