BuildKit and the build cache

BuildKit is the engine that turns a Dockerfile into an image. The default Docker CLI turns it on with DOCKER_BUILDKIT=1; Docker Desktop and modern docker buildx route through it by default. Most engineers treat BuildKit as a faster version of the old builder when it is in fact a different model: a content-addressable graph with an explicit cache layer.

BuildKit as a graph

A Dockerfile is a sequence of instructions. The old Docker builder treated each RUN, COPY, and ADD as a layer to push in order, and could only reuse a layer when its byte-for-byte command, parent, and prior layer all matched. BuildKit treats the inputs of each step - source files, environment, mounts, secrets - as nodes in a graph and produces a layer whose digest is computed from those inputs. If the inputs do not change, the layer is reused; if any input changes, only the steps downstream of the change are rebuilt.

flowchart LR
    A["Source files"] --> B["deps step"]
    B --> C["build step"]
    D["Base image"] --> B
    E["package.json"] --> B
    B --> F["Final image"]
    C --> F

Two consequences follow:

• A change to one source file invalidates the step that consumes that file but not unrelated steps. In the graph above, changing a .c file does not rebuild the package install.
• Steps that consume files mounted at build time - via RUN --mount=type=cache,target=... - share their cache across builds without baking those files into a layer.

The cost of the graph is that BuildKit must know what each step consumes. Files that matter must be COPY-ed or explicitly mounted; context files that are not consumed should be excluded via .dockerignore. A step that reads arbitrary files from the build context will see “too much” and invalidate the cache more often than needed.

What the cache actually is

BuildKit keeps two kinds of cache:

• Local cache. A per-runner cache directory (/var/lib/buildkit on a Linux host). It is keyed on content and is read by the same runner it was written by. Local cache does not survive runner replacement.
• Remote / exported cache. A cache that is pushed to and pulled from a registry as an OCI artifact, addressed by content. A runner references it with --cache-from type=registry,ref=ghcr.io/org/app:cache.

The CI workflow in practice:

export COMMIT_SHA=$(git rev-parse --short HEAD)
docker buildx build \
  --cache-from type=registry,ref=ghcr.io/org/app:cache \
  --cache-to type=registry,ref=ghcr.io/org/app:cache,mode=max \
  --tag ghcr.io/org/app:$COMMIT_SHA \
  --push .

The --cache-from line is the read side. On a typical PR with no changes to package.json or to the base image, every step hits the cache; the build finishes in seconds. On a PR that changes a Go file, the package-install step hits cache, the compile step rebuilds, and downstream steps inherit a fresh layer.

Cache mounts

Cache mounts are the second optimisation. A cache mount is a bind that BuildKit manages across RUN steps without baking the contents into a layer:

RUN --mount=type=cache,target=/root/.cache \
    pip install -r requirements.txt

Pip’s download cache lives at /root/.cache/pip; the cache mount points at that path and persists across runs. The first cold run downloads every wheel; the second run reuses the wheels for any package whose version has not changed. Cache mounts work for pip, apt, go mod, npm, maven, and any tool whose intermediate artefacts live under a known directory.

The trade-off is that cache mounts are opaque to the layer: the contents of /root/.cache are not part of the image and are not recorded in provenance. For tooling whose cache contents do not need to be audited, this is an unalloyed win; for tooling whose contents must be reproducible, it must be turned off.

The cost of the cache

Three costs worth naming explicitly:

1. Disk. Local cache grows until pruned. Self-hosted runners need a cron that runs docker buildx prune or a size cap.
2. Time to first build. A new repository on a new runner takes the full build time once, before any cache exists. Pipeline design has to accept this and budget for cold builds.
3. Trust. As above. Anything pulled from a cache is, formally, an unverified input that the build consumed.

Production discipline

• Use BuildKit, not the legacy builder. Set DOCKER_BUILDKIT=1 in CI; pin docker/build-push-action to a version that defaults to BuildKit; do not run docker build without it.
• Excluded context aggressively. A bloated .dockerignore shortens the build context and reduces cache invalidation from file events the step never read.
• Use a registry cache for shared CI runners. Ephemeral runners with no local cache pay the cold-build cost every run; a registry cache amortises that cost across the team.
• Lock BuildKit versions across the team. Different runner versions produce subtly different layers; in a heterogeneous fleet, lock the image.
• Audit what the cache mount hides. If a regulator or auditor cares about the build’s contents, the cache mount must be documented or removed.

Cross-course references

• Containerisation for Production Sysadmins - Parts V-VII (advanced Docker) cover the runtime side of the same graph: layers, copy-on-write, and storage drivers.
• Build Automation for Production Sysadmins - Parts II-III (incremental builds) cover the same graph at the Make/Bazel level; BuildKit is the containerisation of that idea.

Quiz