Build caching — when it hits and when it does not

A Dockerfile that builds in 30 seconds when only one line changed is a Dockerfile that uses its cache correctly. A Dockerfile that re-builds everything from scratch on every commit is a Dockerfile that wastes minutes of CI time and gigabytes of network traffic.

The difference between the two is almost never cleverness. It is knowing precisely what BuildKit puts into a cache key, and putting the volatile inputs after the expensive steps rather than before them.

How the cache works

The cache is content-addressable. For each RUN, COPY, or ADD step, BuildKit computes a cache key from:

• The instruction text.
• The contents of files referenced (for COPY/ADD).
• The cache mounts declared.
• The build arguments.
• The base image’s cache key.

If the computed cache key matches an existing layer’s key, the step is skipped. If not, the step runs and the resulting layer becomes the new cached layer.

When a layer is regenerated, all subsequent layers are invalidated. The cache cannot skip ahead.

FROM ubuntu:24.04
RUN apt-get update && apt-get install -y nginx  # layer 2
COPY ./app /app                                  # layer 3
RUN pip install -r /app/requirements.txt         # layer 4

When requirements.txt changes, layer 2 (apt-get) is cached, but layer 3 (COPY) and layer 4 (pip install) re-run. When app/app.py changes, layer 4 re-runs.

The exact rule, per instruction type

The summary above is true but too coarse to design with. The documented rules differ sharply by instruction, and the difference is where the surprises live.

Two consequences worth writing on a sticky note:

1. RUN never notices the world changed. Freshness of anything a RUN downloads is your problem, solved by pinning or by deliberately busting the cache — never by hoping.
2. COPY notices everything except mtime. If a step is rebuilding and you cannot see why, the cause is a file’s content or mode inside the copied set, not its timestamp.

Why COPY . . before RUN npm ci destroys caching

This is the single most common cache defect in production Dockerfiles, and the reason is entirely mechanical.

FROM node:22-slim
WORKDIR /app
COPY . .          # key covers EVERY file in the context
RUN npm ci        # key = f(parent key, "npm ci")

RUN npm ci has a fixed command string, so its own contribution to the key never changes. But its key also folds in the key of its parent — the COPY . . — and that key is a checksum over every file in the build context. Edit one line of README.md, and:

1. The COPY . . checksum changes, so that layer is rebuilt.
2. The rebuilt layer has a new digest.
3. RUN npm ci inherits that new parent digest, so its key changes even though npm ci is byte-identical.
4. Four minutes of npm ci run again, for a README edit.

The fix is to split the copy so that the volatile files land after the expensive step:

FROM node:22-slim
WORKDIR /app
COPY package.json package-lock.json ./   # changes when deps change
RUN npm ci                                # survives ordinary commits
COPY . .                                  # changes every commit
CMD ["node", "server.js"]

Now RUN npm ci sits behind a COPY whose checksum covers two files. It re-runs when a dependency changes — which is correct — and not otherwise. Docker’s own cache-optimization guide gives this exact shape: copy the package-management files, install, then copy the project files.

Order for cache efficiency

Place instructions that change less often earlier in the Dockerfile. Place instructions that change more often later.

Stable to volatile:

1. Base image (FROM) — almost never changes.
2. System packages (apt-get install) — changes when package versions change.
3. Language dependencies (pip install, npm install) — changes when dependencies change.
4. Application source (COPY ./app) — changes every commit.

A good Dockerfile:

FROM python:3.12-slim
WORKDIR /app

# Step 1 — install system packages (changes rarely)
RUN apt-get update && apt-get install -y libpq-dev && rm -rf /var/lib/apt/lists/*

# Step 2 — install Python dependencies (changes occasionally)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Step 3 — copy source (changes every commit)
COPY . .

CMD ["gunicorn", "app:app"]

When requirements.txt changes, only step 3 re-runs. When source changes, step 3 also re-runs.

A bad Dockerfile:

FROM python:3.12-slim
WORKDIR /app
COPY . .                         # changes every commit
RUN pip install -r requirements.txt  # invalidated by every commit

Every source change invalidates the pip install layer.

Cache mounts for package managers

BuildKit supports RUN --mount=type=cache for caching package manager state across builds:

RUN --mount=type=cache,target=/root/.cache/pip \
  pip install -r requirements.txt

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
  apt-get update && apt-get install -y libpq-dev

The cache mount persists across builds (separately from the layer cache). Pip reuses downloaded wheels; apt reuses downloaded debs. Builds become dramatically faster on CI.

Cache mounts are not part of the image. They live on the build host and are pruned with docker buildx prune.

Sharing semantics, and the concurrency trap

sharing is the option nobody reads and everybody eventually needs. The Dockerfile reference documents three values, defaulting to shared:

shared is correct for a content-addressed download cache — pip wheels, Go module downloads, npm’s _cacache — where two writers adding different files to the same directory cannot corrupt each other.

shared is wrong for anything holding a database with its own lock discipline. /var/cache/apt is paired with /var/lib/apt, and apt expects exclusive access to its lists and lock files. Two concurrent builds on the same builder — which is exactly what a CI runner building three services in parallel does — will interleave writes and produce errors that look like disk corruption:

E: Could not get lock /var/cache/apt/archives/lock. It is held by process 42
E: Unable to acquire the dpkg frontend lock

That is what sharing=locked exists for, and why the apt example above carries it while the pip example does not.

Verifying a cache hit

“The build felt fast” is not verification. Wall-clock time on a CI runner varies for a dozen reasons that have nothing to do with the cache. Ask BuildKit directly.

$ docker buildx build --progress=plain -t myorg/myapp:dev . 2>&1 | grep -E 'CACHED|DONE|transferring context'

#4 [internal] load build context
#4 transferring context: 3.21kB done
#6 [2/5] WORKDIR /app
#6 CACHED
#7 [3/5] COPY package.json package-lock.json ./
#7 CACHED
#8 [4/5] RUN npm ci
#8 CACHED
#9 [5/5] COPY . .
#9 DONE 0.1s

CACHED means the step was reused. DONE with a duration means it executed. That output is the acceptance criterion for a caching change: make the edit, rebuild, and assert that RUN npm ci says CACHED. If it says DONE 214.7s, the change did not work, and no amount of the build “feeling quicker” changes that.

Note the transferring context line as well. A number in the hundreds of megabytes there is a missing .dockerignore, and it costs you on every build whether the cache hits or not.

docker buildx du
docker buildx du --verbose | head -40
docker system df

Inline cache for CI

For CI, export the cache inline with the image:

docker buildx build --cache-to=type=inline --tag myorg/myapp:1.0.0 --push .

The image manifest embeds cache metadata. Downstream pulls of the image can use the embedded cache for the next build.

docker buildx build --cache-from=type=registry,ref=myorg/myapp:1.0.0 .

If the registry image was built with inline cache, the cache metadata is present and reused.

Inline cache carries only the layers that ended up in the image. On a multi-stage build the build stage is not in it, so the expensive compile step gets no reuse. Inline cache is always mode=min; if the compile stage is what costs you, you need a registry or local backend at mode=max.

Cache backends

BuildKit cache can be stored in:

There is no default: unless you pass --cache-to, nothing is exported anywhere and a fresh builder starts cold.

The default docker driver supports inline, local, registry and gha, and the last three only with the containerd image store enabled. s3 and azblob need a different driver, normally docker-container:

docker buildx create --name ci --driver docker-container --use
docker buildx build \
  --cache-to=type=s3,region=us-east-1,bucket=myorg-buildcache,name=myapp,mode=max \
  --cache-from=type=s3,region=us-east-1,bucket=myorg-buildcache,name=myapp \
  --tag myorg/myapp:1.0.0 \
  --push \
  .

mode=max stores every layer including intermediate build stages; mode=min (the default) stores only the layers that reached the final image. Docker’s own guidance is the honest trade-off: “min cache is typically smaller (which speeds up import/export times, and reduces storage costs), max cache is more likely to get more cache hits.”

Knowledge check