Container build secrets, image layers and runtime injection

An image is not a filesystem. It is an ordered list of content-addressed blobs, each one a serialised set of changes to be applied on top of the previous. That distinction decides the whole of container secret handling, because a change set can record that a file should disappear without being able to reach back into a blob that has already been sealed, digested and pushed. Everything that follows is a consequence of that one property.

A deletion is a marker, not an erasure

The OCI image specification describes three kinds of change a layer can carry: additions, modifications and removals. Removals are represented by whiteout entries, and a whiteout is an empty file whose name is the .wh. prefix followed by the basename of the path to be deleted. The specification states that whiteouts apply only to resources in lower or parent layers, which is exactly the admission that the lower layer still contains them.

flowchart TD
    L1["Layer 1 blob\nCOPY npmrc with token"] --> L2
    L2["Layer 2 blob\nRUN npm install"] --> L3
    L3["Layer 3 blob\n.wh.npmrc marker"] --> V["Merged view:\nno npmrc visible"]
    L1 -. "still shipped, still pullable" .-> R["Registry blob store"]

The merged view a running container sees has no .npmrc in it. The registry still holds layer one, byte for byte, addressed by its own digest, and anybody who can pull the image can pull that blob. So can anybody who has the image on disk.

IMAGE=registry.example.com/payments/api:1.4.2
docker save "$IMAGE" -o /tmp/api.tar
tar -tf /tmp/api.tar

That sequence extracts the layer tarballs from the image without a container runtime being involved. There is no privileged access, no exploit and no clever trick: the format is designed to be distributable, and every consumer of the image is a consumer of every blob in it.

The same reasoning explains the limit of multi-stage builds. A multi-stage build helps only when the credential never entered a layer that ends up in the final stage. If a builder stage writes the token and the final stage copies only compiled output, the token never joins the published image. If the final stage itself writes the token and later deletes it, multi-stage has changed nothing.

Build arguments leave a permanent record

Passing a credential with --build-arg feels like passing it out of band, and it is not. The Docker documentation states directly that build arguments and environment variables are inappropriate for passing secrets because they are exposed in the final image, and that build arguments may persist in image metadata, in provenance attestations and in the image history.

# ANTI-PATTERN. Do not build this way.
# The token is recorded in image history and in provenance
# attestations, and the deleted file remains in the lower blob.
FROM node:22-slim
ARG NPM_TOKEN
RUN printf '//registry.npmjs.org/:_authToken=%s\n' "$NPM_TOKEN" > /root/.npmrc \
 && npm ci \
 && rm -f /root/.npmrc

Three separate leaks exist in that file. The value reaches the image history through the ARG and the instruction that used it. It reaches the provenance attestation that the builder generates alongside the image. And the .npmrc written in that layer stays in that layer forever, because the rm in the same instruction still produces a whiteout relative to what the instruction wrote. Squashing and re-tagging do not help, because the token has already been distributed with every push that happened before somebody noticed.

Secret mounts are the mechanism that works

BuildKit exposes a credential to one instruction and to no layer. A secret mount takes a secret from the build client and makes it temporarily available inside the build container for the duration of the build instruction, which is precisely the property the previous section was missing. It has been available by default for a long time: BuildKit became the default Linux builder in Docker v23.0 and the legacy builder was deprecated in the same release, so no DOCKER_BUILDKIT incantation is needed.

FROM node:22-slim
WORKDIR /srv/app
COPY package.json package-lock.json ./
RUN --mount=type=secret,id=npm_token,required=true,mode=0400 \
    NPM_TOKEN="$(cat /run/secrets/npm_token)" npm ci
COPY . .

The default target path is /run/secrets/ followed by the secret id, the default file mode is 0400, and the default owner is uid and gid zero. An env option exists if a tool insists on reading an environment variable rather than a file. The mount exists only while that RUN executes and never becomes part of any layer, so no whiteout is involved and nothing survives into the published image.

NPM_TOKEN_FILE="$HOME/.config/npm-ci-token"
docker build \
  --secret id=npm_token,src="$NPM_TOKEN_FILE" \
  -t registry.example.com/payments/api:1.4.2 .

The required=true option in the Dockerfile is not decoration. Its default is false, which means a mistyped or missing secret id produces an empty file, a build that succeeds, and an artefact that fails at run time for reasons that look nothing like a credential problem. Setting it turns a silent misconfiguration into a build error, which is the only place it is cheap to fix.

For private Git dependencies, BuildKit forwards an agent socket rather than a key when the build is started with the SSH option, and it recognises two predefined build secrets, GIT_AUTH_TOKEN and GIT_AUTH_HEADER, for HTTPS remotes. Both routes keep the key material on the client.

Runtime injection and where the plaintext rests

Having kept the credential out of the image, you still have to get it into the process. The choice of injection path decides who can read it on the host.

An environment variable is the weakest option, and the Compose documentation says so: environment variables risk unintentional exposure, are visible to processes that inherit them, and are routinely printed into logs during debugging. A file is better because it can carry a mode and an owner, and because a stack trace does not dump the filesystem.

services:
  api:
    image: registry.example.com/payments/api:1.4.2
    secrets:
      - db_password
secrets:
  db_password:
    file: ./secrets/db_password

Compose mounts each secret as a file under /run/secrets/ named after the secret, so the application reads a path rather than an environment variable. On Swarm the same path is served from a very different place: the value is held in the Raft log, which is encrypted and replicated across managers, delivered only to nodes running a task that has been granted explicit access, mounted into an in-memory filesystem, and unmounted and flushed from node memory when the task stops. Kubernetes, covered in the first two lessons of this part, uses a tmpfs mount with the same intent and a very different authorisation model.

Note the asymmetry those three mechanisms share. Each of them can tell you which workloads were granted the value, and none of them can tell you which processes read it, copied it into a log line, or sent it to a third party. Injection is an authorisation control, and the audit question after an incident is answered by the application, by the log store and by the credential’s own usage records, not by the orchestrator that delivered the file.

One more distinction is worth holding on to. An ENV instruction in a Dockerfile does two separate things: it records a value in the image configuration, where anybody who pulls the image can read it, and it places that value in the environment of every process the container starts. Runtime injection through an orchestrator does only the second. The first is what makes a build-time environment variable permanent, and it is why the injection mechanism you choose at run time cannot repair a decision made at build time.

Production discipline

1. Never pass a credential as a build argument. It is recorded in history and provenance by design, and no later instruction removes it.
2. Set required=true on every secret mount. The default of false converts a typo into a green build and a runtime failure somewhere else entirely.
3. Treat any image that carried a credential as burned. Rotate first, rebuild second, and only then worry about tags and registry cleanup.
4. Prefer a mounted file to an environment variable. A file carries a mode and an owner and does not travel into every child process and crash dump.
5. Scan published images, not just Dockerfiles. The Dockerfile in the repository may be clean while an image built from an older revision is still the one running.

Cross-course references

• Docker & Containers for Production Sysadmins - Parts V (Dockerfiles & BuildKit) and XIV (Secrets) cover the build system and the injection mechanisms in operational depth; this lesson treats them only as trust boundaries around key material.
• Kubernetes for Production Sysadmins - Part LXIV (Kubernetes Supply Chain Security) covers the registry, digest and signing controls that decide who can pull the blobs discussed here.
• Git, CI/CD & GitOps for Infrastructure Engineers - Part XLII (CI Secrets) covers how the credential reaches the builder in the first place, which is where a leaked build argument usually originates.

Quiz