Image and secret hardening — proving nothing leaked

An image is a filesystem plus a config, both of which are read by everyone who can pull it. The hardening question is not “did we mean to put a credential in there” — nobody means to — but “can we demonstrate that we did not”.

This lesson is the demonstration. Four properties, each with a command that returns a verdict: the image runs as a non-root user, the running container’s identity is pinned to a digest, no layer contains a credential, and every secret the application uses arrived at runtime.

Property 1 — the image does not default to root

for i in $(docker image ls --format '{{.Repository}}:{{.Tag}}' | grep -v '<none>'); do
U=$(docker image inspect "$i" --format '{{.Config.User}}')
printf '%-44s USER=%s\n' "$i" "${U:-ROOT}"
done

registry.example.com/api:1.4.0               USER=10001
registry.example.com/web:1.4.0               USER=10001
postgres:16                                  USER=ROOT
redis:7-alpine                               USER=ROOT

Third-party images defaulting to root is normal — postgres drops privileges itself after start-up, and redis expects you to set the user. What matters is that the runtime overrides it, which the previous lesson’s audit already checks. What this check catches is your own image shipping without a USER line, so that the safety depends entirely on every deployment remembering --user.

Use a numeric UID in the USER instruction, not a name:

RUN useradd --system --uid 10001 --no-create-home app
USER 10001:10001

A named user requires /etc/passwd to resolve it, which distroless and scratch images may not have, and which means the runtime cannot verify the UID without starting the container.

Property 2 — you know exactly what is running

A tag is a mutable pointer. api:1.4.0 today and api:1.4.0 next month can be different images, which makes every statement about “the image we audited” unverifiable.

docker ps -q | while read -r c; do
  docker inspect --format '{{.Name}} {{.Config.Image}} {{.Image}}' "$c"
done

/api registry.example.com/api:1.4.0 sha256:4f2a...c19d
/web registry.example.com/web:1.4.0 sha256:9b71...02af

for c in $(docker ps -q); do
  RUNNING=$(docker inspect --format '{{.Image}}' "$c")
  TAG=$(docker inspect --format '{{.Config.Image}}' "$c")
  CURRENT=$(docker image inspect "$TAG" --format '{{.Id}}' 2>/dev/null || echo missing)
  [ "$RUNNING" = "$CURRENT" ] || echo "DRIFT: $TAG running $RUNNING, tag now $CURRENT"
done
true

DRIFT: registry.example.com/api:1.4.0 running sha256:4f2a...c19d, tag now sha256:81e0...77b3

That output means somebody rebuilt and re-pushed 1.4.0. Whatever was scanned and approved is not what is running, or will not be after the next restart.

Pin deployments by digest so the question cannot arise:

services:
  api:
    image: registry.example.com/api@sha256:4f2ac19d0000000000000000000000000000000000000000000000000000c19d

Property 3 — no layer contains a credential

This is the property people get wrong, because they check the final filesystem rather than the layers.

IMG=registry.example.com/api:1.4.0
docker history --no-trunc --format '{{.CreatedBy}}' "$IMG" \
  | grep -inE 'password|passwd|secret|token|api[_-]?key|BEGIN [A-Z ]*PRIVATE KEY' \
  || echo 'OK: no credential-shaped strings in history'

OK: no credential-shaped strings in history

IMG=registry.example.com/api:1.4.0
docker image inspect "$IMG" --format '{{range .Config.Env}}{{println .}}{{end}}' \
  | grep -iE 'password|secret|token|key=' \
  || echo 'OK: no credential-shaped environment variables'

OK: no credential-shaped environment variables

For the layer contents themselves, export and search. This is slow and worth doing once per release rather than per build:

IMG=registry.example.com/api:1.4.0
WORK=$(mktemp -d)
docker save "$IMG" -o "$WORK/image.tar"
tar -xf "$WORK/image.tar" -C "$WORK"
grep -rlE 'BEGIN (RSA|OPENSSH|EC|PGP) PRIVATE KEY|AKIA[0-9A-Z]{16}' "$WORK" || echo 'OK: no key material in layers'
rm -rf "$WORK"

A dedicated secret scanner does this better and with far fewer false negatives. Run one in CI — the point of showing the manual version is that you can reproduce a finding on a host with nothing installed.

Property 4 — secrets arrive at runtime, and only at runtime

docker ps -q | while read -r c; do
  N=$(docker inspect --format '{{.Name}}' "$c")
  docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' "$c" \
    | grep -iE '(password|secret|token|apikey)=..' \
    | sed "s|^|$N |"
done
true

/legacy-batch DB_PASSWORD=REDACTED
/adminer ADMINER_PASSWORD=REDACTED

Two findings. docker inspect is readable by every member of the docker group, environment variables are inherited by every child process, and /proc/<pid>/environ exposes them to anything with the same UID in the container.

The correct shape is a file the runtime provides:

services:
  api:
    image: registry.example.com/api@sha256:4f2ac19d00000000000000000000000000000000000000000000000000000000
    environment:
      DB_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password

secrets:
  db_password:
    file: ./secrets/db_password

docker exec api ls -l /run/secrets/
CID=api
PID=$(docker inspect --format '{{.State.Pid}}' "$CID")
sudo grep 'run/secrets' "/proc/$PID/mounts"

total 4
-r--r--r-- 1 10001 10001 33 Aug 11 09:14 db_password
tmpfs /run/secrets tmpfs ro,relatime,size=65536k,inode64 0 0

ro on the mount and mode 0444 owned by the container UID is the target state. A secret on a writable mount can be modified by the application, which defeats rotation.

Build-time secrets

A credential needed during a build — a private package registry token, an SSH key for a private module — must never be an ARG or a COPY. BuildKit mounts it for the duration of one RUN and leaves nothing in the layer:

# syntax=docker/dockerfile:1
FROM golang:1.23 AS build
RUN --mount=type=secret,id=npmtoken,target=/run/secrets/npmtoken \
    NPM_TOKEN=$(cat /run/secrets/npmtoken) && \
    npm ci --registry https://npm.example.com

docker build --secret id=npmtoken,src=./npm_token.txt -t app:1.0.0 .

The mount exists only while that RUN executes. Prove it afterwards with the same docker history grep — the secret’s id appears, its value does not.

1. **Assert a non-root numeric USER** in every image you build, and check it in CI rather than in review.
2. Fail the build on a secret-scanner finding, over history, config and layer contents.
3. Publish by digest and deploy by digest; keep the tag as a label for humans.
4. Verify signatures at deploy time against the digest, not the tag.
5. Audit running containers for credential-shaped environment variables on the same schedule as the container hardening audit.
6. **Confirm /run/secrets is a read-only tmpfs** holding exactly the secrets the service needs and no more.

Sanity check