A secret is already in the image — layer forensics and remediation

The earlier lessons in this part are about not putting secrets into images. This one starts from the position everybody eventually finds themselves in: it is already there, it was pushed six weeks ago, and somebody has just asked what the blast radius is.

Why the obvious fix does not work

The Dockerfile that causes this is nearly always some variant of:

FROM debian:13-slim
COPY deploy_key /root/.ssh/id_ed25519
RUN git clone git@example.com:org/private-lib.git /src \
 && rm -f /root/.ssh/id_ed25519

The author reasoned: the key is deleted before the image is finished, so the image does not contain the key. That reasoning would be correct for a filesystem. It is wrong for an image, because an image is not a filesystem — it is an ordered stack of filesystem diffs, and each one is a separate, immutable, independently addressable blob.

COPY deploy_key produced a layer containing the key. RUN ... rm produced a later layer containing a whiteout marker — a small file that instructs the union filesystem to hide the path at runtime. The key’s layer is untouched, still in the image, still pushed to the registry, and still extractable by anybody who can pull the image.

Three places to look

A secret gets into an image by one of three routes, and each is found differently. Check all three — finding one does not mean there is only one.

1. The image configuration

ENV values are stored in the image config and are visible to anyone who can pull, without extracting anything:

$ docker image inspect example.com/api:2.3.0 --format '{{range .Config.Env}}{{println .}}{{end}}'

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=C.UTF-8
NPM_TOKEN=REDACTED_LOOKS_LIKE_A_REAL_TOKEN
DATABASE_URL=postgres://api:REDACTED@db.example.com:5432/app

This is the worst of the three because it needs no skill to exploit and it is also applied to every container started from the image, so the value shows up again in docker inspect of the container and in /proc/1/environ.

2. The build history

$ docker history --no-trunc --format '{{.Size}}\t{{.CreatedBy}}' example.com/api:2.3.0

0B      ENV NPM_TOKEN=REDACTED_LOOKS_LIKE_A_REAL_TOKEN
41.2MB  RUN /bin/sh -c npm ci --registry https://registry.example.com # buildkit
1.6kB   COPY deploy_key /root/.ssh/id_ed25519 # buildkit
0B      RUN /bin/sh -c rm -f /root/.ssh/id_ed25519 # buildkit

Read this as a confession log. COPY deploy_key followed by a deletion is the signature of the anti-pattern at the top of this lesson, and the 1.6kB layer it created is exactly where the key still lives.

3. The layer contents

The definitive check. docker save writes the image as a tar of OCI blobs, and each blob is a layer you can unpack and search.

$ docker save example.com/api:2.3.0 -o /tmp/api-2.3.0.tar && tar -tf /tmp/api-2.3.0.tar | head

blobs/
blobs/sha256/
blobs/sha256/0f31a2c4b8e1d7a690c2f4e83b1d5c6a7e9f0182b3c4d5e6f7a8b9c0d1e2f3a4
blobs/sha256/3c9e1f7b2a8d4056c1e9f2b3a4d5e6f7089a1b2c3d4e5f60718293a4b5c6d7e8
index.json
oci-layout

Then walk the blobs and list what each contains. Layer blobs are gzipped tars; config blobs are JSON, and tar will simply fail on them, which is why the loop below tolerates that:

#!/usr/bin/env bash
# Find a filename of interest across every layer of a saved image.
set -uo pipefail

BUNDLE=/tmp/api-2.3.0.tar
WORK=$(mktemp -d)
PATTERN='id_ed25519|\.npmrc|credentials|\.pem$'

tar -xf "$BUNDLE" -C "$WORK"

find "$WORK/blobs" -type f | while read -r blob; do
  if tar -tzf "$blob" 2>/dev/null | grep -Ei "$PATTERN"; then
    printf 'above paths are in layer %s\n' "$(basename "$blob")"
  fi
done

rm -rf "$WORK"

And to pull the file back out and confirm it is the real thing:

$ tar -xzOf /tmp/work/blobs/sha256/3c9e1f7b2a8d root/.ssh/id_ed25519 | head -c 40

-----BEGIN OPENSSH PRIVATE KEY-----

Remediation, in the only order that works

1. Revoke and rotate the credential. Deploy key, token, password — invalidate it at the issuer. This is the only step that reduces risk; everything after it is cleanup.
2. Check the audit log at the issuer for use of the credential from anywhere you do not recognise, over the whole window since the image was first pushed.
3. Enumerate the exposure. Which tags contain the layer? Who pulled them? Which running containers came from them? The supply-chain part of this course covers the blast-radius search across a fleet.
4. Fix the build. Use a BuildKit secret mount or a multi-stage build so the material is never committed to a layer.
5. Rebuild and push new tags, then redeploy anything running the old ones.
6. Delete the old tags and run registry garbage collection — knowing that this is hygiene, not containment.
7. Write the detection. A secret scanner in CI, failing the build. Without it, the next one lands the same way.

The fixed build, for reference — the mounted secret is available to the RUN and is not written to any layer:

# syntax=docker/dockerfile:1
FROM debian:13-slim
RUN --mount=type=ssh \
    git clone git@example.com:org/private-lib.git /src

docker build --ssh default -t example.com/api:2.4.0 .

Knowledge check

Where next

Finding a leak is the reactive half. The next lesson is the preventive half: what actually reaches the build context, and how to make CI refuse a commit that carries a credential.