critical riskservice affecting~90 min

Runbook: Respond to a leaked secret

1 · Prerequisites

Confirm every item is in place before any state change.

You can revoke the credential at its issuer, or you can reach someone who can, right now
You know which identity the credential belongs to and what it grants
You have access to the issuer audit log to assess use of the credential
The secret manager is reachable so a replacement can be issued and distributed
You can reach the security on-call, and the incident channel is open
You have access to the registry with permission to delete tags, if an image is involved
You can rebuild and redeploy the affected images

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Identify the issuer and the blast radius: what does this credential grant, and to what
· Confirm the credential is still live before assuming the worst is over; a revoked credential is a different, smaller incident
· Establish the earliest possible exposure time, not the time you noticed
· Determine the leak surface: version control, an image layer, container configuration, a log, or a shared screenshot
· IMG=registry.example.com/team/app:1.4.2; docker history --no-trunc "$IMG" shows build-time values captured in layer commands
· CONTAINER=app; docker inspect -f "{{range .Config.Env}}{{println .}}{{end}}" "$CONTAINER" lists environment variables visible to anyone who can reach the daemon
· Check whether the credential also exists in a CI secret store, a systemd unit, or a running container environment
· Confirm you have somewhere to record findings that is not the leaked channel itself

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1REVOKE FIRST. Invalidate the credential at its issuer before any investigation. Every minute of investigation is a minute the credential still works.
2Record the exact revocation time; it is the boundary between "could have been used" and "could not"
3Announce in the incident channel that the credential is revoked, so nobody re-adds it to unblock a failing job
4ROTATE SECOND. Issue a replacement, write it to the secret manager, and distribute it to every legitimate consumer
5Restart or redeploy the consumers so they pick up the replacement, then confirm each with a real operation, not a config file check
6ASSESS THIRD. Pull the issuer audit log for the window from earliest possible exposure to revocation, and look for use from unexpected sources
7Identify the leak surface precisely, because the remediation differs entirely between them
8If the secret is in version control: treat it as public from the moment of the commit. Remove it from the working tree, and treat history rewriting as cleanup, never as containment.
9If the secret is in an image layer: rebuild the image without it, push under a new tag, redeploy, then delete the affected tags from the registry
10If the secret was a build-time ARG: confirm with docker history --no-trunc, then rebuild using a secret mount instead of a build argument
11If the secret is in container configuration: rebuild the container from a compose file that reads the value from a secret or an env file that is not in version control
12If the secret reached a log: identify every destination that log was shipped to, including the log aggregator, its backups and its index
13Search for other copies of the same secret before closing: the same value is frequently present in more than one place
14Write the timeline: first exposure, detection, revocation, rotation complete, exposure assessment complete
15Close with a prevention action that would have caught this specific leak surface, not a general reminder to be careful

4 · Verification

Confirm the procedure actually fixed the problem.

✓The old credential FAILS against the issuer: an authenticated call using it returns 401 or the issuer equivalent, from an isolated configuration
✓The replacement credential SUCCEEDS for every legitimate consumer, evidenced by a completed operation rather than by a stored config value
✓docker history --no-trunc on the REBUILT image contains no occurrence of the secret: piping it through grep for the value returns no match
✓docker inspect -f "{{range .Config.Env}}{{println .}}{{end}}" on the running container shows no secret value in the environment
✓The affected image tags are gone from the registry: a pull of the old tag by name fails with a not-found error
✓No running container still uses an affected image: docker ps --format "{{.Names}} {{.Image}}" lists only the rebuilt tag
✓The issuer audit log for the exposure window has been reviewed end to end, and every use is attributed to a known consumer
✓A repository-wide search for the secret value returns no match in the working tree, in CI configuration, or in the compose files
✓The prevention control is in place and demonstrably fires: a deliberate test commit containing an obviously fake secret is blocked by the scanner

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Revocation is deliberately irreversible and must not be rolled back. If revocation broke a service, issue a replacement credential — never restore the leaked one.
↶If the rebuilt image is faulty, redeploy the previous KNOWN-GOOD image that does not contain the secret; never redeploy the image that carries it
↶If a rotation broke a consumer, roll that consumer forward onto the replacement credential rather than back onto the old one
↶Registry tag deletion is not reversible. Confirm the digest and the tag before deleting, and keep the rebuilt replacement pushed and verified first.
↶If a history rewrite in version control breaks collaborators, restore from the pre-rewrite backup ref. This does not affect the incident: the secret was already compromised at commit time and is already revoked.

6 · Escalation

When the runbook isn't enough, contact:

· The credential grants production data access: escalate to security on-call immediately, in parallel with revoking; do not serialise
· The audit log shows use from an unexpected source address or at an unexpected time: this is an active compromise, not a leak. Escalate and follow the compromised-host response.
· You cannot revoke the credential yourself: escalate to the issuer owner and to security on-call at once, and state the blast radius in the first message
· The image was pushed to a public registry: escalate to security and to the service owner; assume it was pulled, and assume automated scrapers found it
· The secret belongs to a third party or a customer: escalate to the service owner and to legal or compliance before any external communication
· The leak has been public for longer than the audit log retention period: escalate, since you cannot prove non-use and must plan on the assumption of use

The order, and why it is this order

Phase	Action	Why it is here
1	Revoke	Ends the exposure. Nothing else does.
2	Rotate	Restores service on a credential that was never exposed.
3	Assess	Determines what was done with it while it was live.
4	Remediate the surface	Stops the same leak recurring.
5	Record	The timeline is what the review will ask for.

Assessment last is not a lack of urgency. It is the recognition that assessment does not reduce risk and revocation does. The audit log will still be there in an hour. The window during which the credential works will not be.

Never paste the secret

While handling this incident you will be tempted to paste the value into a chat message, a ticket, or a grep command that lands in shell history. Do not. Refer to it by identity — “the deploy token for svc-deploy” — and search for it by reading a file that already contains it rather than by typing it.

Where a command below must match on a value, it reads it from a file with restricted permissions and never echoes it.

Phase 1: Revoke

Go to the issuer and invalidate the credential.

Docker Hub personal access token: Account settings → Personal access tokens → Delete. Deactivating is not enough for a leak; delete it.
Cloud provider key: disable, then delete. Disabling first gives you a moment to spot breakage, but do not stop at disabled.
Database password: change it at the database, not only in the application configuration.
Signing key: revoke and publish the revocation; anything signed after the exposure time is suspect.

Record the revocation timestamp in UTC. It is the single most important fact in the timeline.

Phase 2: Rotate

Issue the replacement, distribute it through the secret manager, and redeploy the consumers. The registry credential rotation runbook covers finding every consumer; the difference here is that there is no overlap window, because the whole point is that the old credential must not work.

Expect breakage during this phase, and prefer it to the alternative.

Phase 3: Assess exposure

Read-only / Safewhat is exposed where— Reads image and container metadata. No changes.

IMG=registry.example.com/team/app:1.4.2
CONTAINER=app

# Build-time values captured in the layer commands
docker history --no-trunc "$IMG"

# Environment as the daemon holds it - visible to anyone who can reach the daemon
docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' "$CONTAINER"

# Build arguments recorded on the image
docker inspect -f '{{json .Config.Labels}}' "$IMG"

# Which containers run an affected image?
docker ps --format '{{.Names}}	{{.Image}}'

Phase 4: Remediate by leak surface

The four surfaces need four different responses. Getting this wrong produces an incident that is closed while the secret is still readable.

Surface A: committed to version control

Treat the secret as public from the instant of the commit. Not from when the repository was made public, not from when someone noticed — from the commit.

Removing the file in a later commit does nothing: the value remains in history, in every clone, in every fork, and in the forge’s own object store. Rewriting history is worth doing as cleanup, but it is not containment and it must never delay revocation. Clones already taken keep the value regardless.

Surface B: baked into an image layer

Destructiveremove the affected tag— Deleting a registry tag is not reversible. Confirm the rebuilt replacement is pushed and deployed first.

REG=registry.example.com
REPO=team/app

# 1. Confirm the replacement exists and is what you think it is
docker pull "$REG/$REPO":1.4.3
docker inspect -f '{{index .RepoDigests 0}}' "$REG/$REPO":1.4.3

# 2. Confirm nothing still runs the affected tag
docker ps --format '{{.Names}}	{{.Image}}' | grep ':1.4.2' || echo 'none running'

# 3. Delete the affected tag at the registry.
#    Docker Hub: repository -> Tags -> delete the tag.
#    A self-hosted distribution registry needs storage.delete.enabled
#    set to true, deletion by manifest digest, and a subsequent
#    garbage collection run to remove the blobs.

Deleting a tag from a public registry does not undo the exposure. Assume it was pulled. Automated scrapers index public registries continuously, and the value must be treated as compromised whatever the tag does afterwards.

Surface C: exposed in container configuration

A value passed as -e or in a compose environment: block appears in docker inspect output, and anyone who can reach the daemon can read it. So can anything with the Docker socket mounted, which is one more reason never to mount it.

Move the value to a file the container reads at runtime, referenced by a compose secrets: entry or an env file kept out of version control, and recreate the container so the old configuration stops existing.

Surface D: written to a log

Follow the log to every destination, not just the local file: docker logs, the JSON log file under the data root, the log shipper’s own spool, the aggregator, the aggregator’s backups, and the search index. Any one of them retains the value after the others are purged.

Then fix the emitter. A log line that prints a credential prints it every time, and purging without fixing the emitter buys you until the next request.

Phase 5: Record the timeline

Five timestamps, in UTC:

Earliest possible exposure.
Detection.
Revocation.
Rotation complete.
Exposure assessment complete.

The gap between 1 and 3 is the exposure window and it is the number the review will focus on. The gap between 2 and 3 is the one you control, and this runbook exists to make it small.

Prevention that matches the surface

A general “be careful with secrets” action item changes nothing. Pick the control that would have caught this leak:

Surface	Control that would have caught it
Version control	Pre-commit and server-side secret scanning, enforced on the default branch
Image layer	Secret mounts in the build, plus a CI scan of `docker history --no-trunc`
Container configuration	Compose `secrets:` and file-based injection instead of `environment:`
Logs	A redaction filter at the emitter, and a log-content scan in the aggregator
All of them	Short-lived credentials, so a leak expires on its own

Then test the control. A scanner that is configured but does not fire is not a control. Commit an obviously fake value and confirm it is blocked.