A secret is exposed: the first hour

A secret is exposed from the moment a copy of it exists somewhere you do not control. That moment is not when the scanner alerts, not when the pull request merges, and not when somebody notices. The first hour decides whether the exposure ends as a short incident note or as a breach notification, and most teams get the ordering wrong in exactly the same way.

Cleanup is not a response

The instinct, when a credential appears in a repository, is to make it disappear. Rewriting history feels like undoing the mistake. It is not. The value of a credential is set entirely by what the issuer will accept, and the issuer has no idea where the copies live. A database password printed into a build log authenticates just as well after you delete the log.

Git makes this concrete. Objects are addressed by the hash of their content, so rewriting history does not edit anything; it writes new commits and moves refs to point at them. The original objects are still in the object database until something prunes them, every clone and every fork taken before the rewrite still holds them, and whether your hosting platform prunes unreferenced objects, and on what schedule, is a question about that platform that you must answer before you call removal complete. In parallel the same value is sitting in CI caches, artefact stores, container image layers, log pipelines and half a dozen laptops.

So the only action that changes what an attacker can do is the one that makes the credential worthless. Everything else changes what a future reader sees.

• The clock starts at commit time. Size the exposure window from the author date of the commit that introduced the value, not from the timestamp on the alert. The gap between those two numbers is usually the most alarming figure in the whole incident.
• Assume a public exposure was read. You cannot prove that nobody fetched it, and the cost of assuming otherwise is the breach you did not respond to. Treat any push to a public namespace as disclosed.
• The value is not the only thing that leaked. The variable name, the endpoint it authenticates to, the account it belongs to and the policy attached to it are all useful to somebody who now knows exactly which door this key opens.

The ordered response

flowchart TD
    A["Exposure detected"] --> B["1. Revoke or rotate"]
    B --> C["2. Contain the exposed path"]
    C --> D["3. Determine usage"]
    D --> E["4. Determine exposure"]
    E --> F["5. Remediate storage and distribution"]
    F --> G["6. Audit for siblings"]
    G --> H["7. Prevent recurrence"]
    F --> R["Repository cleanup, optional and last"]

The order is not arbitrary and it is not a checklist you may reshuffle to suit the room. Step one removes the attacker’s capability. Step two closes the route the copy travelled, which might mean making a repository private, revoking a webhook or disabling a runner. Step three asks what the credential actually did while it was live. Step four asks who could have read it, which is a different population from step three. Step five fixes the reason a plaintext value was in a file at all. Step six looks for the siblings, because a credential in a repository is rarely alone. Step seven changes the system so the same class of mistake is caught by a machine next time. Cleanup, if you do it, hangs off step five and never blocks step one.

Revoking and rotating without causing your own outage

Revocation and rotation are not the same operation. Revocation invalidates the credential at the issuer. Rotation issues a replacement and moves consumers onto it. Which one leads depends on the shape of the credential, and getting this wrong turns a contained security event into a self-inflicted production outage.

The overlap in the second row is where discipline matters. A dual-credential window is a legitimate technique, and it is also the most common place a response quietly dies. Write the revocation time down before you start the cut-over, measure the window in minutes, and page yourself if it is still open at the deadline. A response review should never have to record that the rotation was deferred to the next maintenance window, because a credential known to be in a stranger’s hands does not get to wait for a change advisory board.

Determining usage and exposure

Two questions look similar and have different answers. What did this credential do? And who could have read it? The first sizes the technical damage. The second sizes the notification.

# Pin the facts before they move. Every later timing claim in the
# incident record is measured from these two values.
INCIDENT="/var/incidents/2026-08-26-payments-token"
mkdir -p "$INCIDENT"
date -u +%FT%TZ > "$INCIDENT/detected-at"

REPO="$HOME/src/payments-api"
git -C "$REPO" fetch --all --prune

# When did the string enter history, on any ref, and where does it
# still live at a branch tip?
git -C "$REPO" log --all --format='%H %aI %an' -S 'lab-only-not-real' \
  > "$INCIDENT/introducing-commits"
git -C "$REPO" grep -l 'lab-only-not-real' \
  $(git -C "$REPO" for-each-ref --format='%(refname)') \
  > "$INCIDENT/refs-still-carrying" || true

The pickaxe search finds commits where the number of occurrences of the string changed, which is what you want: it names the commit that introduced the value and the commit that removed it, and the difference between those two author dates is your exposure window.

Revocation itself needs an independent check, because the console that reports a credential deleted is reporting its own intent. Use the credential and observe the refusal:

# Prove the revocation landed by using the credential, not by
# re-reading the page that said it was revoked.
BAO_ADDR="https://vault.example.com:8200"
export BAO_ADDR
bao kv get kv/app/config

A revoked token returns an authorisation failure rather than data. Record that result in the incident file next to the revocation timestamp; it is the difference between believing the credential is dead and knowing it.

Usage comes from the issuer, not from the repository. A secret manager audit device is the strongest source available, because it records every request the credential made, including the ones that were refused. In OpenBao the audit record carries the token and its accessor as HMAC values rather than in clear, along with the request path, the operation, the caller address and whether the policy allowed it. That design has two consequences worth internalising: you can correlate every request made by one credential without the log itself ever containing that credential, and a run of denials from one accessor is the recognisable signature of somebody probing the edges of a stolen token’s policy.

Remediate the path, then clean the repository

Once the credential is dead, fix the thing that put a plaintext value into a tracked file. That usually means moving the value into a secret manager and giving the application an identity it can use to fetch it, or replacing the long-lived credential entirely with a federated short-lived one. Add the detection that would have caught it: a scanner in the pre-receive path or the pipeline, tuned so that a finding blocks rather than annotates.

Repository cleanup comes last and is optional. It is worth doing when the repository is public, when a compliance obligation requires removal, or when leaving the value in place invites somebody to try it against a system you have not thought of. It is not worth treating as the fix, and it should never be the reason the revocation waited.

Production discipline

1. Write the revocation timestamp before the cut-over starts. An overlap window with no recorded deadline becomes a permanent second credential nobody remembers to remove.
2. Give the responder standing authority to revoke. If revoking a production credential needs an approval that takes forty minutes, your first-hour response is a thirty-minute wait followed by a rushed thirty minutes.
3. Record the exposure window as two timestamps, not as a duration. Author date of the introducing commit and time of revocation. Durations get rounded down in retellings; timestamps do not.
4. Rotate the siblings. A credential committed by a person or a pipeline is evidence about how that person or pipeline handles credentials generally, so enumerate what else they hold.

Cross-course references

• Git, CI/CD & GitOps for Infrastructure Engineers - Part XXXV (Secrets in Git) covers why deleting a value from the current file leaves it in the history, which is the mechanism this lesson refuses to treat as a remedy.
• Linux for Production Sysadmins - Part LXXXI (Incident Command) covers the roles, severity assessment and evidence preservation that wrap around the seven steps described here.
• Observability for Production Sysadmins - Part LXXXII (Secrets and Sensitive Telemetry) covers how a monitoring pipeline becomes another uncontrolled copy of the value you are trying to invalidate.

Quiz