Runbook: Respond to a leaked secret

A credential has escaped. It is in a commit, or a run log, or a screenshot in a ticket, or a --diff output somebody pasted into chat.

The order of the response is the whole runbook, and it is the opposite of what feels urgent.

When to use this runbook

• A credential was committed in plaintext.
• A secret appeared in a run log, a --diff output, or a CI artefact.
• A vault password was shared over an insecure channel.
• A screenshot or paste containing a credential left the team.
• A departing operator had credentials that were never rotated.

Blast radius

Everything the credential grants access to, for the entire period from first exposure to revocation. That period is the number the incident is measured by, and every step in this runbook is arranged to shorten it.

If the leaked secret is the vault password, the blast radius is every secret the vault contains - because whoever had the password had the plaintext of all of them. That case fans out into a rotation of every credential in the vault, each through its own owner’s process.

Step 1: Declare, and bound the window

cd /srv/automation/repo

# When was the value introduced, and by whom?
git log --all --oneline -S'REPLACE_ME_LEAKED_VALUE' -- . | tail -5
git log --all --format='%H %ci %an %s' -S'REPLACE_ME_LEAKED_VALUE' | tail -3

# Which refs currently contain it?
git grep -l 'REPLACE_ME_LEAKED_VALUE' $(git rev-list --all) 2>/dev/null | head

git log -S searches for commits that changed the number of occurrences of a string - it finds the commit that introduced the value even if the file has since been deleted.

Record: first exposure timestamp, who committed it, whether the repository is public, whether it is mirrored, and how many people have clone access. Those five facts determine everything about the urgency.

Step 2: Enumerate the consumers - before touching anything

Rotation breaks every consumer that still holds the old value. Find them first, or the rotation becomes an outage.

# In the repository
grep -rn 'db_password\|DB_PASSWORD' --include='*.yml' --include='*.j2' \
--include='*.env' . --exclude-dir=.git

# On the controllers
sudo grep -rl 'REPLACE_ME_LEAKED_VALUE' /home/ansible /srv/automation 2>/dev/null

# In CI
echo 'Pipeline variables, deploy keys, cached artefacts'

# On managed hosts - the rendered config that uses it
ansible all -b -m shell \
-a 'grep -rl "REPLACE_ME_LEAKED_VALUE" /etc /opt 2>/dev/null | head' -o

Write the list down. Every entry needs to be updated in Step 4 and verified in Step 5, and the one you forget is the one that produces the outage at revocation.

Step 3: ROTATE - issue a new credential

At the source system, through its owner’s process.

# Database credential - through the database owner
echo "New password generated by the DBA and delivered via the password manager"

# API token - through the provider
echo "New token issued in the provider console; scope reviewed at the same time"

# SSH key - see the SSH credential rotation runbook, which handles the fleet
ssh-keygen -t ed25519 -f ./id_ed25519_new -C 'reissued 2026-08-11' -N ''

# Vault password - see the vault credential rotation runbook

Two things worth doing while you are here, because you will not get another convenient moment:

• Review the scope. A leaked credential that turned out to have far more access than it needed is a second finding. Issue the replacement with the narrowest scope that works.
• Put the new value in the password manager first, before using it anywhere. A replacement credential that exists only in one file has simply moved the single point of failure.

Step 4: Distribute, and confirm each consumer

Work down the list from Step 2. For each consumer, install the new credential and confirm it works before moving on.

# Encrypt the new value under the vault
ansible-vault encrypt_string --vault-id prod@/home/ansible/.vault-pass \
--encrypt-vault-id prod 'REPLACE_ME_NEW_VALUE' --name 'db_password' \
>> inventories/production/group_vars/db/vault.yml

ansible-vault view --vault-id prod@/home/ansible/.vault-pass \
inventories/production/group_vars/db/vault.yml | head -3

# Against the source system, using the NEW value only
PGPASSWORD='REPLACE_ME_NEW_VALUE' psql -h db01.example.com -U app -c 'SELECT 1'

# From each consumer
ansible-playbook -i inventories/production site.yml \
--limit app --tags config --check --diff

Both credentials are valid at this point. That overlap is deliberate: it is what makes Step 4 reversible and Step 5 safe. It is also the period during which the exposure is still open, so do not linger in it.

Step 5: REVOKE - this is the step that ends the exposure

# Database: drop or change the old password so it cannot authenticate
echo "DBA revokes the old credential at the database"

# API token: revoke in the provider console; note the revocation timestamp

# SSH key: remove from authorized_keys fleet-wide - the SSH rotation runbook

# Record the exact revocation time. It is the end of the exposure window.
date -u +%Y-%m-%dT%H:%M:%SZ | tee revocation-timestamp.txt

Step 6: Prove the old credential is dead

A verification that cannot fail is not a verification. This one can, and must be run.

if PGPASSWORD='REPLACE_ME_LEAKED_VALUE' \
   psql -h db01.example.com -U app -c 'SELECT 1' >/dev/null 2>&1; then
echo 'STILL VALID - THE EXPOSURE IS NOT CLOSED'
exit 1
else
echo 'revoked OK'
fi

Then confirm at the source system’s own audit trail, not just from the client side:

echo "Provider console: token status shows Revoked, with a timestamp"
echo "Database: the old role or password no longer exists"
echo "Record both, with timestamps, in the incident notes"

“Superseded” is not “revoked”. A token that has been replaced but not invalidated still works, and a rotation that only issued a new credential has not closed anything.

Step 7: Hunt for the other copies

The commit is rarely the only place. Secrets propagate.

VALUE='REPLACE_ME_LEAKED_VALUE'

# Controller run logs
sudo grep -rl "$VALUE" /var/log/ansible/ 2>/dev/null

# Fact cache
sudo grep -rl "$VALUE" ~/.ansible/facts_cache /var/cache/ansible 2>/dev/null

# Backup files that template/copy left on managed hosts
ansible all -b -m shell \
-a "grep -rl '$VALUE' /etc --include='*.conf.*' 2>/dev/null | head" -o

# Shell history on the controller
sudo grep -l "$VALUE" /home/*/.bash_history /root/.bash_history 2>/dev/null

# CI artefacts and job logs
echo 'Pipeline job logs, cached artefacts, container image layers'

# Tickets, chat, wiki, screenshots
echo 'Search the ticket system and chat archive for the value'

Step 8: Audit the exposure window

# The source system's authentication log for the window
echo 'Database: connection log for the app user between first exposure and revocation'
echo 'Provider: API access log for the token, same window'

# Unexpected source addresses are the finding
echo 'Compare source addresses against the known consumers from Step 2'

Look for use from an address that is not one of your consumers. Absence of evidence is weak - many systems do not log successful authentication in enough detail - and that limitation belongs in the incident notes rather than being read as “no misuse occurred”.

Step 9: Only now, clean the repository

The credential is already dead. This step is hygiene: it stops the next person finding a value and wasting time on it, and it removes an obvious artefact from the repository.

git log --all --oneline -S'REPLACE_ME_LEAKED_VALUE' | wc -l
git for-each-ref --format='%(refname)' | wc -l
echo 'Count the forks, mirrors and long-lived clones - each needs re-cloning'

# git-filter-repo is the maintained tool; filter-branch is deprecated and
# is documented upstream as unsafe and extremely slow for this purpose.
git clone --mirror https://git.example.com/infra/automation.git automation-mirror
cd automation-mirror

printf 'REPLACE_ME_LEAKED_VALUE==>REDACTED\n' > /tmp/replacements.txt
git filter-repo --replace-text /tmp/replacements.txt

# Review before pushing - this rewrites every affected commit hash
git log --oneline -5

If the repository is public, also request that the hosting provider purge cached views of the affected commits. Do not treat that request as having closed anything - the value was already scraped.

Step 10: Fix the mechanism

The credential leaked through a path. Close it.

# .pre-commit-config.yaml
repos:
- repo: https://github.com/gitleaks/gitleaks
  rev: v8.28.0
  hooks:
    - id: gitleaks

- name: Configure the application database connection
ansible.builtin.template:
  src: app.conf.j2
  dest: /etc/app/app.conf
  owner: app
  group: app
  mode: '0640'
no_log: true
notify: Restart app

Other mechanism fixes, in rough order of value:

• Move the secret out of the repository entirely, to an external secret manager looked up at run time. Then a repository leak is not a credential leak.
• CI secret scanning as a merge gate, so the pre-commit hook is not the only barrier.
• Review who can read the repository. A credential in a vault- encrypted file is only as protected as the vault password’s distribution.
• Stop using --diff with templates that render secrets, or set no_log on those tasks.

Step 11: Write the timeline

The number that matters:

First exposure       2026-07-02 14:11 (commit 8b2e1a4, pushed to main)
Discovered           2026-08-11 09:40
Rotation issued      2026-08-11 10:25
Consumers confirmed  2026-08-11 11:50
REVOKED              2026-08-11 11:58
History rewritten    2026-08-12 16:00 (scheduled, coordinated)

Exposure window: 40 days, 21 hours.
Time from discovery to revocation: 2 hours 18 minutes.

Both durations are findings. The first measures detection; the second measures response. They have different fixes - scanning gates for the first, rehearsal and a written consumer list for the second - and recording them separately is what makes each improvable.

Common patterns

Escalation

Escalate when:

• On discovery. Immediately. This is an incident.
• Rotation requires a credential owner you are not.
• The secret grants access to a third-party or customer system - notification obligations have deadlines.
• The exposure window cannot be bounded, or the repository is public or mirrored outside your control.
• A consumer cannot be updated before revocation. Extending the exposure to avoid an outage is the service owner’s decision, and the answer is usually the outage.