← All runbooks in Secrets, PKI & Certificates

high risksecurity relevant~75 min

Runbook: Respond to a Compromised API Key

1 · Prerequisites

Confirm every item is in place before any state change.

Authority at the issuing provider to disable or delete this credential, or an on-call owner who holds it
Read access to the issuing provider audit trail covering the whole period from suspected exposure to revocation
Read access to the secret manager audit device, so that reads of the stored copy can be attributed
The identity and owner of every consumer that reads this credential, because revocation stops all of them at once
A means of restarting those consumers, since many read their credentials once at start-up and never again
An incident record open before the first action, with a place to record the exposure and revocation timestamps

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Identify the credential by its own identifier, not by its purpose. Providers issue several keys per account and the one described in the report is often not the one in the incident. Record the key identifier, the account it belongs to and the date it was issued.
· Establish the earliest moment it could have been exposed, and say how you know. A commit push time, a log line, a laptop theft report or a supplier notification each give a different starting point. Record the basis alongside the timestamp.
· List every consumer of this credential before revoking. Search the secret manager for readers of the path, and search deployment manifests for the environment variable name. Revocation is an outage for every one of them.
· Read the policy that governs the credential rather than describing it from memory. A key that everybody calls read-only frequently is not, and the policy document settles it in seconds.
· Confirm the secret manager is available and its audit device is active. bao audit list -detailed shows the configured devices. An incident is a poor time to discover that nothing was recording reads.
· Confirm you can write the replacement to the path the consumers already read. If the replacement lands somewhere else, every consumer needs a code change during an incident.
· Decide in advance whether to disable or delete. Where the provider offers both, disable stops use immediately while keeping the identifier present for the audit work that follows. Delete once the audit is complete.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Disable or revoke the credential at the issuing provider before anything else. This is the only action that removes access. Everything else in this procedure describes the damage or prevents the next one, and none of it shortens the attacker window by a second.
2Record the revocation timestamp immediately. Every audit query in this runbook is bounded by it, and reconstructing it afterwards from a console screenshot is how timelines become approximate.
3Prove the credential is dead from a machine that took no part in the revocation. Present it once and require a refusal. Consumers that cached it at start-up will still appear healthy, so do not use their health as evidence.
4Establish what the credential was actually authorised to do, from the policy document. Scope determines the whole shape of the incident: a key that could only read one path is a contained event, and a key that could write or issue is a much larger one.
5Read the issuing provider audit trail across the exposure window. Look for callers you cannot attribute rather than for operations that look malicious. A stolen key is usually used to perform ordinary reads from an unexpected origin.
6Read the secret manager audit trail for the same window. The audit device records who read the path holding the credential, from where, and whether the request was permitted. Denied requests carry a policy result of not allowed and are as informative as the successful ones.
7Do not search either audit trail for the secret value. The secret manager does not record values, and tokens appear only as HMAC digests. Search by path, by accessor and by source address instead.
8Establish how the credential reached the wrong hands and who else holds a copy. Trace every path the value travelled: the secret store, the deployment manifest, the container image, the developer laptop, the ticket, the chat message and the supplier that was sent it.
9Issue the replacement and write it to the path the consumers already read. bao kv put kv/app/config writes a new version under the same path, which is what lets consumers pick it up without a code change.
10Restart or reload each consumer and confirm it has taken the new value. A process that read the credential once at start-up will keep failing silently against the revoked one until it is restarted.
11Remove the copies found while tracing distribution, in order of exposure. Chat messages and tickets first, because they reach the widest audience, then images and manifests, then the developer copies.
12Replace the standing credential with one that expires. A dynamically issued credential with a short lease turns this class of incident into a lease revocation, and bao lease revoke removes the underlying account rather than merely deauthorising it.
13Add the credential to the inventory with an owner, an expiry and a rotation cadence. A credential nobody owns cannot be rotated on schedule and will reappear in a future incident.
14Write the timeline before closing. Exposure basis, exposure time, detection time, revocation time, the audit findings from both trails, the copies removed, and the residual risk you accepted.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The issuing provider reports the credential as disabled, revoked or absent, read from its own interface after the change rather than inferred from the request that made it.
✓A single authenticated call with the old key from a host outside the revocation path is refused, and the refusal is recorded with its timestamp.
✓Every consumer is healthy on the replacement, observed from its own health endpoint or its own request log rather than from the deployment tool.
✓The issuing provider audit trail shows no successful call with the old key after the recorded revocation timestamp.
✓The secret manager audit trail shows reads of the credential path only from the accessors belonging to the known consumers, across the whole window.
✓A read of the credential path by an identity that should not have it is refused, confirming the policy is what the document says it is.
✓A search of every removal target identified while tracing distribution, including rebuilt images and updated manifests, finds no remaining copy of the old value.
✓The inventory entry exists and names an owner, an expiry date and a rotation cadence.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶The revocation does not roll back. If a consumer cannot use the replacement, repair the consumer or issue a second replacement; never re-enable the compromised key to restore service.
↶If the replacement was written to the wrong path, write it again to the correct one. The previous version remains available under the version history of the path, so nothing is lost by correcting it.
↶If a consumer fails after the restart for an unrelated reason, roll back that consumer to its previous release, not to the previous credential.
↶If revoking a lease removed more dynamic credentials than intended, reissue them from the same role. They are meant to be disposable, and reissuing is cheaper than diagnosing what depended on which lease.
↶If the secret manager becomes sealed during the work, the correct response is to unseal with the key threshold, not to bypass the manager by distributing the credential by hand.
↶If deleting rather than disabling the key breaks the audit work, continue the audit from the exported records already captured, and note in the record that the identifier no longer resolves at the provider.

6 · Escalation

When the runbook isn't enough, contact:

· The audit trail shows successful calls you cannot attribute to your own systems: this is an intrusion. Hand it to the security incident responder immediately and continue this procedure alongside.
· The credential could write, deploy, publish or issue further credentials: every action it could have taken is in scope, and the owner of the affected system has to be in the response before recovery decisions are made.
· The credential belonged to a third party or was issued to you by a supplier: only they can revoke it, and the relationship owner needs to be engaged inside the same hour with a stated deadline.
· Revocation would take down a service with a contractual availability commitment: the decision to accept the outage belongs to the service owner, and it is made in minutes, not deferred.
· The secret manager audit device was not enabled during the window: you cannot establish who read the credential. Escalate that gap as a finding in its own right rather than reporting the incident as fully understood.
· The same credential is found to be in use by a team you did not know about: scope has changed and the incident commander needs to reset the consumer list before the replacement is issued.

An API key is a bearer credential: whoever holds the bytes is the account. There is no second factor to fall back on, no session to invalidate and no device binding to appeal to. That makes the response unusually simple to state and unusually easy to get out of order. Revoke at the issuer, then find out what happened.

Two separate audit trails matter here and teams routinely conflate them. The issuing provider records what the key did. The secret manager records who read the copy you stored. The first answers whether you have an intrusion; the second answers how the value escaped, and it is the one most organisations have never queried under pressure.

Keep one further distinction in view. Revocation is instantaneous and total at the issuer, but a consumer that read the credential once at start-up will continue to present the dead value and will keep failing until it is restarted. A quiet dashboard after revocation proves nothing, in either direction.

When this runbook applies, and when it does not

It applies when an API key, personal access token, service account key or similar bearer credential is believed to be held by someone who should not hold it, whether through a leak, a lost device, a supplier incident or an unexplained pattern of use.

It does not apply when:

The credential is short-lived and has already expired. Confirm the expiry first. Revoking an expired credential produces paperwork and no security benefit, and the real finding is the print path or the leak path that exposed it.
The credential is a certificate and private key pair. That is a key replacement and a revocation decision with its own procedure, and the effectiveness of revocation depends on what actually checks it.
The credential is an SSH key. SSH revocation runs through a revocation list and an authorised keys sweep, which is a different set of mechanisms entirely.
What leaked was an identifier rather than a secret. Key identifiers, account identifiers and accessor values are designed to appear in logs. Confirm which half of the pair you are holding before you take an outage.

Blast radius

Action	Reversible?	What it costs if wrong
Disabling the key at the provider	Usually yes, where the provider supports re-enabling	Every consumer fails at once. That is the intent, and it is why the consumer list comes first.
Deleting the key at the provider	No	The identifier may stop resolving in the provider console while you still need it for the audit work.
Writing a new version to the credential path	Yes, the previous version remains under the version history	Consumers that pinned a specific version keep reading the compromised value.
Destroying the version metadata for that path	No	The version history and the record of who read what are gone permanently.
Revoking a lease	No	The underlying account is removed, not merely deauthorised, so anything still using it fails immediately.
Restarting every consumer at once	Yes	A simultaneous restart is a self-inflicted availability event. Stagger it unless the credential is actively being abused.

Step 1 - Revoke at the issuer and timestamp it

Service impact possibleon the responder workstation, immediately after the provider action— Record the revocation moment that bounds every audit query below.

$ INCIDENT=/run/incident
install -d -m 0700 "$INCIDENT"

# Disable or delete the credential at the provider that issued it,
# using that provider's own interface. Prefer disable where both exist:
# it stops use now and keeps the identifier resolvable for the audit.

date -u +%FT%TZ | tee "$INCIDENT/revoked-at"

# Record the identifier itself, so later queries name the same thing.
printf '%s\n' 'key identifier recorded in the incident record' \
| tee -a "$INCIDENT/notes"

Nothing else in this runbook competes with this step. If the consumer list is incomplete, revoke anyway and rebuild the list from the failures that follow. An incomplete list costs you a noisy fifteen minutes; a deferred revocation costs you the whole exposure window.

Step 2 - Prove the credential is dead

Read-only / Safefrom a host that took no part in the revocation— Present the old key once and require a refusal.

$ INCIDENT=/run/incident

# The old value stays in a curl config file, so it is never visible in
# the process list of the machine running the probe.
curl --fail --silent --show-error \
   --config "$INCIDENT/old-key.curlrc" \
   https://api.example.com/v1/account
echo "probe exit status: $?"

# A healthy consumer is not evidence: it may have cached the value.
date -u +%FT%TZ | tee "$INCIDENT/probe-refused-at"

Run the probe from a machine with no ambient credentials of its own. A laptop already authenticated to the same provider can make a dead key look alive, because the client picks up the working credential from the environment instead of the one you handed it.

Step 3 - Establish the real scope from policy

Read-only / Safeagainst the secret manager that stores the credential— Read the policy governing the credential path rather than describing it from memory.

$ # The audit device must be active or none of the next steps are possible.
bao audit list -detailed

# Read the path as a consumer identity would, to confirm the boundary.
bao kv get kv/app/config

# The same identity attempting a neighbouring path should be refused.
bao kv get kv/app/other

# Key-value version 2 splits data from metadata, so read on the data
# path does not grant list. Confirm which the policy actually allows.
bao kv list kv/app

A denied request returns Code: 403 with permission denied, and the path in the error tells you which of the two namespaces was consulted: a list request resolves against the metadata path rather than the data path. That distinction is the most common cause of a policy that appears broader or narrower than it is, and settling it now prevents an over-broad replacement policy being written under pressure.

Step 4 - Read both audit trails across the window

Read-only / Safeagainst the secret manager audit log— Attribute every read of the credential path inside the exposure window.

$ INCIDENT=/run/incident
AUDIT=/openbao/audit/audit.log
FROM=$(cat "$INCIDENT/exposed-at")
TO=$(cat "$INCIDENT/revoked-at")

# Who read the credential path, from where, and when.
jq -r --arg from "$FROM" --arg to "$TO" '
  select(.time >= $from and .time <= $to)
| select(.request.path == "kv/data/app/config")
| [.time, .request.operation, .request.remote_address, .auth.accessor]
| @tsv' "$AUDIT" | tee "$INCIDENT/credential-reads.tsv"

# Refused attempts are as informative as the permitted ones.
jq -r 'select(.auth.policy_results.allowed == false)
| [.time, .request.path, .request.remote_address]
| @tsv' "$AUDIT" | tail -50

Do not search either audit trail for the secret itself. The audit device records the request and the outcome, not the value, and tokens appear only as HMAC digests, so a search for the credential returns nothing whether or not it was read. Attribution runs on the accessor, the path and the source address. Compare the accessor list against your known consumers, and treat any accessor you cannot name as the finding.

Step 5 - Trace the copies and repair the distribution path

Read-only / Safeacross deployment manifests and image definitions— Find every place the value travelled other than the secret manager.

$ REPO=/srv/work/platform
cd "$REPO"

# Names, not values. The variable name is what leaks the location.
grep -rIn --include='*.yaml' --include='*.yml' --include='*.tf' \
   -e 'API_KEY' -e 'apiKey' -e 'api_key' . | head -40

# Build arguments and environment lines persist in image history, so a
# later removal instruction does not take the value out of the layer.
grep -rIn -e 'ARG ' -e 'ENV ' --include='Dockerfile*' . | head -20

# Anything committed alongside a manifest is a copy you have not counted.
git log --all --full-history --diff-filter=A --format='%H %cI %s' \
  -- '*.env' '*.tfvars' | head -20

A credential in a container image is not removed by a later instruction that deletes the file: removals are recorded as whiteout markers and the earlier layer still ships the plaintext, so the image has to be rebuilt and republished. A credential in Terraform configuration reaches state and plan files, and marking the variable sensitive redacts the output without keeping the value out of state. Each of those is a separate copy with a separate removal action.

Step 6 - Replace the standing credential with one that expires

Configuration changeagainst the secret manager— Write the replacement to the existing path, then show the dynamic alternative.

$ INCIDENT=/run/incident

# A new version under the same path lets every consumer pick the
# replacement up without a code change.
bao kv put kv/app/config api_endpoint=https://api.example.com

# For a credential the manager can issue itself, the lease is the
# control. Look it up before revoking so the window is on record.
LEASE=$(cat "$INCIDENT/lease-id")
bao lease lookup "$LEASE"
bao lease revoke "$LEASE"

Revoking a lease removes the underlying account rather than merely deauthorising the credential, so a database role issued that way ceases to exist and any further connection attempt is refused by the database itself rather than by a policy layer. That is the difference this step is buying: a credential that expires on its own turns the next incident from a revocation exercise into a lease that had already gone.

Common pitfalls

Symptom	Cause	Action
The audit log shows nothing for the credential	The search used the secret value; the device records requests and HMAC digests, never values	Search by path, accessor and source address instead
The consumer still works after revocation	It read the credential once at start-up and cached it	Restart the consumer, then re-probe from outside
A read policy was granted but listing still fails	Version 2 of the key-value store resolves list against the metadata path, not the data path	Grant list on the metadata path deliberately, or stop listing
Everything is refused with a 503	The secret manager is sealed and is refusing all requests	Unseal with the key threshold and retry; do not route around the manager
No audit device was configured	In this release audit devices cannot be enabled through the API, so an attempt to add one during the incident fails	Add the audit stanza to the configuration file and restart the service
The replacement key leaks again a month later	The distribution path was never repaired, only the value was changed	Work Step 5 to completion and move the consumer to a dynamic credential

Verification

Read the issuing provider’s own interface and confirm the credential is disabled or absent, then probe once from an unrelated host and record the refusal with its timestamp. Confirm each consumer is healthy on the replacement from its own health endpoint or request log rather than from the tool that deployed it. Query the provider audit trail and confirm no successful call with the old key after the revocation timestamp. Query the secret manager audit trail and confirm that reads of the credential path across the window come only from accessors you can name. Attempt a read of that path as an identity that should not have it and confirm the refusal. Search the removal targets from Step 5, including rebuilt images and updated manifests, for any surviving copy of the old value. Finally, confirm the inventory entry exists and names an owner, an expiry and a rotation cadence.

Rollback

Revocation is final and re-enabling the compromised key to restore service is not an option at any point. If the replacement landed at the wrong path, write it again at the correct one; the previous version remains in the version history, so correcting the mistake loses nothing. If a consumer fails after its restart for an unrelated reason, roll that consumer back to its previous release rather than to the previous credential. If a lease revocation removed more dynamic credentials than intended, reissue them from the same role, since disposability is the whole point of issuing them that way. If the secret manager seals during the work, unseal it with the key threshold rather than distributing the replacement by hand. If deleting rather than disabling the key has broken the audit work, continue from the records already exported and note in the incident record that the identifier no longer resolves.