← All runbooks in Secrets, PKI & Certificates

medium riskservice affecting~60 min

Runbook: Rotate an Application Credential

1 · Prerequisites

Confirm every item is in place before any state change.

Authority to create and to revoke credentials in the system that issues them, not merely to read them
An audit or access record that shows when a credential was last used, because idleness has to be observed rather than assumed
A written list of every consumer of this credential, or the time to build one before starting
Write access to each place the credential is stored: the secret manager, the pipeline variable, the configuration management layer
Knowledge of the duty cycle of every consumer, including the batch jobs that run weekly or monthly
A rollback decision already agreed: how long both credentials stay valid, and who may end the window early

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Establish that you can observe use before you change anything. If nothing records which credential authenticated and when, you cannot tell whether the old one is idle, and the rotation becomes a guess with an outage attached. Some systems will not let an audit device be enabled through the API at all, so this may be a configuration change with its own restart.
· Identify the credential by a stable handle, never by its value. Most systems expose an accessor or an identifier that is safe to write in a ticket. Audit records commonly store a keyed hash of the token rather than the token, so the accessor is what lets you match a log line to the credential you are retiring.
· List every consumer and every store separately. A consumer is a process that authenticates. A store is a place the value sits at rest. One credential often has three consumers and five stores, and it is the forgotten store that reintroduces the old value after the rotation.
· Read the authorisation attached to the current credential. The replacement must be no wider. A rotation is a good moment to narrow scope, but widening it accidentally, because the new role was copied from a template, is a quiet privilege escalation.
· Confirm the lease or expiry semantics. A credential that is renewable behaves differently from one with a fixed lifetime, and a consumer holding a renewable lease can outlive the rotation window without ever re-authenticating.
· Find the slowest consumer. The rotation window has to cover at least one full duty cycle of the least frequent job. A monthly reconciliation task that authenticates once on the third of the month is the consumer that breaks in six weeks.
· Agree what a failure looks like and who calls it. Distribution is incremental, so there must be a named person who can stop the rotation and a defined signal that means stop.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Record the current credential's accessor, its authorisation, its issue time and its expiry in the change. Do not record the value. Everything that follows refers to this handle, and an audit review months later needs to be able to follow it without the secret being in the ticket.
2Capture a baseline of use from the audit record. Note how often the credential authenticates, from which sources, and at what times. Without a baseline there is no way to distinguish a consumer that has migrated from one that is simply between runs.
3Create the new credential beside the old one, with authorisation copied deliberately rather than inherited. Both are valid at this point, which is the whole point of the pattern: nothing is broken while distribution proceeds, and any single consumer can be rolled back independently.
4Test the new credential out of band before any consumer receives it. Authenticate with it yourself and perform one read the application performs. A credential that cannot do the job is far cheaper to discover now than after it has been distributed to twelve places.
5Confirm the new credential cannot do more than the old one. Attempt one operation that should be denied and check that it is refused. A rotation that silently grants list or write where the previous credential had read is a finding waiting to be raised against you.
6Distribute to one consumer first, chosen for being observable and low risk. Update its store, restart or reload it if it caches credentials at startup, and watch it authenticate with the new credential in the audit record before touching the next consumer.
7Work through the remaining consumers in order of blast radius, smallest first. Update the store, reload the consumer, confirm the new authentication appears. Resist the temptation to update every store in one pass, because a single bad value then fails everything at once and you lose the ability to attribute it.
8Watch the old credential fall idle over at least one full duty cycle. Idle means the audit record shows no authentication with that accessor, over a period long enough to include the least frequent consumer. Silence for an hour proves nothing about a weekly job.
9Chase every consumer that is still using the old credential. Each remaining authentication in the audit record has a source address and a time. Those two fields are usually enough to identify the host, and a consumer nobody can identify is itself the finding.
10Decision point: confirm idleness before revoking. Revocation is the irreversible half of this procedure. Do not proceed on a schedule, proceed on the evidence, and if a consumer cannot be identified, extend the window rather than revoking and waiting to see who complains.
11Revoke the old credential and record the timestamp. Revocation should be immediate and complete rather than a change of expiry date, because an expiry that is merely brought forward still leaves a working credential in the meantime.
12Prove the old credential now fails. Attempt to use it and confirm the refusal. A revocation nobody tested is an assumption, and the assumption is usually wrong when the credential was also cached somewhere.
13Purge the old value from every store on the inventory. A revoked credential in a pipeline variable is harmless today and confusing forever afterwards, and the next incident responder will waste an hour on it.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The audit record shows every consumer from the inventory authenticating with the new credential's accessor, and shows no authentication with the old one across a period covering the slowest consumer's duty cycle.
✓An attempt to use the old credential is refused, tested explicitly rather than inferred from the absence of complaints.
✓The new credential is refused for at least one operation that the old credential was also refused, proving the authorisation was not widened.
✓Each consumer's own health check or readiness endpoint reports success after its reload, read from the consumer rather than from the credential system.
✓Every store named in the inventory holds the new value and no store holds the old one, confirmed by reading each store rather than by recalling that it was updated.
✓The change record names the old and new accessors, the distribution order, the observed idle period and the revocation timestamp.
✓No error rate or latency change is visible in the application's own telemetry across the rotation window.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶A consumer fails immediately after receiving the new credential: restore the old value in that consumer's store and reload it. The old credential is still valid during the window, which is precisely why the window exists.
↶Several consumers fail at once: stop distributing, restore the old value everywhere it was changed, and investigate the new credential's authorisation rather than the consumers. A shared failure points at the credential, not at the hosts.
↶The new credential works for reads but fails an operation nobody tested: restore the old value for the affected consumer, widen the new authorisation deliberately, retest out of band, and resume distribution.
↶The old credential has already been revoked when a consumer surfaces: this is an outage. Issue a third credential immediately rather than trying to restore the revoked one, and treat the missed consumer as an inventory defect to be fixed afterwards.
↶The rotation is abandoned partway: leave the new credential in place and valid, restore the old value where it was changed, and record which consumers hold which. An abandoned rotation with no record is how an estate ends up with two live credentials and no owner for either.

6 · Escalation

When the runbook isn't enough, contact:

· A consumer using the old credential cannot be identified from the audit record: escalate to the platform owner before revoking, because revoking blind converts an unknown consumer into an unattributed outage.
· The credential turns out to be shared by teams outside your service: escalate to each owning team and re-plan, since the duty cycle and the rollback authority now belong to more than one group.
· The audit device is absent or incomplete and cannot be enabled inside the window: escalate to the secret manager owner. Rotating without observation is possible but it is a different, riskier plan and somebody senior should agree to it.
· The credential is used by a third party or an external partner: escalate to the relationship owner, because their deployment schedule now sets the length of your rotation window.
· The reason for the rotation is suspected exposure rather than routine hygiene: escalate to the security on-call and switch to the compromised credential procedure, which shortens the window deliberately and accepts the risk of an outage.

Rotating a credential means two credentials exist at once for a while. That overlap is not sloppiness, it is the mechanism. Issue the new credential, distribute it consumer by consumer, watch the old one stop being used, and only then take it away. Every credential rotation that caused an outage skipped the middle two steps and went straight from issue to revoke.

The step that gets skipped most is the observation. Teams distribute the new value, wait an hour, see no alerts, and revoke. An hour proves nothing about a job that runs on the first of the month, and nothing at all about the standby instance that has not restarted since March. Idleness has to be measured against the slowest consumer in the inventory, not against the patience of the person doing the rotation.

Everything below is deliberately independent of what kind of credential it is. The pattern is the same for an API key, a token, a role identifier and secret pair, or a service account key. What changes is where the value is stored and how the consumer picks up a new one, and that is exactly what the inventory in the first step is for.

When this runbook applies, and when it does not

It applies when an application credential must be replaced and the service must stay up: the credential is older than policy allows, a team member with access has left, the credential was created by hand and nobody remembers its scope, or an audit requires evidence of periodic rotation.

It does not apply when:

The credential is known to be leaked. Then the overlap window is a liability rather than a safety net. Use the compromised credential procedure, which revokes first and accepts the disruption, because every minute the old value stays valid is a minute an attacker can use it.
The system issues short-lived dynamic credentials already. If each consumer receives its own leased credential that expires on its own, there is nothing here to rotate. Fix the lease duration instead.
The credential is a database password. The pattern is the same but the connection pool changes the failure timing completely, so use the database procedure, which is written around that behaviour.
You cannot observe use at all. Then this procedure cannot reach its own verification step. Get the audit record working first, or take the decision explicitly and with the right authority.

Blast radius

Action	Reversible?	What it costs if wrong
Reading the audit record and the authorisation	Yes	Nothing
Creating the new credential	Yes	One more credential to track until cleanup
Testing the new credential out of band	Yes	Nothing, and it catches most scope mistakes
Updating one consumer’s store	Yes	That consumer only, and only until it is restored
Reloading a consumer to pick up the value	Yes	A restart, visible if the consumer is not redundant
Revoking the old credential	No	Every consumer still holding it fails at once
Revoking before observing idleness	No	An outage whose cause is a job that runs monthly

Step 1 - Inventory consumers and stores separately

# Consumers: processes that authenticate. Stores: places the value rests.
grep -rlI 'APP_API_CREDENTIAL' /etc/ /srv/config 2>/dev/null
systemctl list-units --type=service --state=running | grep -E 'app-|worker-'
crontab -l 2>/dev/null
ls -1 /etc/systemd/system/*.timer 2>/dev/null

Write the two lists in separate columns. A consumer is something that authenticates; a store is somewhere the value sits at rest. The counts are rarely equal, and the mismatch is the interesting part: a store with no consumer is a leftover that will confuse the next person, and a consumer with no known store is a value somebody pasted into an environment variable years ago.

Step 2 - Confirm you can see the credential being used

Read-only / SafeCaptured in the course lab: an audit device confirmed active— Without this, idleness cannot be measured and the rotation is a guess.

$ bao audit list -detailed

Path     Type    Description    Replication    Options
----     ----    -----------    -----------    -------
file/    file    n/a            replicated     file_path=/openbao/audit/audit.log

Check this before you create anything. In current OpenBao releases an audit device cannot be enabled through the API at all, and the refusal says so plainly: use declarative, configuration-based audit device management instead. That makes enabling one a configuration change to the server with its own restart, which is a different conversation from a credential rotation and needs to happen first.

Step 3 - Create the new credential and test it out of band

Read-only / SafeCaptured in the course lab: an application login response— The accessor is the safe handle. The token value never goes in a ticket. Response trimmed to the auth block.

$ bao write -format=json auth/approle/login role_id="$ROLE_ID" secret_id="$SECRET_ID"

"auth": {
  "client_token": "s.<REDACTED>",
  "accessor": "n5GSr33ga2aoj00X03Z3y4Qh",
  "policies": ["app-read", "default"],
  "token_policies": ["app-read", "default"],
  "metadata": {"role_name": "app-role"},
  "orphan": true,
  "lease_duration": 1200,
  "renewable": true
}

Two fields do most of the work here. The accessor is a handle you can safely write into the change record and match against audit lines later. The lease duration tells you how long a consumer can keep working without re-authenticating, which is the floor on how long your observation window has to be before silence means anything.

Step 4 - Prove the new credential is not wider than the old

Read-only / SafeCaptured in the course lab: an operation correctly refused— A rotation that quietly widens scope is a privilege escalation nobody approved.

$ bao kv put kv/app/config x=y

Code: 403. Errors:

* permission denied

Pick an operation the old credential could not perform and confirm the new one cannot either. Path shapes matter here: a store that splits data from metadata will refuse a listing even when the read succeeds, because those are different paths and a read grant does not carry a list grant. Test the exact operations the application performs rather than a convenient approximation of them.

Step 5 - Distribute one consumer at a time

# Update this consumer's store, reload it, then stop and watch.
install -o root -g app -m 640 /root/rotation/new-credential /etc/app/credential
systemctl reload app-api.service
systemctl --no-pager --lines=20 status app-api.service

# Confirm the consumer is healthy from its own endpoint, not from the issuer.
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/healthz

Start with the consumer that is easiest to observe and cheapest to roll back. Watch it authenticate with the new accessor before moving on. Updating every store in a single pass feels efficient and removes your ability to attribute a failure, which is the one thing you will want when something goes wrong at the fourth consumer.

Step 6 - Watch the old credential fall idle

Read-only / SafeCaptured in the course lab: what an audit record actually contains— Tokens are hashed, the outcome is recorded, and the secret value never appears. Shown wrapped, with the hashes abbreviated.

$ tail -n 1 /openbao/audit/audit.log

{"time":"2026-08-26T21:26:20.555636512Z","type":"response",
"auth":{"client_token":"hmac-sha256:da33377eba1c...","accessor":"hmac-sha256:5eb6ce9e...",
       "policies":["app-read","default"],"policy_results":{"allowed":false}, ...},
"request":{"operation":"read","mount_point":"kv/","mount_type":"kv",
          "path":"kv/data/app/other","remote_address":"127.0.0.1", ...},
"response":{"data":{"error":"hmac-sha256:b0a0f532..."}},
"error":"1 error occurred:
	* permission denied

"}

Every line carries a remote address, an operation and a path, which is normally enough to identify the host still using the old credential. Watch for a period that covers the slowest consumer in the inventory: if a reconciliation job runs monthly, an observation window of a week is not evidence of anything. This is the step that takes calendar time rather than effort, and shortening it is what turns rotations into incidents.

Step 7 - Revoke, then prove the refusal

# Revoke by accessor so the value never has to be handled again.
bao token revoke -accessor n5GSr33ga2aoj00X03Z3y4Qh

# Then confirm the refusal rather than assuming it.
BAO_TOKEN=old-value-under-test bao kv get kv/app/config || echo "old credential refused as expected"

Revoke rather than shortening an expiry: a credential whose expiry has been brought forward still works until then, and the window you just closed carefully reopens for the remainder. Then test the refusal explicitly, and finally remove the old value from every store on the inventory, so that the next responder does not spend an hour investigating a credential that has not worked for months.

Common pitfalls

Symptom	Cause	Action
A job fails weeks after the rotation looked clean	Its duty cycle was longer than the observation window	Issue a replacement now, and record the true duty cycle in the inventory
A consumer keeps using the old credential after its store was updated	It caches credentials at startup and was never restarted	Restart it, then confirm the new accessor appears in the audit record
Everything works until a failover	The standby instance has an older store nobody updated	Add standby and disaster recovery instances to the inventory as first-class consumers
The new credential works for reads and fails a write	Authorisation was copied narrower than intended	Widen deliberately, retest out of band, resume distribution
The audit record shows no use of either credential	The audit device is not capturing this path or is not enabled	Fix observation before continuing; do not revoke blind
The old value is still in a pipeline variable	Stores were not reconciled after revocation	Purge every store on the inventory and note it in the change
The rotation is repeated a month later from scratch	The change record captured no accessors and no inventory	Record accessors, stores and duty cycles so the next rotation is an hour

Verification

The rotation is complete when the audit record shows every consumer in the inventory authenticating with the new accessor and none with the old, across a period covering the slowest consumer’s duty cycle. An explicit attempt to use the old credential is refused, and the new credential is refused for at least one operation the old one was also refused, proving no scope was gained. Each consumer’s own health endpoint reports success after its reload. Every store named in the inventory holds the new value and none holds the old, confirmed by reading each store. The change record names both accessors, the distribution order, the observed idle period and the revocation time.

Rollback

While both credentials are valid, rollback is per consumer and costs a file and a reload: restore the old value in that consumer’s store and reload it. If several consumers fail together, stop distributing and restore everywhere you changed, then investigate the credential rather than the hosts, because a shared failure points at the authorisation. If the new credential handles reads and fails an untested operation, restore that consumer, widen the scope deliberately, retest out of band and resume. Once the old credential is revoked there is no rollback: a consumer that surfaces afterwards needs a third credential issued immediately. If the rotation is abandoned partway, leave the new credential valid, restore the old value where it was changed, and write down which consumer holds which, because an undocumented half-rotation is worse than either end state.