A lease was revoked while a running job was still using it

Reported symptoms

The reporting platform runs a consolidation job every night against appdb. It takes about an hour. It draws a read-only database credential from the secrets engine at start-up, and a renewer keeps that credential alive for the length of the run. The design is deliberate: leases here are short, so a credential that escapes is worthless within minutes.

The run started at 21:23:43 UTC. At 21:27 it alerted.

The channel filled up with a very confident, entirely wrong diagnosis:

• The job is stuck at sixty per cent. It finished its extraction phase and cannot open the connections it needs for the write-back phase.
• The failure is a PostgreSQL fatal error saying a role does not exist. Somebody dropped a user. The database team is paged.
• The database is fine by every measure anyone checks. Administrative connections work, the connection count is normal, replication is current, and its log contains nothing but the rejected logins.
• Two other services take dynamic credentials from the same database and neither has noticed anything.
• The secret manager is unsealed and answering. That is used, in the first five minutes, to rule it out.

Nobody restarts the worker. Forty minutes of the run are already spent and a restart throws them away. That decision is reasonable, and it is the reason the incident lasted fifty-one minutes instead of one.

Evidence provided

$ jq '{lease_id, lease_duration, renewable, data}' /var/log/report-worker/run-4471.json

{
"lease_id": "database/creds/app-readonly/xoHI541EXoFgKn1OTiusOdd9",
"lease_duration": 120,
"renewable": true,
"data": {
  "password": "[REDACTED]",
  "username": "v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423"
}
}

$ psql -U v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423 -d appdb -c "SELECT 1;"

psql: error: connection to server at "127.0.0.1", port 5432 failed: FATAL:  role "v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423" does not exist

Read that username rather than the word FATAL. The generated prefix says the credential came from the secrets engine, not from a human, and that single fact routes the incident correctly.

$ bao lease lookup database/creds/app-readonly/xoHI541EXoFgKn1OTiusOdd9

expire_time     2026-08-26T21:25:43.891514518Z
issue_time      2026-08-26T21:23:43.891514368Z
renewable       true
ttl             1m59s

$ psql -U v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423 -d appdb -c "SELECT current_user, now();"

                   current_user                   |              now
--------------------------------------------------+-------------------------------
v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423 | 2026-08-26 21:23:43.940523+00

$ psql -U bao_admin -d appdb -c "SELECT rolname, rolvaliduntil FROM pg_roles WHERE rolname LIKE 'v-token-app-read-%';"

                     rolname                      |     rolvaliduntil
--------------------------------------------------+------------------------
v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423 | 2026-08-26 21:25:48+00

$ psql -U bao_admin -d appdb -c "SELECT rolname FROM pg_roles WHERE rolname LIKE 'v-token-app-read-%';"

$ bao lease revoke database/creds/app-readonly/xoHI541EXoFgKn1OTiusOdd9

All revocation operations queued successfully!

$ grep -F 'sys/leases' /openbao/audit/audit.log | tail -5

Work the evidence before reading on

Nothing here is broken in the sense of being faulty. Every component did precisely what it was configured to do, and the outage is entirely in the gaps between them.

1. The catalogue query returns zero rows. Is that damage, or is it the expected result of a correctly executed operation? What would a partial result have told you instead?
2. A fresh request for the same role succeeds during the incident and the new credential connects. Given that, how much of the estate can you exonerate in one command, and what is left?
3. The database enforces its own expiry on the generated role, set at issue time. What does that tell you about who is authoritative for the lifetime of a dynamic credential, and what still happens if the secret manager becomes unreachable?
4. The revocation is at 21:26 and the first failure is at 21:27. Which of the two things that had to be true for that gap to become an outage is the one you can actually fix?

Before continuing: say what the worker should have done in the millisecond after the database refused it, and why that is a property of the client and not of the secret manager.

Root cause

Revocation worked, and worked completely

The secrets engine holds revocation statements for the role. When the lease is revoked it runs them, the generated role is dropped, and the username stops existing. Every subsequent connection attempt is refused by the database at authentication, before any query runs.

That is the contract of a dynamic credential and the reason it is worth the complexity: the credential is real, unique to one consumer, and can be withdrawn in one command with no coordination. The zero rows in the catalogue are the receipt.

Nothing in the platform could say who was holding the lease

The quarterly access audit produced a list of lease identifiers, sorted by age, headed candidates for clean-up. It could not do better, because the credential request carried no metadata: no workload name, no environment, no owner. Age was the only available signal, and age was read as evidence of abandonment.

A long-lived lease on a short default duration is in fact evidence of the opposite. It means something has been renewing it continuously, which is exactly what a live consumer looks like.

The consumer could only obtain a credential by dying

The worker reads the credential once, at start, and passes it into the connection pool as configuration. There is no path from an authentication failure back to the secrets engine. The pool treats the refusal as a transient connection error, so it retries the same dead username, forever, with backoff.

A restart would have fixed it in under a second. The team did not restart, for a perfectly sound reason, and that is the honest shape of this incident: the only recovery available was one the operators were right to resist.

Resolution

1. Rule out the fast wrong answer explicitly and say so in the channel: nobody recreates the database role by hand. It restores service in seconds and leaves a permanent unmanaged credential that no rotation, revocation or audit will ever reach. Do not restart PostgreSQL either, because nothing is wrong with it.
2. Read the username in the failure rather than the severity keyword. A generated dynamic username names the secrets engine as the owner of the credential, and it is the fastest routing signal available in the whole incident.
3. Confirm the timeline from the audit device rather than from anyone's memory: find the revocation request, its timestamp, and the accessor that issued it, and confirm it precedes the first application failure.
4. Prove the supporting systems are healthy before touching the consumer. Request a fresh credential for the same role and connect with it from a bastion host. If that succeeds, the database, the network, the engine and the role configuration are all exonerated in one step, and everything remaining is in the client.
5. Restart the worker so it fetches a new credential, and accept that the run is lost. This is the only recovery the current design offers, and delaying it does not make it cheaper.
6. Re-run the consolidation job once the worker is authenticating, and confirm the run window still fits before the downstream report deadline. If it does not, tell the report consumers now rather than at 06:00.
7. Revoke nothing else while the incident is open, including the remaining entries on the audit list. The reasoning that produced this revocation produced the rest of that list, and it has not been corrected yet.
8. Record the lease identifier, the revocation time, the first failure time and the recovery time. The interval between the revocation and the correct diagnosis is the finding, and it is a property of the error text and the inventory, not of anyone's competence.

Verification

1. Query the database activity view for the worker connections and confirm the connected username is a new generated name, different from the one in the failure. The database is the party that accepts or refuses the credential, so it is the only witness worth asking.
2. Confirm a role for that new username exists in the catalogue and carries a validity horizon in the future.
3. Watch that horizon move forward once as the renewer runs. Renewal configured and renewal working are different claims, and only the second one survives the next two minutes.
4. Check the audit device for a credential read at the restart time attributed to the worker AppRole accessor rather than an operator token, which proves the new credential arrived by the normal path and was not injected by hand under pressure.
5. Confirm exactly one active lease exists for the role. A worker restarted twice during an incident easily leaves a second lease that nobody is renewing, and it will expire quietly tonight.
6. Confirm the consolidation job completes and the downstream report lands.
7. Reconcile the row counts against the last successful run. A job that authenticates and then produces a short report has a different problem, and it will be blamed on this one.

Prevention

• Record the consumer at issue time. Attach the workload name, the environment and an owner to every credential request, so the lease inventory answers who holds this rather than how old is this. An audit that can only see age will keep producing lists of leases it wrongly believes are abandoned.
• Separate emergency revocation from clean-up revocation. Revoking a single lease believed compromised stays immediate and needs no approval; that is the entire value of the mechanism. Revoking a prefix, or anything from a list, needs a second pair of eyes, 24 hours of notice to the named owner, and no change freeze in progress.
• Make consumers re-fetch on authentication failure. At most five attempts with exponential backoff capped at 30 seconds, then a fresh request to the secrets engine. This is the control that makes every other one optional, and it costs about twenty lines.
• Alert on renewal failure, not on credential expiry. Two failed renewals within 60 seconds is roughly half a lease of warning on this configuration, and it fires before any user notices.
• Drill it quarterly. Revoke a live lease in staging without notice, require recovery within 60 seconds and no human action, and treat a failed drill as a release blocker for that service.
• Teach the username shape. Every engineer on call should be able to look at a generated dynamic username inside a database error and route the incident in ten seconds. That habit is worth more than any dashboard here.