It worked for exactly fifteen minutes after every restart

Reported symptoms

The orders API runs on four instances, app-01 to app-04, behind a load balancer, and reads from appdb using a dynamic credential. The lease is short on purpose: two minutes, renewed continuously, so a credential lifted out of a process is worthless before anyone can carry it anywhere. Until this evening the arrangement had been running for eleven months without anybody thinking about it.

Release 4.9.0 rolled out between 21:20:10 and 21:23:42 UTC. The deployment gate soaked for five minutes, found all four instances healthy at 21:28:45, and marked the release complete.

• At 21:35 the first instance starts failing and by 21:41 all four are, in the same order and at the same spacing as the rollout, which nobody notices at the time. The failures did not start at deployment time, which is used within a minute to rule the release out.
• The error is a PostgreSQL fatal message about a role that does not exist. The database team is pulled in.
• appdb is healthy on every measure. Administrative connections work, the connection count is low, replication is current, and its log carries nothing but the rejected logins.
• The secret manager is unsealed. A credential requested by hand during the incident works immediately, and that is taken as clearing it.
• The health endpoint is green on all four instances throughout, so the load balancer keeps sending traffic to processes that cannot serve it.

At 21:47 somebody restarts the service. Everything recovers instantly. At 22:02 it fails again. A second restart at 22:05 buys another recovery, and at 22:20 it fails again. It is the third identical interval that finally gets someone to look at a clock.

Evidence provided

$ journalctl -u orders-api.service --since '21:23' --until '21:45' --no-pager

$ bao lease lookup database/creds/app-readonly/xoHI541EXoFgKn1OTiusOdd9

expire_time     2026-08-26T21:25:43.891514518Z
issue_time      2026-08-26T21:23:43.891514368Z
renewable       true
ttl             1m59s

Read renewable true as an offer rather than a promise. The engine will extend this lease whenever it is asked. It has no way to ask on anyone’s behalf, and it will not.

$ grep -F 'database/creds/app-readonly/xoHI541EXoFgKn1OTiusOdd9' /openbao/audit/audit.log | jq -r '.request.path' | sort | uniq -c

$ grep -F 'sys/leases/renew' /openbao/audit/audit.log | jq -r '.auth.metadata.role_name' | sort | uniq -c

$ psql -U v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423 -d appdb -c "SELECT 1;"

psql: error: connection to server at "127.0.0.1", port 5432 failed: FATAL:  role "v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423" does not exist

$ psql -U bao_admin -d appdb -c "SELECT rolname, rolvaliduntil FROM pg_roles WHERE rolname LIKE 'v-token-%' ORDER BY rolvaliduntil;"

$ git diff v4.8.3..v4.9.0 -- deploy/orders-api.service src/bootstrap/database.go

$ grep -n 'max_lifetime' /etc/orders-api/pool.toml

Work the evidence before reading on

Nothing in this incident is faulty. The database is well, the secret manager is well, and the secrets engine performed the operation it guarantees, on time. The whole outage lives in the distance between an event and its symptom.

1. app-02 took its lease at 21:23:43 and it expired at 21:25:43. That instance did not fail until 21:38:45. Account for those thirteen minutes, using one number from the evidence.
2. Every restart restores the service and it fails again fifteen minutes later, three times. Which of the two intervals in this incident is fifteen minutes long, and which is two, and which one is the cause?
3. The audit device shows one credential read and no renewals for this workload, and regular renewals for two others against the same role. Say what that pair of observations proves and, more usefully, what it rules out.
4. The health endpoint was green while every request was failing. What is it actually measuring, and what would it have had to do instead for the deployment gate to have caught this at 21:26?

Before continuing: name the component that was supposed to keep this credential alive, and say what shipped in its place.

Root cause

The renewal was deleted, and nothing replaced it

Release 4.9.0 removed the agent from the unit and moved the credential fetch into the application, calling the client library once during start-up. The change was reviewed as a dependency clean-up, which is a fair description of the diff and a poor description of its effect: the agent was not a dependency, it was the component performing renewal.

The lease was 120 seconds and marked renewable. Renewable means the engine will extend the lease when asked. Nothing asked. At 21:25:43 the lease reached its expire time, the engine ran the revocation statements configured on the role, the generated database role was dropped, and the username ceased to exist. Every part of that is the system working correctly, and the audit device records it as an uneventful lease expiry rather than as anything resembling an error.

The pool kept serving with a credential that no longer existed

PostgreSQL authenticates a connection when the connection is opened. It does not reauthenticate on each statement, and dropping a role does not close sessions that were already established under it. The pool had opened its connections at 21:23:45 with a credential that was valid at that moment, and it went on using them.

So for thirteen minutes the service was, from the outside, completely healthy while holding a credential that had been revoked. The failure arrived when the pool started retiring connections at their fifteen minute maximum lifetime and could not replace them. That is the source of the interval, and it is why restarting appeared to fix the problem: a restart obtains a fresh credential, opens a fresh pool, and starts the same fifteen minute countdown again.

The health check could not fail

The endpoint wired into the load balancer and into the deployment gate returns the state of the connection pool object. The pool object was fine. It existed, it had a configuration, and it reported a count. It never attempted to obtain a connection, so it never discovered that it could not.

That is what allowed the rollout to complete: the gate soaked for five minutes, comfortably spanning 21:25:43, and asked four instances a question none of them could answer wrongly. It is also why no instance removed itself from rotation once the failures began, so all four kept receiving traffic they could not serve for fifty-seven minutes.

Resolution

1. Stop restarting the service. Each restart buys fifteen minutes, throws away every in-flight request, and resets the only pattern capable of identifying the fault. Write the four times down instead: process start, first failure, restart, next failure.
2. Refuse the lease extension before anyone builds it. Raising the lease from two minutes to a day adds no renewal, relocates the identical failure to the same time tomorrow, and multiplies the window in which a stolen credential is useful by several hundred. Refuse the hand-made role for the same reason: recreating the generated username in PostgreSQL with a chosen password restores service in seconds and leaves an account that nothing rotates, nothing revokes and no access review can see.
3. Confirm the diagnosis from the audit device rather than from the application logs. Filter for the lease identifier and establish that there is a credential read and no renewal request anywhere in the lease lifetime.
4. Run the same filter against a workload that is working. Renewals arriving every sixty seconds for two other services against the same role are what turn an absence into evidence, because they prove renewal is available and functioning on this cluster.
5. Roll back to 4.8.3, which runs the agent. Rolling back restores renewal, it is a change already proven in this environment, and it is far faster and far safer than writing a renewal loop under pressure at half past ten at night.
6. Confirm recovery over an interval longer than the pool maximum connection lifetime before standing down. Anything under fifteen minutes of stability is indistinguishable from the three restarts that preceded it.
7. Remove the health endpoint from the rollout gate until it executes a query. Leaving it wired in means the next release carrying this defect will also soak cleanly and also be marked successful.
8. Record the interval between the lease expiry at 21:25:43 and the failure of that instance at 21:38:45. That gap is the finding, and it belongs to the pool configuration rather than to anyone's judgement during the incident.

Verification

1. Query the database activity view for the connections belonging to the service and record the generated role name they are authenticating as. The database is the party that accepts or refuses the credential, so it is the only witness whose answer settles anything.
2. Repeat that query twenty minutes later and confirm connections are still being established. Surviving an interval longer than the pool maximum connection lifetime is the only observation that distinguishes a repair from a restart.
3. Read the audit device for renewal requests attributed to the workload role and confirm they arrive on a regular cadence at or before half the lease duration. Renewal configured and renewal running are different claims, and only the second one survives the next sixty seconds.
4. Confirm the role in the database catalogue carries a validity horizon in the future, and watch that horizon move forward once. A horizon that never advances is a renewal that is being requested and refused, which is a different fault with an identical symptom.
5. Confirm exactly one active lease exists per instance. Four instances restarted three times under pressure leave leases nobody is renewing, and each of those will expire quietly tonight whether or not anyone is watching.
6. Stop the database in a staging environment and confirm the health endpoint now reports unhealthy within its check interval. Until that has been demonstrated once, the rollout gate and the load balancer are both asking a question that cannot be answered wrongly.
7. Confirm the customer facing error rate has been flat for longer than one pool lifetime, and reconcile the order counts for the incident window so that anything lost between 21:38 and 22:38 is known rather than assumed.

Prevention

• Renew at half the lease. A renewer attempts renewal at or before 50 per cent of the lease duration, which on a 120 second lease means every 60 seconds, and it escalates rather than continuing quietly after two consecutive failures. Whether that runs in an agent or in the application matters far less than whether it runs at all.
• Alert on the absence of renewal, not on the failure it eventually causes. If a lease marked renewable comes within 30 seconds of its expire time with nothing recorded against its renew path, page. On this configuration that fires roughly 90 seconds before a single user notices anything.
• Make the health check do the work it claims to check. An endpoint that reports the state of a pool object reports that the object exists. One that executes a trivial query fails when the database refuses the credential, which is the entire reason it is wired into a load balancer and a deployment gate.
• Bound the detection delay with the pool. Setting the maximum connection lifetime below the credential lifetime forces reauthentication once per credential period, so a broken renewer becomes visible within one lease instead of thirteen minutes later. It costs a few extra connections a minute.
• Treat a change to how a workload gets a credential as a change to the credential. Removing an agent is not a dependency clean-up. It relocates the renewal obligation into the application, and the review should refuse the change until somebody can name what renews now.
• Drill it quarterly. Block renewal for one staging workload without warning. Require the failure to be visible within 60 seconds and the instance out of rotation within 90, and treat a missed drill as a release blocker for that service.
• Borrow the instrumentation from the platform courses. Observability for Production Sysadmins - Part XI (Blackbox) covers probing a service the way a client does rather than the way it describes itself, and Part XVIII (AlertingRules) covers expressing an alert on something that failed to happen, which is the harder half of this. Linux for Production Sysadmins - Part LXXII (Secrets) covers running a credential agent alongside a unit and the ordering that keeps the two in step.