An API starts returning 502 exactly thirty minutes after every deployment

Reported symptoms

orders-api is a twelve instance service that reads a downstream integration credential from the secret manager and refreshes it every fifteen minutes. It has run without a secrets incident for fourteen months. The secret manager is a three node OpenBao cluster at bao.example.com, and four other services read from the same mount.

The incident channel filled up faster than the facts did:

• 09:30, checkout starts returning 502. The morning peak was at 09:18, so the first message in the channel is about capacity.
• 09:34, the service is scaled from twelve instances to twenty. The error rate gets worse, which is read as evidence of a downstream bottleneck.
• 09:41, an engineer restarts one instance and it recovers instantly. The theory becomes a connection pool leak.
• 10:11, the restarted instance fails again. Somebody restarts it again. It recovers again.
• 10:22, a second engineer points out that the four other services on the same mount have not logged a single error all morning.
• 10:26, the first person writes down the two restart timestamps and subtracts them.

The scaling made it worse for a reason nobody worked out at the time: new instances do not inherit the age of old ones, but they do inherit the defect, so twenty instances failing on a thirty-minute cycle produce more errors than twelve. An engineer had already proposed a scheduled restart every twenty minutes, which would have worked, and which would have buried the incident for another fourteen months.

Evidence provided

# orders-api instance 7, application error log
2026-08-26T09:30:14.882Z level=error component=secrets op=read path=kv/data/app/config
    status=403 body="* permission denied" attempt=1
2026-08-26T09:30:14.884Z level=error component=secrets op=read path=kv/data/app/config
    status=403 body="* permission denied" attempt=2
2026-08-26T09:30:14.951Z level=error component=http route=/checkout status=502
    cause="downstream credential unavailable"

$ BAO_TOKEN=$(cat /run/orders-api/token) bao kv get -mount=kv app/config

Error reading kv/data/app/config: Error making API request.

URL: GET https://bao.example.com:8200/v1/kv/data/app/config
Code: 403. Errors:

* permission denied

$ bao write -format=json auth/approle/login role_id=@/etc/orders-api/role_id secret_id=@/run/orders-api/secret_id

"auth": {
  "client_token": "s.REDACTED",
  "accessor": "REDACTED",
  "policies": ["app-read", "default"],
  "token_policies": ["app-read", "default"],
  "metadata": {"role_name": "app-role"},
  "orphan": true,
  "lease_duration": 1200,
  "renewable": true
}

Read those two fields again, because they are the entire incident. lease_duration is 1200, which is twenty minutes. renewable is true, which is the server stating that this credential may be extended on request. The service has been receiving that answer at every startup for fourteen months and discarding both halves of it.

$ bao kv get -mount=kv app/config

That pair is the whole diagnosis. Same path, same role, same policy, same host, one token refused and one accepted. The difference between them is age.

# fleet correlation, gathered 10:40 UTC
instance   started      first 403    interval   zone   requests/min at failure
i-07       09:00:11     09:30:14     0:30:03    a      412
i-03       09:00:09     09:30:12     0:30:03    b        8
i-11       09:36:52     10:06:55     0:30:03    a      380
i-14       09:41:20     10:11:24     0:30:04    c      121
i-19       10:02:47     10:32:50     0:30:03    b       44
i-20       10:02:48     10:32:52     0:30:04    a       39

The interval is constant to within a second across three availability zones and an order of magnitude of load. Instance i-03 was serving almost no traffic and failed on the same schedule as the busiest one. Whatever this is, it is a clock, not a capacity limit.

# /etc/orders-api/secrets.yaml, unchanged since the service was written
auth:
  method: approle
  login_on_startup: true
refresh:
  downstream_credential_seconds: 900

# platform change record CHG-4471, closed
2026-08-25T14:02Z  remove scheduled 10-minute restart from orders-api
                   reason: memory leak fixed in release 3.8.0, workaround no longer required
                   risk: none, restart was cosmetic

$ bao token lookup -accessor REDACTED

Work the evidence before reading on

Nothing in this incident is subtle once you stop reading the 403 as a permissions problem. The interesting question is why a service that has worked for fourteen months stopped on a Wednesday when nothing about the service, the policy or the mount changed.

1. The login response contains two numbers that describe the credential rather than the secret. What are they, and what does the service do with either of them?
2. The failure interval is thirty minutes and three seconds, but the lease duration is twenty minutes. Reconcile those two numbers using the refresh timer in the configuration.
3. The same path answers 403 to one token and returns data to another token carrying the same policies. What does that eliminate, and how confident should you be in the elimination?
4. The change record from 25 August says the risk was none. What was the removed restart actually doing, and how would you have found that out before removing it?

Before continuing: say why this service worked yesterday, and state what the maximum uptime of any instance was before 25 August.

Root cause

The service treats a twenty-minute credential as a permanent one

The AppRole login response is unambiguous. lease_duration is 1200 seconds and renewable is true. The secrets client logs in once, at startup, stores the token, and has no timer, no renew call and no branch that reacts to a 403 by authenticating again. Twenty minutes after each process starts, the token it is holding stops existing on the server, and every subsequent read is refused. This is not a configuration error or a policy error. It is a missing feature that the login response has been describing accurately at every startup since the service was written.

The thirty-minute constant is the refresh timer, not the token

The token expires at twenty minutes, but nothing reads a secret at twenty minutes. The service refreshes its downstream credential every nine hundred seconds, so it reads at fifteen minutes, which succeeds with five minutes of token life remaining, and again at thirty minutes, which is the first read attempted after expiry. The extra three seconds are startup and scheduling jitter. That is why the interval is a constant rather than a distribution, and why load, zone and instance size have no effect on it.

A restart loop had been performing renewal by accident

Until 25 August, a scheduled restart replaced every process every ten minutes. No process ever reached fifteen minutes of uptime, so no process ever performed a second read, so the missing renewal never mattered. The change record calls the restart cosmetic and rates the risk as none, which was true of the memory leak it was written for and completely false of the authentication behaviour it had quietly become responsible for. Removing it did not break the service. It stopped concealing a defect that had been present for fourteen months.

Resolution

1. Classify the 403 before touching anything. Log in with the same AppRole from an operator host and read the same path. A read that succeeds proves the policy is intact and the credential is at fault; a read that fails proves the opposite and means restarting instances will achieve nothing. This gate costs fifteen seconds and it selects between two incompatible remediations.
2. Confirm the service is available rather than sealed. A sealed secret manager answers Code: 503 with a sealed message and never answers 403, so a 403 already rules unavailability out, but check the cluster state anyway before making changes under the assumption that the server is healthy.
3. Restore service with a rolling restart across the fleet, and record the start time of each instance. Every restarted instance now carries a known thirty-minute clock, so the restart buys you a bounded window rather than a fix, and the team should be told that explicitly.
4. Do not reinstate a scheduled restart as the remedy. It works, which is the danger: it re-hides the defect, makes availability depend on an unowned timer, and prevents any instance from ever living long enough to prove the fix later.
5. Do not extend the role token lifetime to days, and do not issue the service a root token. Both convert a credential the service failed to renew into a credential nothing can meaningfully revoke, and the second one also removes every policy boundary the mount was designed around.
6. Implement renewal in the secrets client. Renew when half the lease duration remains, which is approximately every ten minutes for a 1200 second token, and use the lease duration returned by the server rather than a hardcoded interval, so that a future change to the role does not silently reintroduce the bug.
7. Add the re-authentication branch. A renewal can be refused, for example after the token has already expired or the role has been rotated, and a client whose only recovery path is renewal will fail permanently the first time that happens. Treat a refused renewal as a signal to perform a fresh AppRole login and then re-read every value being held.
8. Remove the scheduled restart from the workaround register only after renewal is proven in production for longer than four token lifetimes, and record in that register what the restart had actually become load-bearing for.

Verification

1. Read the audit device output for the running instance accessor and confirm renewal requests are arriving at roughly ten minute intervals and are recorded as allowed. This is the server observing the client, which is independent of anything the application chooses to log about itself.
2. Check the shape of the authentication stream in the same audit output. Before the fix it contained exactly one login per process and nothing else; afterwards it should contain one login per process plus a steady renewal rhythm. A login rate that still tracks the restart rate means the renewal loop is not running.
3. Look up the live token and confirm its remaining lifetime rises after each renewal instead of falling monotonically. A renewal that is issued but ignored, for example because the client discards the response, looks identical in application logs and completely different here.
4. Run one instance untouched for ninety minutes with the fifteen-minute refresh timer active, which is four and a half token lifetimes, and confirm zero 403 responses across that window. A test shorter than the lease duration cannot observe this class of defect and must not be accepted as evidence.
5. Revoke the running token deliberately and confirm the instance re-authenticates and keeps serving. This is the only way to exercise the branch that runs when renewal is refused, and it is the branch most likely to be missing.
6. Confirm the new metric is present and correct by comparing the exported remaining lifetime against the value the server reports for the same accessor. A metric that reports a plausible but stale number is worse than no metric.
7. Fire the alert deliberately by holding one canary instance without renewal and confirm the warning arrives at five minutes of remaining lifetime and the page arrives at two. An alert nobody has ever seen fire is a hypothesis.

Prevention

• Export remaining credential lifetime as a metric. Warn when any instance falls below twenty-five per cent of its lease duration, five minutes for this token, and page at ten per cent. The defect is visible for minutes before any user sees a 502.
• Alert on the status code from the secrets client. One 403 in five minutes warns, five in five minutes pages. This incident spent fifty-six minutes being investigated as a capacity problem while the precise status code sat in the application log.
• Gate releases on a longevity canary. Run one instance for four times the token lifetime with the refresh timer active. Every test in this pipeline finished in under six minutes, which is why fourteen months of green builds proved nothing.
• Register workarounds with an owner and an expiry date. Before removing one, run the system without it in a staging environment for longer than the longest timer in the system. The change record rated this removal as no risk because nobody knew what the restart had come to be responsible for.
• Move lifecycle out of the application. An agent or sidecar that owns login, renewal and re-authentication gives every service one correct implementation instead of a dozen partial ones written by people who each read the login response once.
• Review role lifetimes against client behaviour quarterly. Compare each role token lifetime with the renewal interval the client actually implements, and treat a client with no renewal interval as an open finding rather than a note.
• Make uptime a dimension of every incident question. When a service fails on a schedule rather than under a condition, plot the failures against process age before plotting them against traffic. This incident had six instances failing at the same interval across three availability zones and an order of magnitude of load, and that pattern was in the fleet view for fifty minutes before anybody subtracted two timestamps.