Leases and TTLs: choosing an exposure window you can actually operate

A lease is not a timer attached to a secret. It is a contract with two obligations on it. The issuer promises the credential will keep working until a stated instant and promises to destroy it afterwards. The holder accepts that it must come back before that instant or lose access. Almost every operational surprise in this area comes from a team that read the first obligation and skipped the second.

Reading a live lease

Every dynamic secret carries a lease, and the lease is queryable by anyone with authority over it. This is the record for a PostgreSQL credential issued from a role whose default_ttl is two minutes:

$ bao lease lookup database/creds/app-readonly/xoHI541EXoFgKn1OTiusOdd9
Key             Value
---             -----
expire_time     2026-08-26T21:25:43.891514518Z
id              database/creds/app-readonly/xoHI541EXoFgKn1OTiusOdd9
issue_time      2026-08-26T21:23:43.891514368Z
last_renewal    <nil>
path            database/creds/app-readonly
renewable       true
ttl             1m59s

Five of those fields carry operational weight. issue_time and expire_time are exactly 120 seconds apart, which is the role’s default_ttl honoured to the microsecond. ttl is not a configuration value but the remaining lifetime at the moment of the query, so it is the field you read during an incident. renewable tells you whether extension is even possible for this lease, and last_renewal being empty tells you the holder has never once attempted it. In a fleet where clients are supposed to renew, an empty last_renewal on a lease older than a few minutes is a defect report.

path matters for a different reason. It is the class the credential belongs to, and it is what a prefix revocation targets. A lease listing grouped by path is the closest thing to an inventory of live credentials that most estates ever get.

Renewal, re-issuance, and the three ceilings

Renewal extends an existing lease. Re-issuance discards it and asks for a new credential. They are different operations with different failure modes, and the difference is where most client bugs live.

LEASE=database/creds/app-readonly/xoHI541EXoFgKn1OTiusOdd9

# Ask for another 60 seconds of life, measured from now.
bao lease renew -increment=60s "$LEASE"

# Give the credential back early when the work finishes.
bao lease revoke "$LEASE"

Two properties of renewal are counter-intuitive and both are documented. The increment is measured from the current time, not added to the end of the current TTL, so renewing a lease with 90 seconds remaining by an increment of 60 seconds shortens it. And the increment is advisory: the backend that owns the secret may ignore it entirely and return whatever it is willing to grant. A client that assumes it received what it asked for, and schedules its next renewal accordingly, will eventually schedule that renewal after the credential has already died. Read the returned lease duration; never trust the requested one.

Three ceilings bound how long any lease can live, and the effective limit is the smallest of them. The mount’s max_ttl caps the role, the system-wide max_lease_ttl caps the mount, and the authentication method may suggest a value that is less permissive but never more. The system defaults are worth knowing because they are generous enough to hide a misconfiguration for a month:

default_lease_ttl = "768h"
max_lease_ttl     = "768h"

That is 32 days on both. If a role definition forgets to set its own TTLs, credentials silently inherit a lifetime long enough that nobody notices the design was never applied.

flowchart LR
    A["Credential issued\nttl = 2m"] --> B{"Holder returns\nbefore expiry?"}
    B -- "renew" --> C["New expiry from now\ncapped by max_ttl"]
    C --> B
    B -- "no" --> D["Lease expires\nissuer revokes automatically"]
    B -- "max_ttl reached" --> E["Renewal refused\nre-issuance required"]

The path on the right of that diagram is the one clients forget. Renewal cannot extend a lease past the role’s max_ttl, so a credential issued with a two-minute TTL against a ten-minute max_ttl dies at ten minutes regardless of how diligently it has been renewed. A client that implements only renewal works perfectly in testing and fails after exactly max_ttl in production, which is a long enough delay for the change that introduced it to be forgotten.

The trade-off, stated honestly

“Shorter TTLs are better” is repeated widely enough to pass for received wisdom, and it is only half true. A TTL sets two quantities at once, and they move in opposite directions.

The first is the exposure window. If a credential leaks the instant it is issued and nobody notices, it stops being useful after its TTL elapses. A two-minute TTL means a leaked credential in a support ticket is almost certainly inert by the time anyone reads the ticket. An eight-hour TTL means it is probably live.

The second is your tolerance for the issuer being unavailable. A workload survives an issuer outage only for as long as its current credential remains valid. With a two-minute TTL, an issuer outage longer than two minutes takes down every workload that needs a new connection. With an eight-hour TTL and a client that renews at half of its lifetime, roughly four hours of cover exists before anything user visible happens. The TTL is simultaneously the maximum exposure and the maximum outage budget, and choosing it means deciding which of those two risks your estate is worse at handling.

There is a third quantity that is easy to overlook. Request load on the issuer scales with the fleet size divided by the TTL. Four hundred workloads on a two-minute TTL generate a sustained 3.3 credential operations per second before any retries, and every one of them is a write that creates and later destroys a principal in the target database. The same fleet on an eight-hour TTL generates roughly one operation every seventy-two seconds. The short-TTL configuration is not merely more chatty, it puts continuous churn into a database’s role catalogue and continuous volume into the audit log.

A defensible way to choose is to work backwards from the failure you would rather have. Short TTLs suit credentials with a wide blast radius and a fleet that fails gracefully when a connection is refused. Longer TTLs suit workloads that cannot tolerate a dependency on the issuer during startup, or that run in a network segment where reaching the issuer is itself unreliable. Mixing both in one estate is entirely reasonable, and is usually a sign that somebody thought about it.

Diagnosing a credential that expired under a running process

The symptom is distinctive once you have seen it. A service that has been healthy for some fixed interval starts failing on new database connections while its existing pooled connections continue to work, producing a partial outage that recovers on restart and then recurs after the same interval. The interval is the clue: match it against default_ttl and max_ttl on the role.

Work the diagnosis in this order. Look up the lease and read ttl and last_renewal. An empty last_renewal on a lease that should have been renewed several times means the client never implemented renewal. A populated last_renewal combined with a failure at exactly max_ttl means renewal is implemented and re-issuance is not. A lease that does not exist at all means it was revoked or already expired, and the audit record will say which.

Then check the target system independently, the way the previous lesson did. If the principal is gone, the lifecycle worked as designed and the defect is in the client. If the principal still exists but the credential is refused, look at the database-side expiry rather than the lease. If the principal exists and works, the failure was never about credentials at all, and you have just ruled out an entire branch of the investigation cheaply.

Production discipline

1. Set TTLs on the role, never inherit them. An unset TTL takes the system default of 32 days, which is indistinguishable from having no dynamic-credential strategy while looking exactly like having one.
2. Read the granted lease duration, not the requested one. The increment is advisory. Schedule the next renewal from what came back, and rebuild the schedule after every renewal.
3. Implement re-issuance and test it on purpose. Force a client through max_ttl in a pre-production environment before production does it for you at an hour of its choosing.
4. Alert on leases that have never been renewed. An empty last_renewal across a class of leases is a fleet-wide client defect that is invisible until the day it matters.
5. Record the TTL decision with its reasoning. Write down the exposure window and the outage budget the number implies, so the next person to shorten it knows what they are also shortening.

Cross-course references

• Observability for Production Sysadmins - Part XIX (Alertmanager) covers routing and inhibition, which is what keeps a fleet-wide credential expiry from arriving as four hundred separate pages.
• Linux for Production Sysadmins - Part XXIV (Time) covers clock synchronisation, without which an expiry timestamp means something different on the issuer than it does on the host reading it.
• Kubernetes for Production Sysadmins - Part LX (ServiceAccounts) covers the projected token that the kubelet rotates on a very similar schedule, and the same renew-versus-reload distinction applies there.

Quiz