Secret manager outage and its failure modes

The most confusing thing about a secret manager outage is that most of production carries on working. Dashboards stay green, requests succeed, and the only signal is a handful of pods that will not start. An hour later, unrelated services begin failing against a database that is perfectly healthy. Both symptoms come from the same seal, and they arrive at different times because different applications hold their secrets differently.

What sealed means, and what a client sees

A sealed manager holds its data encrypted and cannot read it. The key that unlocks the storage is protected by a root key, and the root key is held in storage encrypted under a separate unseal key. A threshold of the shares produced at initialisation reconstructs that unseal key, which then decrypts the root key into memory. Until enough shares are supplied, the process is running, listening and answering, and it can do nothing useful.

Unseal Key 1: <REDACTED>
Unseal Key 2: <REDACTED>
Unseal Key 3: <REDACTED>

Vault initialized with 3 key shares and a key threshold of 2. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 2 of these keys to unseal it
before it can start servicing requests.

That initialisation banner is worth reading as an operational statement rather than as setup output. Restarting the process seals it. Stopping it seals it. A node that reboots for a kernel update at 04:00 comes back sealed and stays that way until two people with different shares turn up. OpenBao prints “Vault” in these strings, which is a naming legacy rather than a mistake in your reading.

The client-visible symptom is unambiguous, which is one small mercy:

Code: 503. Errors:

* Vault is sealed

A 503 with that error body means the manager is up and refusing, not down and unreachable. Distinguishing the two matters, because an unreachable manager might be a network partition affecting one availability zone while a sealed one affects everybody.

Three populations, three different outages

flowchart TD
    S["Manager seals at T0"] --> A["Startup fetch,\nvalue cached in memory"]
    S --> B["Per-request fetch"]
    S --> C["Dynamic credential\nwith a lease"]
    A --> A2["No impact until\nsomething restarts"]
    B --> B2["Total failure at T0"]
    C --> C2["Fails when the lease\nTTL expires"]

An application that reads its secrets once at startup and keeps them in memory is unaffected for as long as its processes keep running. This is the dangerous population, because the outage is latent rather than absent: a rolling deployment, an autoscaler scaling in and back out, a node drain, an out-of-memory kill or a liveness probe restart all convert it into a hard failure at a moment nobody associates with the secret manager. The correct operational response is to freeze anything that restarts a process, and to say so explicitly in the incident channel, because the natural instinct during an incident is to restart things.

An application that fetches per request fails completely and immediately. This is loud, unambiguous, and recovers on its own the instant the manager returns. It is also, on the day, the population that tells you the manager is sealed at all.

An application holding a dynamic credential is on a timer. This is the population that produces the second wave, and it is the one people forget.

What happens to existing leases

A dynamic credential is issued with a lease, and the lease carries a duration, an issue time and an expiry:

expire_time     2026-08-26T21:25:43.891514518Z
issue_time      2026-08-26T21:23:43.891514368Z
renewable       true
ttl             1m59s

While the manager is sealed, renewal requests fail like every other request. The lease keeps counting down regardless, because nothing about a lease depends on the manager continuing to exist. When it expires, the credential stops working, and it stops working at the backend rather than at the manager. A dynamic PostgreSQL credential is a real role in the database carrying a VALID UNTIL timestamp set by the creation statement, so the database refuses the login on its own schedule with the manager entirely out of the picture.

That is the mechanism behind the delayed second wave. The deadline for the whole incident is set by your shortest lease TTL, not by your patience, and it applies to services that were completely healthy when the seal happened. A fleet standardised on a two minute TTL has two minutes; a fleet on eight hour leases has most of a working day. Neither is wrong, but you must know which one you are.

Revocation is affected in the same direction. Revoking a lease is work the manager performs against the backend, and a sealed manager performs no work, so credentials that ought to have been revoked stay live until their own expiry catches them. Whether pending revocations are replayed when the manager returns is a property of your manager and its storage backend. Establish that answer in a drill, not during the incident.

Not every application should fail the same way

It is tempting to declare a standard, usually that everything should fail closed. Resist the general rule and decide per secret, because the failure semantics belong to what the secret protects.

• An authorisation secret fails closed. A service that cannot verify a token must refuse the request. Serving traffic without the check is not degraded operation, it is an authorisation bypass.
• A configuration value with a safe default can fail open. A feature flag or a rate limit that falls back to a conservative compiled-in value keeps the service useful and is far better than a crash loop.
• A credential for a downstream dependency should fail with a cached value and a bounded staleness. Keep the last known good value, keep serving, and alert on the age of the cache rather than on the fetch failure.

Writing this down per secret, before the outage, is what turns a seal into a partial degradation instead of a full stop. It also tells you what to test: an application whose fail-open path has never been exercised has an untested code path standing between you and the incident.

Recovery order

Unseal first, with the threshold supplied by different people. The progress counter is explicit about where you are:

bao operator unseal
bao status

After the first share the status shows Unseal Progress 1/2 with Sealed true; after the second, Sealed false. Verify with an independent read of a known non-secret canary path rather than by trusting the status output, then work the populations in order: the per-request services recover by themselves and need only confirmation; the cached-value services need nothing at all and must still not be restarted until you are confident the manager is stable; the lease holders need either a renewal or a restart, and their restart is safe only once the manager is genuinely serving. Unfreeze deployments last, and say so, because a freeze that is never explicitly lifted becomes a second incident on its own.

Production discipline

1. Record the retrieval pattern of every consumer. Startup fetch, per-request fetch or lease holder. That column is the impact assessment, precomputed.
2. Know your shortest lease TTL as a number. It is the deadline for the incident, and it is usually shorter than anybody in the call assumes.
3. Alert on the age of a cached secret, not only on fetch failures. A service quietly serving a value it has been unable to refresh for six hours is the failure you want to see before the lease expires.
4. Hold unseal shares across people and locations. A threshold of two out of three is theatre if one person can produce both shares from the same drawer.

Cross-course references

• Kubernetes for Production Sysadmins - Part XXXIII (Cordon, Drain and Uncordon) covers the node operations that turn a latent secret manager outage into a live one, which is why they are the first thing to freeze.
• Observability for Production Sysadmins - Part XXI (Alert Inhibition) covers the dependency-aware suppression that stops a single seal from paging every downstream service at once.
• Linux for Production Sysadmins - Part LXIII (Cluster Time, DNS and Identity Dependencies) covers the general pattern of designing around a central dependency that will eventually be unavailable.

Quiz