Skip to main content
RunBook Academy

← All break/fix scenarios in Secrets, PKI & Certificates

intermediatesecret-outage~40 min

Sealed secret manager: the outage that only affects what restarts

Reported symptoms

  • ●At 07:18 UTC the payments deployment stalls: new Pods never become ready, while the Pods they were meant to replace keep serving customer traffic normally
  • ●The morning batch fleet that starts at 06:40 has failed every job, and each job log ends at the point where it fetches a database credential
  • ●A low-priority alert for the reporting worker has been firing quietly since 04:09 and was triaged overnight as a flaky database connection
  • ●Twenty-four long-running services are completely healthy and have not logged a single error, which the incident channel reads as proof that the secret manager is fine
  • ●The secret-manager service is running on all three nodes, the load balancer completes TCP connections to all three, and the TLS certificate is valid for another 96 days
  • ●The only change in the window is a routine kernel patch that rebooted the three nodes between 03:12 and 03:31 UTC and was recorded as successful

Evidence

  • · A read of any key path returns HTTP 503 with the message that the store is sealed, identically from all three nodes and from every client language in the estate
  • · The seal status command on each of the three nodes reports the node as sealed, initialised, with an unseal progress of zero out of a threshold of two
  • · The refusal is an application-layer 503 rather than a refused connection, which is why every reachability probe in the estate stayed green throughout
  • · The host uptime on all three nodes begins during the patch window, matching the change record exactly
  • · The AppRole login response that each workload receives carries a token lease of 1200 seconds, so no client token in the estate outlived 03:32
  • · The twenty-four healthy services all read static key-value secrets once at process start and hold them in memory, and none of them has restarted since before the patch window
  • · Every failing workload is either starting for the first time since 03:12 or renewing a lease, and lease renewal is itself an API call that returns the same 503
  • · The storage backend is intact and every node reports itself initialised, so no data has been lost and no node needs to be rebuilt
Diagnosis and resolutionclick to reveal

Root cause

A Shamir seal protects the data at rest with a root key that the process holds only in memory. Restarting, stopping or re-sealing a node discards that key, and the node returns to service sealed: listening, answering, and refusing every request with a 503 until a threshold of key shares reconstructs the root key. The patch window rebooted all three nodes, so all three came back sealed at 03:31 and the cluster had no unsealed member from that moment on. Two independent defects made this an incident rather than a two-minute chore. The first is that the estate runs a manual Shamir seal with no auto-unseal, so every restart of every node is by definition an outage that requires two named humans and their key shares. The second, and the one that made this a three-hour undetected outage, is that nothing in the monitoring estate observes seal status. Every probe measured reachability, and a sealed node is perfectly reachable. It answers on the API port, completes the TLS handshake, and returns a well-formed refusal. The blast radius then followed the credential model exactly. A workload holding a static value in memory needs nothing from the secret manager and survived indefinitely. A workload that needed a token, a lease renewal or a first read failed the moment it needed it. That is why the failure arrived in three separate waves hours apart, and why the estate looked healthy in the only dashboard anyone checked.

Remediation

Confirm what kind of failure this is before touching anything, because one wrong command here is unrecoverable. A sealed node reports itself as initialised; that single field distinguishes a node that needs unsealing from a node pointed at empty storage. Never run the initialise command to make the error go away. Against an empty or wrongly configured storage path it succeeds, creates a brand new empty barrier with new key shares, and leaves the real data encrypted under a root key nobody will ever reconstruct. Next, page the key-share custodians rather than hunting for shares yourself. Each custodian submits their own share to the node directly; shares must never be collected into one person''s terminal, because at that instant one individual holds the ability to unseal alone and the ceremony has lost its only property. Unseal one node first, confirm it reports itself unsealed, and let the load balancer route to it. Service returns at that point for everything that can simply retry. Then unseal the second and third nodes so the cluster has redundancy again, rather than leaving a single unsealed node carrying the estate. Only now restart the failed workloads, in dependency order, starting with the ones other services call. Do not paste secret values into environment variables as a stopgap. That converts a recoverable outage into an unmanaged copy of every production credential, spread across hosts nobody is tracking, which then has to be rotated.

Verification

Prove the service is working by watching something other than the command you just ran. The seal status command tells you what you just changed to the node; it does not tell you that a real workload can authenticate and read. Take the application''s own role credentials on a host that was never part of the incident, log in, and read a real path. Then read the audit device''s file directly and confirm that the login and the read were both recorded with the application''s accessor rather than an operator token, which proves the request travelled the same path a workload uses. Confirm the second and third nodes are unsealed too, so the next reboot of a single node is not another outage. Watch the batch fleet actually complete rather than merely start: a job that fetches its credential successfully and then fails on something else is a different incident. Confirm the stalled deployment finishes and that the new Pods pass readiness, then check that the reporting worker whose alert has been firing since 04:09 has recovered and that the alert has cleared by itself. Finally, restart one low-risk service deliberately. Everything in the estate that was healthy during the outage was healthy because it had not restarted, and until something restarts successfully you have not tested the thing that was broken.

Prevention

Configure auto-unseal so that a restart is a restart rather than an incident. An auto-unseal seal delegates the unwrapping of the root key to an external key service, which turns the recovery from a two-person ceremony into a node coming back on its own. Keep the Shamir shares as the recovery path and rehearse them quarterly against a restored backup, with a written pass mark of thirty minutes from page to unsealed. If the estate must stay on a manual seal, raise the share count to five with a threshold of three, held by five custodians across at least three locations, and audit the escrow every quarter for shares that have quietly ended up in one safe. Alert on seal status directly, polled every 30 seconds, warning at 60 seconds sealed and paging at 120 seconds, and make that alert independent of the load balancer so it cannot be masked by a healthy reachability probe. Make every consumer resilient: cache the last known good credential, retry with capped exponential backoff rather than exiting on the first failure, and treat a 503 as retryable while treating a 403 as fatal. Alert when lease renewal fails twice in succession, because renewals fail long before the credential does and that gap is the free warning. Finally, spread the secret-manager nodes across separate patch groups so no single maintenance window can take all of them.

Reported symptoms

The estate is a payments platform with roughly sixty services. Secrets live in a three-node OpenBao cluster, bao-1 to bao-3, behind a VIP published as vault.example.com. The cluster was initialised two years ago with three key shares and a threshold of two. It has been reliable enough that nobody has thought about it since.

The kernel patch window ran between 03:12 and 03:31 UTC and rebooted all three nodes. The change record says successful, and by any check the patching job performs it was.

The first page arrived at 07:18. By then the incident channel already contained four separate stories:

  • The reporting worker has been alerting since 04:09. The overnight triage note reads intermittent database connectivity, watching.
  • The morning batch fleet started at 06:40 and every job failed. The logs all stop at the same place, where the job fetches a database credential.
  • A scheduled deployment of payments-api began at 07:05. New Pods never reach readiness. The old Pods are untouched and are still serving every customer request, so the graphs are flat and nothing customer-facing has moved.
  • Twenty-four other services have logged nothing at all. Somebody uses that to argue that the secret manager cannot be the problem, because most of the estate is reading secrets perfectly well.

That last claim is the one worth pulling on, and it is wrong in an interesting way. None of those twenty-four services has read a secret since before 03:12.

Evidence provided

Read-only / Safebao-1 at 07:20 UTC, and identical from bao-2 and bao-3
$ bao kv get kv/app/config
Code: 503. Errors:

* Vault is sealed

Illustrative output

Read-only / Safethe same three nodes, measured the other way
$ bao status

The 503 is the whole diagnosis if you read it as a protocol answer rather than as an error string. A refused connection means nothing is listening. A 503 means a healthy process accepted the request, understood it, and declined to serve it. That is why every reachability probe in the estate stayed green for four hours.

Read-only / Safebao-1, bao-2 and bao-3
$ uptime --since
Read-only / Safethe same role captured in the staging replica: a workload token lives 1200 seconds
$ bao write auth/approle/login role_id=@role_id.txt secret_id=@secret_id.txt
"auth": {
  "client_token": "s.[REDACTED]",
  "accessor": "[REDACTED]",
  "policies": ["app-read", "default"],
  "token_policies": ["app-read", "default"],
  "metadata": {"role_name": "app-role"},
  "orphan": true,
  "lease_duration": 1200,
  "renewable": true
}

Illustrative output

That number sets the clock on the whole incident. Every client token in the estate was issued with a 1200 second lease. Renewal is an API call. From 03:31 the API returned 503 to every caller, so no token in the estate outlived 03:32, whether its holder noticed or not.

Read-only / Safereport-worker: the first casualty, three hours before the page
$ journalctl -u report-worker --since '04:00' --until '04:20' --no-pager
Read-only / Safethe exculpatory evidence, and the reason it exculpates nothing
$ systemctl show --property=ActiveEnterTimestamp ledger-api pricing-api settlement-api

Work the evidence before reading on

The interesting question is not why the secret manager is refusing. It says why, in plain language, in the first block. The interesting question is why the failures arrived in three waves separated by hours.

  1. A 503 and a refused connection are different answers from different layers. Which one did every health probe in the estate test for, and what would each of them have returned at 03:32?
  2. Twenty-four services are healthy. Name the property they share, and then name what would happen to any one of them if it were restarted right now.
  3. The reporting worker failed at 04:09, not at 03:31. What was it doing in those thirty-eight minutes, and what does the gap tell you about how long the estate can coast?
  4. The node reports itself as both sealed and initialised. Which of those two fields would change if the storage backend had been lost instead, and why does the distinction decide the very first command you are allowed to run?

Before continuing: decide what a sealed node should look like to your monitoring, and explain why a reachability probe cannot see it.

Root cause

The seal is not a state the process can restore on its own

Everything in the store is encrypted with a data key, and that data key is itself wrapped by a root key. The root key exists only in the process memory of an unsealed node. Stopping the process, restarting the host or re-sealing deliberately all discard it, and on the way back up the node has an encrypted barrier it cannot open.

The initialisation record for this cluster stated the consequence plainly, two years before it mattered:

Vault initialized with 3 key shares and a key threshold of 2. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 2 of these keys to unseal it
before it can start servicing requests.

Note also that the tool prints Vault throughout. OpenBao retains the original strings for compatibility, and the incident channel spent eleven minutes arguing about whether the message even came from the right product.

The estate monitored reachability, and a sealed node is reachable

A sealed node binds its listener, terminates TLS, parses the request, consults its policy engine far enough to know it cannot serve, and returns 503. Every layer a probe usually tests is working. The load-balancer backend was up, the certificate was valid, the synthetic check passed. There was no signal anywhere in the estate that observed the one field that had changed.

The blast radius followed the credential model, not the service map

This is why the failures looked random. Sort the estate by what each workload needs from the secret manager and the pattern is exact.

  • Holds a static value in memory. Read once at process start, never read again. Survives indefinitely, and dies the instant it restarts.
  • Renews a lease. Renewal is an API call. Fails at the first renewal after the seal, then fails again on every retry.
  • Fetches at start. Fails immediately on start, which is why the batch fleet died all at once at 06:40 and the deployment stalled at 07:05.

Resolution

  1. Read the seal status on each node and confirm every one reports itself initialised. That single field separates a node that needs two key shares from a node that has lost its storage, and the two situations have opposite first commands. If any node reports itself uninitialised, stop and investigate the storage backend; do not run bao operator init to clear the error.
  2. Page the key-share custodians named in the ceremony document rather than looking for shares yourself. If you can assemble a threshold on your own, that is a finding for the review, and it does not change what you have to do in the next ten minutes.
  3. Have each custodian submit their own share to the same node in turn. The node accumulates progress and reveals nothing until the threshold is met. Shares must not be read out, pasted into chat, or typed by one person on behalf of another, because at that moment one individual can unseal alone and the ceremony has stopped meaning anything.
  4. Confirm the first node reports itself unsealed and let the load balancer route to it. Service is restored at this point for every consumer that retries, and restoring one node quickly is worth more than restoring three tidily.
  5. Unseal the second and third nodes as well, in the same way. Leaving the estate on a single unsealed node means the next single reboot repeats this incident, and you are one node away from doing it all again.
  6. Restart the failed workloads in dependency order, beginning with services that other services call. Anything that exited on its first failed fetch needs a restart; anything still retrying will recover without help. The batch fleet can be re-run once its dependencies are healthy.
  7. Do not paste secret values into environment variables, unit files or Pod specs as a stopgap. That trades a recoverable outage for an unmanaged copy of production credentials on hosts nobody is tracking, and every one of those values then has to be rotated.
  8. Record the exact clock times of the reboot, the first lease failure, the first fetch failure and the page. The three-hour gap between the outage starting and anyone knowing is the actual finding, and it will be argued away by next week if it is not written down now.

Verification

  1. From a host that took no part in the incident, log in with the application role and read a real production path. This proves the whole path a workload uses, which the seal status command does not.
  2. Read the audit device file directly and confirm both the login and the read are recorded against the application accessor rather than an operator token. The audit log is written by the server, so it is an independent witness to what the client believes happened.
  3. Confirm all three nodes report unsealed, not just the one you fixed first. One unsealed node is service restored; three is redundancy restored, and only the second claim survives the next reboot.
  4. Watch the batch fleet complete rather than start. A job that now fetches its credential and then fails on something else is a separate incident, and the distinction is invisible if you only check that the jobs launched.
  5. Confirm the stalled payments-api rollout finishes and the new Pods pass readiness, then confirm the old Pods are retired. The deployment is the only workload in the incident whose failure was masked by healthy predecessors.
  6. Check that the reporting worker recovered and that its alert cleared on its own. An alert that has to be silenced by hand after the fix means the alert is measuring the wrong thing.
  7. Deliberately restart one low-risk service. Every healthy service in this incident was healthy because it had not restarted, so until something restarts successfully you have not tested the thing that broke.

Prevention

  • Move to auto-unseal and keep Shamir as the recovery path. An auto-unseal seal has the node unwrap its root key through an external key service at start-up, so a reboot becomes a reboot. Keep the shares for disaster recovery, rehearse them quarterly against a restored backup, and set a written pass mark of thirty minutes from page to unsealed.
  • Alert on seal status, every 30 seconds. Warn at 60 seconds sealed and page at 120 seconds. Poll each node directly rather than through the VIP, so the alert cannot be answered by whichever node happens to be healthy.
  • Never let a reachability probe stand in for a readiness signal. A sealed node passes every reachability test that exists. The probe that would have fired at 03:32 is one that reads a canary secret with a real role and asserts on the returned value.
  • Split the nodes across separate patch groups. No single maintenance window should be able to restart every member of a quorum. This one did, and it was recorded as a success.
  • Make consumers survive a 503. Cache the last known good value, retry with capped exponential backoff, and distinguish a 503, which is retryable, from a 403, which never becomes true by waiting. A process that exits on its first failed fetch turns a recoverable outage into a restart storm.
  • Alert on renewal failures, not just on expiry. Renewals begin failing long before any credential actually dies. Two consecutive failed renewals is a free thirty-minute warning, and in this incident it was available from 03:32.