← All runbooks in Secrets, PKI & Certificates

high riskservice affecting~45 min

Runbook: Recover an Unavailable Secret Manager

1 · Prerequisites

Confirm every item is in place before any state change.

secrets-pki-lab-19-deploy-secret-manager-and-unseal
secrets-pki-lab-25-recover-from-secret-manager-outage
secrets-pki-rb-17-troubleshoot-secret-manager-authentication
Console or out-of-band access to every node, because a node that will not answer on the API port may still be reachable on the console
The unseal share holders identified by name and reachable now, with a quorum available in the same time zone as the incident
Read access to the server configuration file and to the storage path, so the running configuration can be compared with the intended one
An explicit instruction that nobody runs an initialisation command during this procedure

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Name the symptom before you name the cause. A refusal carrying a status code, a connection refused, a request that hangs forever, and a redirect to another node are four different faults. Write down which one you actually have, from the failing client, before you touch a node.
· Confirm the quorum of share holders is available. Count the people, not the shares. A threshold of two out of three means two humans who each hold a distinct share must be reachable now. Discovering at minute forty that one of them is on a flight is the difference between an outage and an incident review.
· Read the seal state on every node, not just the one the load balancer favours. bao status on each member tells you whether you have one sealed node or a sealed cluster, and those have different recoveries.
· **Read the Initialized line before anything else.** Initialized false on a node you know was initialised is not a data-loss event; it is a storage-path event. Treat it as a red flag against running any command that could write a fresh barrier.
· Check the filesystem that holds the storage backend and the one that holds the audit destination. df -h on both. A full audit filesystem makes requests hang rather than fail, which is the single most misdiagnosed OpenBao outage.
· Confirm which storage backend this deployment actually uses. OpenBao ships four: integrated Raft, PostgreSQL, filesystem and in-memory. The recovery for each is different, and the filesystem backend is deprecated for removal in version 2.7.0.
· Preserve evidence before you change anything. Copy the server log, the current configuration file and the output of bao status from each node into the incident record. Recovery destroys diagnosis.
· Agree who is allowed to type. In a seal incident the dangerous commands are short and the room is loud. One operator at the keyboard, one reading the runbook aloud.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Classify the symptom from the client side. A Code: 503 naming the seal state means the node is up and refusing everything. A connection refused means the process or the listener is down. A hang means something upstream of the request is blocking, and the audit device is the first suspect because auditing is fail-closed. A redirect means you reached a standby.
2**Read bao status on every node and record all of it.** The Sealed, Initialized, Total Shares, Threshold and Unseal Progress lines together describe the state. Any one of them read alone will mislead you.
3**Stop immediately if Initialized reports false on a node that was previously initialised.** The node is not looking at your data. Check the mount, the volume, the container bind and the configured storage path. Running an initialisation command at this point creates a fresh barrier and buries the real one.
4Verify the storage path is mounted and writable. An unmounted volume presents an empty directory underneath it, which is indistinguishable from a fresh deployment until you check the mount table rather than the directory listing.
5Verify the audit destination has space and is not blocked. A device that cannot write makes requests hang indefinitely, and the service will look alive to every liveness probe while serving nobody.
6Confirm the process is running and the listener is bound before blaming the seal. systemctl status and a socket listing answer this in seconds and rule out an entire branch of the tree.
7Assemble the share holders and run the unseal ceremony one share at a time. Shares may be submitted in any order and by different people from different terminals. Unseal Progress counts up until the threshold is met, at which point the node reports Sealed false.
8Never place a share on a command line that a shell history file will keep. Read it into a variable with a silent read, pass the variable, then unset it. Rotating a share that leaked into a history file is a separate incident nobody wants this week.
9Expect every restarted node to be sealed, and unseal each one. A sealed node cannot act as a standby, so a cluster that has been restarted provides no high availability at all until a human has unsealed each member.
10If the deployment uses auto-unseal, diagnose the key provider rather than reaching for the recovery keys. Recovery keys cannot decrypt the root key, so they cannot substitute for a key provider that is unavailable.
11Confirm the audit device came back with the service. bao audit list -detailed must show the device. Declarative devices are created on the active node during restarts and reload events, so a configuration file edited during the incident changes what comes back.
12Restore service to consumers in a deliberate order. Bring back the workloads that mint short-lived credentials first, because they will re-authenticate on their own, and the ones holding cached long-lived material last.
13Only after all of the above may anyone use the word restore. A restore from backup rolls the estate back to the backup point and is the subject of its own runbook. It is never the first response to a sealed node.

4 · Verification

Confirm the procedure actually fixed the problem.

✓bao status on every node reports Sealed false and Initialized true, and the values were read on each node rather than through the load balancer.
✓A workload reads its own secret using its own credential from inside its own container, and the command exits 0.
✓bao audit list -detailed shows the expected audit device with the expected destination path.
✓A new record appears at the audit destination within seconds of a test request, proving the device is writing and not merely listed.
✓The filesystem holding the storage backend and the one holding the audit destination both report free space, checked after service resumed rather than before.
✓A dynamic credential can be issued and then revoked end to end, proving the lease machinery recovered and not just the read path.
✓Standby nodes report unsealed, so the cluster genuinely has failover cover rather than a single unsealed member.
✓The running configuration file matches the version-controlled one, byte for byte, after any edit made during the incident.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If a node was unsealed that should not have been, bao operator seal returns it to the sealed state. Understand that this is a deliberate outage for every caller of that node.
↶If a configuration file was edited during the incident, restore the version-controlled copy and reload rather than leaving an undocumented running state.
↶If a storage volume was remounted or replaced, stop and preserve both copies before any further action. The wrong volume mounted twice is recoverable; the right volume overwritten is not.
↶If an initialisation command was run by mistake, halt the service immediately, do not unseal the new barrier, and preserve the storage directory exactly as it is for the platform owner.
↶If the unseal ceremony was started with the wrong share set, the progress counter can be abandoned by leaving the node sealed; no partial progress grants any access.
↶If recovery made the symptom worse, return every node to the state recorded in the pre-checks and escalate with the evidence you preserved.

6 · Escalation

When the runbook isn't enough, contact:

· A quorum of share holders cannot be assembled: escalate immediately to the platform owner and the security owner together, because the ceiling on this incident is now a people problem with a hard failure mode.
· Initialized reports false on a node that was initialised: escalate before typing anything else. This is a storage or deployment fault and the wrong command here is unrecoverable.
· The key provider for an auto-unsealed deployment is unavailable: escalate to whoever owns that provider. Recovery keys will not unseal in its place.
· Requests hang with no status code and the audit filesystem is full: escalate to the owner of the log sink while you free space, because the same fill rate will recur.
· More than one node reports storage errors: treat it as a storage incident rather than a secret manager incident and page the storage on-call.
· Anyone in the room proposes a restore from backup within the first hour: escalate to the service owner for an explicit decision, because a restore has data-loss consequences that a sealed node does not.

“The secret manager is down” is one alert covering at least five unrelated faults. The barrier may be sealed, so the process is healthy and refusing everything. The process may be stopped, so nothing is listening at all. The storage path may be missing, so the process is healthy and looking at an empty universe. The audit destination may be full, so requests hang rather than fail. Or the node you reached may be a standby that has nowhere to redirect to.

These have different recoveries, and one of them has a neighbouring command that destroys the estate. A node that reports Initialized false when you know it was initialised is not telling you the data is gone. It is telling you that this process is not looking at your storage. Initialising it creates a brand new barrier over the top of that misconfiguration and turns a ten minute mount problem into a restore.

Work the tree in order. Symptom, then seal state, then initialisation state, then storage, then audit sink, then quorum. Nothing below the line you are on can be diagnosed until the line you are on is settled.

When this runbook applies, and when it does not

It applies when OpenBao is not serving requests and you do not yet know why, including the case where it is serving some callers and refusing others.

It does not apply when:

One workload fails and the rest are fine. That is a credential or a grant, and the authentication and authorisation runbooks cover it. A seal incident does not discriminate between callers.
The data is known to be gone and a restore has been authorised. That is a separate procedure with data-loss consequences, and it should never be reached without completing the checks below first.
The service was never successfully initialised. A first deployment that has never worked is a build task. Running incident procedures against it invites exactly the initialisation mistake this page warns about.
You cannot assemble a quorum of unseal shares. Nothing in this runbook substitutes for the shares. Escalate at once rather than spending the first hour on commands that cannot succeed.

Blast radius

Action	Reversible?	What it costs if wrong
Reading status on each node	Yes	Nothing. These are reads.
Checking mounts, disk and the process state	Yes	Nothing.
Freeing space on the audit filesystem	Yes	Deleting the wrong file loses audit history you may need.
Unsealing a node	Yes, by sealing it again	Very little, provided the storage underneath is the right storage.
Editing the server configuration	Only if you kept a copy	Declarative audit devices are rebuilt from the file on reload; a bad edit removes one.
Remounting or replacing a storage volume	Only if both copies survive	The right volume overwritten is not recoverable.
Running an initialisation command	No	Creates a fresh barrier and buries the real one.

Step 1 - Turn the symptom into one of four words

Before any node is touched, decide from the failing client which of these you have. The rest of the tree hangs off this answer.

Client sees	Meaning	Next
`Code: 503` naming the seal state	Process healthy, barrier sealed	Step 2
Connection refused	Process or listener down	Step 6, then Step 2
Request hangs, no status code	Something upstream is blocking; suspect the audit sink	Step 5
Redirect to another address	You reached a standby	Step 2 on the active node

A status code is a gift. A hang is not, because every naive liveness probe will report the service as healthy while it serves nobody.

Step 2 - Read the whole status block, on every node

Read-only / Safebao-1, sealed: healthy process, no access for anyone

$ bao status

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       1
Threshold          1
Unseal Progress    0/1
Unseal Nonce       n/a
Version            2.6.2
Commit Date        2026-08-18T15:48:19Z
Storage Type       inmem
HA Enabled         false

Illustrative output

Initialized true with Sealed true is the ordinary post-restart state and the safest thing on this page: the data is there, the barrier is shut, and a quorum of humans opens it. Note that the status command exits non-zero while sealed, so any monitoring that scripts on the exit code alone cannot tell this apart from a far worse state. The Storage Type, Total Shares and Threshold values above come from the capture host used for this course and will differ on yours; read the field names, not the numbers.

Step 3 - Treat `Initialized false` as a storage fault, never as data loss

Read-only / Safethe state that gets estates destroyed: no barrier visible at all

$ bao status

Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            2.6.2
Commit Date        2026-08-18T15:48:19Z
Storage Type       file
HA Enabled         false

Illustrative output

Total Shares 0 and Threshold 0 mean this process has never seen a barrier. On a node that was serving traffic yesterday, the overwhelming likelihood is that the storage path is empty because a volume is not mounted, a container bind points somewhere new, or the configuration file was replaced with a default.

Step 4 - Prove the storage path is really mounted

# The directory listing lies when a volume is not mounted; the mount table does not.
DATA=/openbao/data
findmnt --target "$DATA" || echo "not a mount point"
df -h "$DATA"
ls -ld "$DATA"

An unmounted volume presents an empty directory beneath its mountpoint, which is exactly what a fresh deployment looks like. findmnt answers the question the listing cannot. Check the ownership as well: a storage directory the service user cannot write is a different fault with the same alert.

Step 5 - Check the audit sink, because auditing is fail-closed

AUDIT=/openbao/audit
df -h "$AUDIT"
ls -l "$AUDIT"

OpenBao will not answer requests that no enabled audit device can record, and when a device is blocked, requests hang until it can write. A full filesystem under the audit path therefore produces a total outage with no error message anywhere. Free space, confirm the device can write, and only then look further up the tree.

Step 6 - Confirm the process and the listener before blaming the barrier

systemctl status openbao --no-pager
journalctl -u openbao -n 100 --no-pager
ss -ltnp | grep -F ':8200' || echo "nothing listening on 8200"

If nothing is listening, no amount of seal diagnosis helps. Start the service, then return to step 2, because a service that has just started is sealed by definition.

Step 7 - Run the unseal ceremony

Shamir splits the unseal key, which decrypts the root key, which decrypts the keyring, which decrypts the data. The shares can be supplied one at a time, in any order, by different people at different terminals. The service counts them until the threshold is reached.

# Each share holder runs this on their own terminal. The share never
# reaches the shell history, and the variable is cleared immediately.
read -r -s -p 'unseal share: ' SHARE
bao operator unseal "$SHARE"
unset SHARE

Each accepted share prints the running total. After the first of two the node is still shut, and says so:

Service impact possiblebao-1, first share accepted, still sealed

$ bao operator unseal "$SHARE"

Sealed             true
Threshold          2
Unseal Progress    1/2

Illustrative output

After the second share the barrier opens and the node reports its cluster identity:

Service impact possiblebao-1, threshold met, barrier open

$ bao operator unseal "$SHARE"

Sealed          false
Threshold       2
Cluster Name    vault-cluster-23a006cb
Cluster ID      e3cb274d-ce14-5d56-0e94-f38bddb6ae60

Illustrative output

The initialisation output says the hard part plainly, and it is worth reading aloud during the ceremony: the service does not store the generated root key, and without a threshold of shares it remains permanently sealed. That sentence is the reason share custody is a named responsibility with a named deputy, not a shared drive.

Step 8 - Restore consumers in a deliberate order

Bring back the workloads that authenticate for themselves and mint short-lived credentials first; they will recover without help. Leave the ones holding cached long-lived material until last, because they will recover with or without you and they crowd the logs while you are still reading them. Verify each tier from the consumer side before starting the next.

Common pitfalls

Symptom	Cause	Action
Sealed again after every restart	Shamir seal with no auto-unseal	Expected. Plan for it, or move the key to a provider
`Initialized false` on a known-good node	Storage path empty or unmounted	Check the mount table, never initialise
Service healthy, nobody served	Audit sink blocked; auditing is fail-closed	Free the audit filesystem
Cluster has no failover after a restart	Standbys are still sealed	Unseal each member, not only the active one
Auto-unseal fails and recovery keys do not help	Recovery keys cannot decrypt the root key	Fix the key provider; escalate to its owner
Monitoring never noticed	Exit code alone conflates sealed with uninitialised	Alert on the `Sealed` and `Initialized` fields

Verification

Recovery is verified from the consumer side, never from the node you have just been typing on. Read Sealed false and Initialized true on each member individually rather than through the load balancer, then have a workload read its own secret using its own credential from inside its own container. Confirm the audit device is listed and, more importantly, that a fresh record lands at its destination within seconds of a test request: listed and writing are different states. Issue a dynamic credential and revoke it again, which exercises the lease machinery rather than only the read path. Check free space on both the storage and audit filesystems now that traffic has resumed, since the fill rate under load is what caused the incident in the first place. Finally, confirm the standbys are unsealed, so the cluster has genuine failover cover and not just a working active node.

Rollback

Rollback in a recovery is mostly about returning to a known state rather than undoing a fix. A node unsealed in error can be sealed again with bao operator seal, which is a deliberate outage for every caller of that node and should be announced before it is typed. A configuration file edited during the incident is restored from version control and reloaded, because declarative audit devices are rebuilt from that file and an undocumented running state will surprise the next person. If a storage volume was remounted or swapped, stop and preserve both copies before doing anything else. If an initialisation command was run by mistake, halt the service, do not unseal the new barrier, and hand the storage directory to the platform owner exactly as it stands. In every case, the evidence captured in the pre-checks is what makes the rollback reviewable.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

When this runbook applies, and when it does not

Blast radius

Step 1 - Turn the symptom into one of four words

Step 2 - Read the whole status block, on every node

Step 3 - Treat Initialized false as a storage fault, never as data loss

Step 4 - Prove the storage path is really mounted

Step 5 - Check the audit sink, because auditing is fail-closed

Step 6 - Confirm the process and the listener before blaming the barrier

Step 7 - Run the unseal ceremony

Step 8 - Restore consumers in a deliberate order

Common pitfalls

Verification

Rollback

References

Step 3 - Treat `Initialized false` as a storage fault, never as data loss