← All runbooks in Secrets, PKI & Certificates

critical riskdata loss risk~120 min

Runbook: Restore the Secret Management Service from Backup

1 · Prerequisites

Confirm every item is in place before any state change.

secrets-pki-rb-19-recover-unavailable-secret-manager
secrets-pki-lab-25-recover-from-secret-manager-outage
A backup and recovery review completed for this service within the last quarter, so the restore point and the retention are known quantities rather than discoveries
Written authorisation from the service owner naming the restore point and accepting the loss of everything written after it
A quorum of the unseal shares that correspond to the backup being restored, held by people who are present now
The version-controlled server configuration and policy documents, because the restored barrier is not the source of truth for either
Root or equivalent access on the target host, and the ability to stop every node that can write to the storage backend
A maintenance window agreed with every team whose workloads authenticate through this service

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the recovery runbook was completed and failed. A restore is never the first response to an unavailable service. If Initialized false has not yet been ruled out as a storage-path fault, stop and go back; a restore on top of an unmounted volume destroys data that was never lost.
· Establish which share set matches the backup. The unseal key is not inside the storage backup. If the shares were rotated between the backup point and now, the set that opens the restored barrier is the one that was in force when the backup was taken. Settle this with the custodians before the service is stopped, not after.
· Read the backup metadata and state the restore point out loud. Record the exact timestamp in the incident log. Every secret version, token, lease and policy written after it will be gone, and someone must have agreed to that in writing.
· Verify the backup can be read before you destroy anything. Extract it to a scratch path on a machine with enough space and confirm the archive is complete. A restore discovered to be corrupt after the live data has been moved is a career-defining afternoon.
· Confirm the audit device is declared in the server configuration file. Audit devices cannot be enabled through the API in this release, so a configuration file without an audit stanza produces a restored service with no audit trail and no way to add one at runtime.
· Confirm free space on both the storage filesystem and the audit filesystem. Auditing is fail-closed; a restored service whose audit device cannot write will hang rather than serve.
· List the credentials that will be rolled back and must therefore be rotated forward. Any secret rotated after the restore point returns to its previous value while the downstream system keeps the newer one. Build that list now, while you still have the incident timeline in front of you.
· Agree the order in which workloads come back. A restored service with every consumer reconnecting at once cannot be observed. Tiers, not a stampede.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Stop every node that can write to the storage backend, and prove each one has stopped. A single surviving writer during a restore corrupts the result in a way that is not obvious until much later. Confirm with the service manager, not by assumption.
2Preserve the current storage directory by renaming it, never by deleting it. Move it aside with a timestamped name on the same filesystem. Until the restore is verified, that directory is the only copy of whatever survived, and its value is unknown.
3Restore the backup into a fresh storage directory and set ownership and mode explicitly. The service user must own the tree and nobody else should be able to read it. A restored directory owned by root is a service that will not start, and a world-readable one is a finding.
4Compare the running server configuration against the version-controlled copy before starting anything. The configuration file governs the listener, the storage path and the audit device, and none of those come from the backup.
5Confirm the audit stanza is present and points at a writable destination. This is the last moment at which it can be fixed cheaply. Once the service is running, the API refuses to create audit devices in this release.
6Start the service and expect it to be sealed. A freshly started node with a restored barrier reports Initialized true and Sealed true. If it reports Initialized false, stop immediately: the restore did not land where the configuration is looking.
7Run the unseal ceremony with the share set that matches the backup. Shares are submitted one at a time in any order until the threshold is met. If the shares do not open the barrier, do not initialise; escalate, because the wrong share set is recoverable and a new barrier is not.
8Verify the audit device is active before reading a single secret. bao audit list -detailed must show the device and its destination, and a test request must produce a new record at that destination within seconds.
9Re-apply every policy from version control rather than trusting the restored copies. The barrier contains the policies as they were at the restore point, which may predate a correction or may be exactly the over-broad grant that caused the incident. Upload the reviewed documents.
10Re-establish auth method roles from version control in the same pass. Token lifetimes, SecretID constraints and bound networks are configuration, and configuration belongs in the repository. Re-issue workload credentials afterwards, because tokens issued after the restore point no longer exist.
11Reconcile dynamic credentials that outlived the restore. A database user created after the restore point still exists in the database while the restored barrier holds no lease for it. Nothing will ever revoke it. List and remove those users deliberately.
12Rotate every credential the restore rolled backwards. A password rotated after the backup point has been reverted inside OpenBao while the downstream system kept the new value. Rotating forward is the only way to make both sides agree, and it must be done before the workloads return.
13Bring workloads back in tiers and verify each tier before starting the next. Short-lived credential consumers first, cached long-lived material last, with a pause between them long enough to read the audit log.
14Keep the preserved pre-restore directory until the service owner signs off. Delete it only as a scheduled task after the incident review, never as the last step of the incident itself.

4 · Verification

Confirm the procedure actually fixed the problem.

✓bao status on the restored node reports Initialized true and Sealed false, read on the node rather than through a load balancer.
✓bao audit list -detailed shows the expected device, and a new record appears at its destination within seconds of a test request.
✓A grep of the audit destination for a known secret value returns nothing, confirming values are stored HMACed rather than in clear.
✓Every policy returned by bao policy read matches the version-controlled document byte for byte.
✓A workload authenticates with a freshly issued credential and reads its own secret from inside its own container.
✓A deliberately chosen forbidden path is still refused with Code: 403, proving least privilege survived the restore.
✓A dynamic database credential can be issued and revoked end to end, proving the lease machinery works and not only the read path.
✓The database reports zero leftover dynamic roles from before the restore, confirming the reconciliation was completed rather than intended.
✓Every credential on the rollback list has been rotated forward, and the downstream system accepts the new value.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the restore is wrong and the service must return to its pre-restore state, stop the service, move the restored directory aside under its own timestamped name, and move the preserved directory back. This is why nothing was deleted.
↶If the shares do not open the restored barrier, stop. Leave the node sealed, leave both directories in place, and escalate. Initialising to obtain a working service destroys the data you are trying to recover.
↶If policies were uploaded and proved wrong, re-upload the previous documents from version control and re-test both an allowed and a refused operation before continuing.
↶If workloads were returned too early and the audit log is unreadable, stop the consumers again rather than pressing on. Observability during a restore is not optional.
↶If a rotation was performed against the wrong system, treat the old value as compromised and rotate again forward. Never restore an old credential to make two systems agree.
↶If rollback is invoked at any point, the service owner must be told before the first command, because returning to the pre-restore state is itself an outage decision.

6 · Escalation

When the runbook isn't enough, contact:

· The share set does not open the restored barrier: escalate to the platform owner and the security owner together and stop typing. This is the single failure mode from which there is no technical recovery.
· The backup archive is incomplete or fails to extract: escalate to the backup owner immediately and preserve the failed attempt for the incident review.
· The restored barrier contains policies that appear to be the cause of the original incident: escalate to the security owner before uploading anything, because the restore has just recreated the exposure.
· Dynamic credentials cannot be reconciled because the downstream owner is unavailable: escalate rather than deleting database roles on your own judgement.
· The restore point is older than the retention any team expected: escalate to the service owner, because the data-loss conversation is now wider than this incident.
· Anyone proposes running an initialisation command to obtain a working service: stop the procedure and escalate. That decision is never an operator decision.

A restore is not a repair. It rewinds the entire service to a moment in the past and discards everything after it: every secret version, every token, every lease, every policy edit and every audit-relevant configuration change. Workloads that were happily authenticated an hour ago will hold tokens the service has never heard of. Passwords rotated last week will revert to their previous values inside OpenBao while the systems that use them keep the new ones.

The second thing to hold on to is where the keys live. The storage backup contains the encrypted keyring and the encrypted root key. It does not contain the unseal key, because the unseal key is split across share holders and is never written to storage. A perfect restore attended by the wrong share set produces a service that is permanently sealed, and the initialisation output states that consequence in as many words: without a threshold of shares, the root key cannot be reconstructed and the service stays sealed for good.

Everything below therefore has one bias. Nothing is deleted until the restore is verified, and no command that writes a fresh barrier is run at any point, for any reason, by anyone.

When this runbook applies, and when it does not

It applies when the storage backend of an OpenBao deployment is genuinely lost or corrupt, the recovery triage has been completed, and the service owner has authorised a restore to a named point in time.

It does not apply when:

The node reports Initialized false and the storage path has not been verified as mounted. That is an unmounted volume until proven otherwise, and restoring over it destroys data that was never lost.
The service is merely sealed. A sealed node is the normal state after a restart. Assemble the share holders and open it.
The deployment uses integrated Raft storage and you intend to restore a cluster snapshot. Snapshot restore is a storage-specific procedure with its own leadership and membership consequences; this page covers restoring the storage volume itself, which is the path that applies to the filesystem and PostgreSQL backends.
The incident is a compromise rather than a loss. Restoring a compromised barrier reinstates the compromise, including whatever grant allowed it. Compromise response comes first; a restore may follow it, but never replaces it.

Blast radius

Action	Reversible?	What it costs if wrong
Reading the backup and its metadata	Yes	Nothing.
Stopping every node	Yes	A full outage for the length of the procedure.
Renaming the live storage directory aside	Only if you kept a copy	This is the copy. Rename, never delete.
Restoring the backup over a fresh directory	Yes, while the preserved copy exists	Everything after the restore point is gone by design.
Unsealing with the wrong share set	Yes	Nothing. It simply will not open.
Running an initialisation command	No	Creates a new barrier and ends any prospect of recovery.
Re-applying policies from version control	Only if you kept a copy	An over-broad document uploaded under pressure becomes the running grant.
Rotating credentials forward	No	The old value stops working everywhere at once. Sequence it.
Deleting the preserved pre-restore directory	No	The last copy of whatever survived. Delete it after sign-off, never during.

Step 1 - Stop every writer and prove it

# On each node that can write to the storage backend.
systemctl stop openbao
systemctl is-active openbao || echo "openbao is stopped on this node"

A single surviving writer during a restore produces a storage tree that starts cleanly and misbehaves days later. Confirm on every member, not on the one you happened to be logged into.

Step 2 - Preserve what is there before you replace it

STAMP=$(date -u +%Y%m%dT%H%M%SZ)
mv /openbao/data "/openbao/data.pre-restore.$STAMP"
mkdir -p /openbao/data

Step 3 - Restore into the fresh directory and fix ownership deliberately

tar -xzf /backup/openbao-data-2026-08-25.tar.gz -C /openbao/data
chown -R openbao:openbao /openbao/data
chmod 700 /openbao/data
find /openbao/data -maxdepth 1 -printf '%M %u %g %p\n'

A restored tree owned by root produces a service that will not start, which is at least loud. A world-readable one produces a service that starts perfectly and is a finding at the next review, which is worse.

Step 4 - Confirm the audit device is declared before the service starts

Audit devices cannot be enabled through the API in this release, so a configuration file with no audit stanza yields a restored service with no audit trail and no runtime remedy. Attempting it anyway produces a refusal that names the reason:

Read-only / Safebao-1, why the audit device must exist in the configuration file

$ bao audit enable file file_path=/openbao/audit/audit.log

Error enabling audit device: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/audit/file
Code: 400. Errors:

* cannot enable audit device via API; use declarative, config-based audit device management instead

Illustrative output

The stanza belongs in the server configuration file, alongside the storage and listener blocks:

audit "file" {
  type    = "file"
  path    = "file/"
  options = { file_path = "/openbao/audit/audit.log" }
}

Upstream documentation writes the same device with two labels, as audit "file" "some-path", and with options as a nested block rather than an assignment. Both forms are accepted by the parser; the shape above is the one that was exercised on the evidence host for this course. Declarative devices are created on the active node during restarts and reload events, so the file is the only thing that decides whether the restored service can be audited at all.

Step 5 - Start the service and read the state before doing anything else

systemctl start openbao
sleep 2
bao status

The expected state is Initialized true with Sealed true. If the node reports Initialized false, the restore did not land where the configuration is looking. Stop, compare the configured storage path against the directory you extracted into, and do not proceed.

read -r -s -p 'unseal share: ' SHARE
bao operator unseal "$SHARE"
unset SHARE

Each share moves the progress counter. When the threshold is met the node reports Sealed false.

Step 7 - Prove the audit device is writing, not merely listed

Read-only / Safebao-1, the audit device must be present before any secret is read

$ bao audit list -detailed

Path     Type    Description    Replication    Options
----     ----    -----------    -----------    -------
file/    file    n/a            replicated     file_path=/openbao/audit/audit.log

Illustrative output

Then make one request and confirm a new record lands at that destination within seconds. Listed and writing are different states, and only the second is an audit trail.

Step 8 - Re-apply policy and roles from version control

The restored barrier holds the policies as they stood at the restore point. That is not the same as the policies you want, and if the incident involved an over-broad grant it is precisely the wrong set.

# Apply the reviewed documents, not whatever the backup happened to hold.
bao policy write app-read /etc/openbao/policies/app-read.hcl
bao policy read app-read

Re-apply auth method roles in the same pass, then issue fresh workload credentials. Tokens issued after the restore point do not exist in the restored barrier, so every consumer needs a new one regardless of what it is currently holding.

Step 9 - Reconcile the credentials the restore left orphaned

A dynamic database user created after the restore point still exists in the database, while the restored barrier holds no lease for it. Nothing will ever expire it, and nothing will ever revoke it.

-- List dynamic users the restored service no longer knows about.
SELECT rolname, rolvaliduntil
  FROM pg_roles
 WHERE rolname LIKE 'v-token-%'
 ORDER BY rolname;

Remove them deliberately, with the database owner present. Then rotate every credential on the rollback list, because a password changed after the backup point has just been reverted inside OpenBao while the downstream system kept the newer value. Rotating forward is the only way to make both sides agree; restoring the old value everywhere is not.

Common pitfalls

Symptom	Cause	Action
`Initialized false` after the restore	Configured storage path is not the restored path	Compare the configuration with the extracted directory
Shares refuse to open the barrier	Wrong share set for this backup point	Escalate; never initialise
Service starts, then hangs	Audit destination missing or full; auditing is fail-closed	Fix the destination and restart
Audit cannot be enabled afterwards	API creation is refused in this release	Add the stanza to the configuration file
Workloads reconnect and fail	Their tokens predate the restore point	Issue fresh credentials, do not widen policy
A rotated password stops working	The restore reverted it inside OpenBao only	Rotate forward, never restore the old value
Orphaned database users linger	Leases for them were lost with the rollback	Reconcile with the database owner

Verification

Verification after a restore is deliberately broader than after any other procedure, because a restore changes everything at once. Read the seal and initialisation state on the node itself. Confirm the audit device is listed and that a fresh record reaches its destination within seconds of a test request, then grep that destination for a known secret value and confirm it returns nothing. Compare every running policy against its version-controlled document. Have a workload authenticate with a newly issued credential and read its own secret from inside its own container, then deliberately request a forbidden path and confirm it is still refused with a 403, which is the only evidence that least privilege survived. Issue and revoke a dynamic credential end to end so the lease machinery is exercised rather than assumed. Finally, confirm the database reports no leftover dynamic roles and that every credential on the rollback list has been rotated forward and accepted by the system that consumes it.

Rollback

Rollback exists because the preserved directory was never deleted. If the restore proves wrong, stop the service, move the restored directory aside under its own timestamped name, and move the preserved one back into place. Tell the service owner before the first command, because returning to the pre-restore state is an outage decision in its own right. If the share set does not open the restored barrier, the correct rollback is to do nothing: leave the node sealed, leave both directories where they are, and escalate. If policies were uploaded and proved wrong, re-upload the previous documents and re-test both an allowed and a refused operation before continuing. A credential rotated against the wrong system is treated as compromised and rotated forward again; the old value is never restored to make two systems agree.