Lab 25: Recover from a secret-manager outage

Objective

A sealed secret manager is the most instructive outage in this course, because almost nothing about it is intuitive. Some applications stop within one request. Others run for hours and then fall over together. A database credential the secret manager issued last week keeps working perfectly, and the secret manager cannot revoke it even though it created it.

You will build that situation deliberately: a real OpenBao instance with durable storage and a Shamir quorum, a real database behind it, one credential already issued, and two consumers written to fail in different ways. Then you will seal it, take the estate apart while it is broken, and recover it.

The output of the lab is not the unseal. It is the preparedness list you write at the end, which is the artefact that decides whether the next occurrence is a fifteen-minute event or a permanent one.

Architecture

OpenBao holds a static secret in a key-value mount and a dynamic credential engine pointed at PostgreSQL. Two consumers read the static secret in deliberately different styles. A credential that OpenBao has already issued lives inside PostgreSQL as an ordinary login role with an expiry.

flowchart LR
    A["rbpki-fetcher-25\nreads the secret every loop"] --> B["rbpki-bao-25\nOpenBao barrier"]
    C["rbpki-cacher-25\nreads it once at start-up"] --> B
    B -- "creates, renews\nand revokes the role" --> D["rbpki-pg-25\nPostgreSQL 17"]
    C -- "uses the issued credential\nwithout asking OpenBao" --> D

The important edge is the bottom one. Once a dynamic credential has been handed out, the path from the application to the database does not pass through the secret manager at all. That single fact explains most of what you are about to observe.

Requirements

• Docker 29.x or an equivalent engine, able to pull openbao/openbao, postgres:17-alpine and alpine:3.22, and to create a user-defined bridge network and named volumes.
• A Linux host with a writable home directory and roughly 700 MB free for images and volumes.
• Everything created is named with the rbpki- prefix and the -25 suffix, so it cannot collide with anything already running.
• No out-of-band access requirement. This lab does not touch SSH, the firewall, the primary interface, or /etc/fstab.

Scenario

You are on call. At 02:40 the paging system reports that the checkout service cannot start new workers, while the workers that are already running are serving traffic normally. Ten minutes later a batch job that has run hourly for a year fails for the first time. The database is healthy. The network is healthy. Someone restarted the secret-manager node during a kernel patch and nobody unsealed it afterwards.

You will now build that estate, break it in the same way, and produce the evidence an incident review will ask for.

Tasks

Task 1 — Record the starting state

LAB="$HOME/rbpki-lab-25"
# The secret manager writes its storage as root inside the container, so hand
# the tree back before removing it from the host.
if [ -d "$LAB" ]; then
  docker run --rm -v "$LAB:/w" alpine:3.22 \
    sh -c 'rm -rf /w/* /w/.[!.]* 2>/dev/null || true'
fi
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

# Record what Cleanup must restore.
{ docker ps -a --format '{{.Names}}' | sort
  docker network ls --format '{{.Name}}' | sort
  docker volume ls --format '{{.Name}}' | sort; } > "$LAB/state.pre-lab"
wc -l "$LAB/state.pre-lab"

Task 2 — Stand up a secret manager that can actually be sealed

A development-mode server starts unsealed and keeps its data in memory, which makes the outage in this lab impossible to stage. Use durable storage and a real barrier.

cat > "$LAB/bao.hcl" <<'EOF'
storage "raft" {
  path    = "/openbao/data"
  node_id = "rbpki-bao-25"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = true
}

audit "file" {
  type    = "file"
  path    = "file/"
  options = { file_path = "/openbao/audit/audit.log" }
}

api_addr      = "http://rbpki-bao-25:8200"
cluster_addr  = "http://rbpki-bao-25:8201"
disable_mlock = true
ui            = false
EOF

docker network create rbpki-net-25
docker volume create rbpki-bao-25-config
docker volume create rbpki-bao-25-data
docker volume create rbpki-bao-25-audit
docker run --rm -v rbpki-bao-25-config:/c -v rbpki-bao-25-data:/d \
  -v rbpki-bao-25-audit:/a -v "$LAB:/src:ro" alpine:3.22 \
  sh -c 'cp /src/bao.hcl /c/bao.hcl && chmod 0777 /c /d /a'

docker run -d --name rbpki-bao-25 --network rbpki-net-25 \
  -e BAO_ADDR=http://127.0.0.1:8200 \
  -v rbpki-bao-25-config:/openbao/config \
  -v rbpki-bao-25-data:/openbao/data \
  -v rbpki-bao-25-audit:/openbao/audit \
  openbao/openbao:latest server
sleep 5
docker logs rbpki-bao-25 | tail -20

tls_disable is here because the lab needs one variable under test, not two. Never write that line on a machine anyone else can reach.

The audit device is declared in the configuration file rather than enabled afterwards, and that is not a stylistic choice in this release. Try it the way most documentation shows and read the refusal.

$ docker exec rbpki-bao-25 bao audit enable file file_path=/tmp/bao-audit.log

Error enabling audit device: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/audit/file
Code: 400. Errors:

* cannot enable audit device via API; use declarative, config-based audit device management instead

This matters for the outage you are about to cause: if audit is configuration rather than runtime state, then a node that comes back without its configuration file comes back without an audit trail, and you will not be able to reconstruct who did what during the incident.

Task 3 — Initialise, and take custody of the quorum

docker exec rbpki-bao-25 bao operator init \
  -key-shares=3 -key-threshold=2 | tee "$LAB/init.out"

The command prints three unseal shares and an initial root token, then a paragraph that is worth reading rather than scrolling past:

Vault initialized with 3 key shares and a key threshold of 2. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 2 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key. Without at least 2 keys to
reconstruct the root key, Vault will remain permanently sealed!

OpenBao’s command line still says “Vault” in a number of strings. It is quoted here exactly as the tool prints it, because that is the text you will search for when you are reading somebody’s incident notes.

K1=$(awk '/Unseal Key 1:/ {print $4}' "$LAB/init.out")
K2=$(awk '/Unseal Key 2:/ {print $4}' "$LAB/init.out")
K3=$(awk '/Unseal Key 3:/ {print $4}' "$LAB/init.out")
ROOT_TOKEN=$(awk '/Initial Root Token:/ {print $4}' "$LAB/init.out")

$ docker exec rbpki-bao-25 bao operator unseal "$K1"

docker exec rbpki-bao-25 bao operator unseal "$K2"
docker exec rbpki-bao-25 bao login "$ROOT_TOKEN"

The barrier opens on the second share, not the first. bao login stores the token inside the container, so the docker exec calls in the rest of the lab do not have to carry it.

Task 4 — Load the estate: a static secret and one issued lease

docker exec rbpki-bao-25 bao secrets enable -path=kv kv-v2
docker exec rbpki-bao-25 bao kv put kv/app/config \
  api_key=lab-only-not-real db_host=rbpki-pg-25

docker run -d --name rbpki-pg-25 --network rbpki-net-25 \
  -e POSTGRES_PASSWORD=lab-bootstrap-pw -e POSTGRES_DB=appdb \
  postgres:17-alpine
for _ in $(seq 1 30); do
  docker exec rbpki-pg-25 pg_isready -U postgres -d appdb && break
  sleep 1
done
docker exec rbpki-pg-25 psql -U postgres -d appdb -c \
  "CREATE TABLE ledger (id serial PRIMARY KEY, note text);
   INSERT INTO ledger (note) VALUES ('opening balance');"

Now connect OpenBao to the database and issue exactly one credential. Keep the JSON: the username and password in it are the only copy.

docker exec rbpki-bao-25 bao secrets enable database
docker exec rbpki-bao-25 bao write database/config/appdb \
  plugin_name=postgresql-database-plugin \
  allowed_roles=app-readonly \
  connection_url='postgresql://{{username}}:{{password}}@rbpki-pg-25:5432/appdb?sslmode=disable' \
  username=postgres password=lab-bootstrap-pw
CREATE_SQL=$(cat <<'SQL'
CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}";
SQL
)
docker exec rbpki-bao-25 bao write database/roles/app-readonly \
  db_name=appdb \
  creation_statements="$CREATE_SQL" \
  default_ttl=15m max_ttl=30m
docker exec rbpki-bao-25 bao read -format=json database/creds/app-readonly \
  > "$LAB/cred.json"
grep -E 'lease_id|username|lease_duration' "$LAB/cred.json"

Read the creation_statements value once more. VALID UNTIL writes the expiry into PostgreSQL itself. The database, not OpenBao, is what enforces the lifetime of this credential from here on.

Task 5 — Two consumers that will fail differently

cat > "$LAB/fetch-every-time.sh" <<'EOF'
#!/bin/sh
# Reads the secret from OpenBao on every iteration. Caches nothing.
while true; do
  TS=$(date -u +%H:%M:%S)
  if V=$(bao kv get -field=api_key kv/app/config 2>&1); then
    echo "$TS OK   fetched=$V"
  else
    echo "$TS FAIL $(echo "$V" | tr '\n' ' ')"
  fi
  sleep 2
done
EOF

cat > "$LAB/read-once.sh" <<'EOF'
#!/bin/sh
# Reads the secret once at start-up, then serves it from memory.
CACHED=$(bao kv get -field=api_key kv/app/config) || {
  echo "startup FAILED: could not read the secret"
  exit 1
}
echo "startup OK: cached the secret"
while true; do
  echo "$(date -u +%H:%M:%S) OK   cached=$CACHED"
  sleep 2
done
EOF
chmod +x "$LAB/fetch-every-time.sh" "$LAB/read-once.sh"

docker run -d --name rbpki-fetcher-25 --network rbpki-net-25 \
  -e BAO_ADDR=http://rbpki-bao-25:8200 -e BAO_TOKEN="$ROOT_TOKEN" \
  -v "$LAB/fetch-every-time.sh:/app.sh:ro" \
  --entrypoint /app.sh openbao/openbao:latest
docker run -d --name rbpki-cacher-25 --network rbpki-net-25 \
  -e BAO_ADDR=http://rbpki-bao-25:8200 -e BAO_TOKEN="$ROOT_TOKEN" \
  -v "$LAB/read-once.sh:/app.sh:ro" \
  --entrypoint /app.sh openbao/openbao:latest
sleep 6
docker logs rbpki-fetcher-25 | tail -2
docker logs rbpki-cacher-25 | tail -2

Both should be printing OK. These two scripts are the whole argument of the lab: the same secret, the same server, one difference in when it is read.

Task 6 — Seal it, and read the failure applications actually see

date -u +'%Y-%m-%dT%H:%M:%SZ' > "$LAB/outage-timeline.md"
echo "seal issued" >> "$LAB/outage-timeline.md"
docker exec rbpki-bao-25 bao operator seal

$ docker exec rbpki-bao-25 bao kv get kv/app/config

Code: 503. Errors:

* Vault is sealed

Capture it, then look at the two consumers.

docker exec rbpki-bao-25 bao kv get kv/app/config \
  > "$LAB/sealed-failure.txt" 2>&1
cat "$LAB/sealed-failure.txt"
sleep 6
echo "--- fetcher ---"; docker logs rbpki-fetcher-25 | tail -3
echo "--- cacher  ---"; docker logs rbpki-cacher-25  | tail -3

The fetcher is failing. The cacher is not, and will not, for as long as it stays up. Neither of them has changed; only the moment at which they chose to read the secret has.

Task 7 — Map the blast radius while it is still broken

Three questions decide how bad a sealed secret manager is: does it answer at all, does already-issued material still work, and can it be managed.

docker exec rbpki-bao-25 bao status; echo "status exit=$?"

The server answers. Sealed reads true, Unseal Progress reads zero out of the threshold, and the command’s exit status is non-zero. Record that exit status: an unauthenticated status endpoint that still responds while everything else fails is exactly what a health check should be watching, and a check that only tests “is the port open” will report this outage as healthy.

Pull the issued credential out of the JSON you kept in Task 4, then use it against the database directly.

DB_USER=$(awk -F'"' '/username/ {print $4}' "$LAB/cred.json")
DB_PASS=$(awk -F'"' '/password/ {print $4}' "$LAB/cred.json")
echo "using role: $DB_USER"

$ docker run --rm --network rbpki-net-25 -e PGPASSWORD="$DB_PASS" postgres:17-alpine psql -h rbpki-pg-25 -U "$DB_USER" -d appdb -tAc 'SELECT current_user, count(*) FROM ledger'

The query returns the dynamic role name and the row count, from a database whose credentials were issued by a system that is currently refusing every request. Nothing about that is a bug. PostgreSQL was handed a login role with an expiry and is enforcing it on its own.

Now try to manage that same credential.

LEASE=$(awk -F'"' '/lease_id/ {print $4}' "$LAB/cred.json")

$ docker exec rbpki-bao-25 bao lease renew "$LEASE"

$ docker exec rbpki-bao-25 bao lease revoke "$LEASE"

Both are refused with the same sealed-barrier error as the secret read. This is the part that turns an outage into a security problem: during the seal you can neither extend the credential that is about to expire nor withdraw one you have just discovered was leaked. The credential is outside your control in both directions.

{
  echo "# Blast radius"
  echo "- status endpoint: answers, reports sealed, exits non-zero"
  echo "- secret reads: refused, HTTP 503"
  echo "- issued database credential: still works against PostgreSQL"
  echo "- lease renew: refused"
  echo "- lease revoke: refused"
  echo "- new credential issuance: refused"
  echo "- consumers that cache at start-up: unaffected while they stay up"
  echo "- consumers that read per request: failing continuously"
} > "$LAB/blast-radius.md"

Task 8 — The restart trap

The cacher is comfortable because it is still the same process it was before the seal. Take that away from it.

docker restart rbpki-cacher-25
sleep 5
docker logs rbpki-cacher-25 | tail -3

It cannot start. Its first action is a secret read, the barrier is closed, and the script exits. This is the mechanism behind the scenario in the introduction: existing workers serve traffic, new ones never become ready, and any event that would normally be harmless becomes fatal. A node drain, an autoscaler, a liveness probe with a short timeout, a routine deploy: each of them converts a partial outage into a total one, and none of them looks related in a timeline.

{
  date -u +'%Y-%m-%dT%H:%M:%SZ'
  echo "cacher restarted during the seal and failed to start"
} >> "$LAB/outage-timeline.md"

Task 9 — Recover with a quorum, and verify from the consumer side

docker exec rbpki-bao-25 bao operator unseal "$K1"
docker exec rbpki-bao-25 bao operator unseal "$K3"
docker exec rbpki-bao-25 bao status
{
  date -u +'%Y-%m-%dT%H:%M:%SZ'
  echo "unsealed with shares 1 and 3"
} >> "$LAB/outage-timeline.md"

Note that the recovery used shares 1 and 3, not the 1 and 2 used at initialisation. Any two of the three reconstruct the key, which is the property that makes a quorum survivable when one holder is unreachable at 03:00.

Verifying on the server is not enough. The instance reporting Sealed false tells you the barrier opened; it does not tell you the estate recovered. Read the consumers instead.

sleep 6
docker logs rbpki-fetcher-25 | tail -2
docker restart rbpki-cacher-25 && sleep 5 && docker logs rbpki-cacher-25 | tail -2
docker exec rbpki-bao-25 bao lease renew "$LEASE"

The fetcher’s lines return to OK with no intervention, the cacher starts successfully this time, and the lease renews. Those three observations, in that order, are what closes the incident.

Task 10 — Capture the deliverables

cd "$LAB"
cat > unseal-ceremony.md <<'EOF'
# Unseal ceremony

Shares: 3. Threshold: 2. Any two shares reconstruct the unseal key,
which decrypts the root key held encrypted in storage.
Holder A: share 1. Holder B: share 2. Holder C: share 3.
No holder stores a second share. No share is stored with the root token.
Recovery order: page two holders, submit shares one at a time, confirm
Unseal Progress advances, then verify from a consumer, not from status.
EOF

cat > preparedness-gaps.md <<'EOF'
# What should have existed before the seal

1. A monitor on the sealed field, not on the TCP port.
2. Share custody recorded by name, tested at least annually.
3. The server configuration under version control, because the audit
   device is configuration and cannot be restored through the API.
4. Consumers that tolerate a read failure by retrying rather than by
   exiting, so a restart during an outage is survivable.
5. A documented decision about which credentials may outlive the
   secret manager, and how they are withdrawn when it is unavailable.
EOF

ls -l outage-timeline.md sealed-failure.txt blast-radius.md \
      unseal-ceremony.md preparedness-gaps.md

Validation

• docker exec rbpki-bao-25 bao status reports Sealed as false and exits zero.
• sealed-failure.txt contains the text Vault is sealed. An empty file means the capture in Task 6 ran before the seal took effect.
• docker logs rbpki-fetcher-25 | tail -2 shows OK lines, and the same command run during Task 6 showed FAIL lines.
• docker logs rbpki-cacher-25 contains one startup FAILED line from the restart during the seal and a later successful start.
• The deliverables outage-timeline.md, sealed-failure.txt, blast-radius.md, unseal-ceremony.md and preparedness-gaps.md exist and are non-empty.

Expected Outcome

$HOME/rbpki-lab-25/
├── bao.hcl
├── blast-radius.md
├── cred.json
├── fetch-every-time.sh
├── init.out
├── outage-timeline.md
├── preparedness-gaps.md
├── read-once.sh
├── sealed-failure.txt
├── state.pre-lab
└── unseal-ceremony.md

You can now answer the question an incident review will put to you: which of our services survive a sealed secret manager, for how long, and what is the first routine event that turns the survivors into casualties.

Troubleshooting

The server container exits immediately after Task 2. The named volumes were not made writable by the container user. Re-run the alpine:3.22 helper that copies the configuration and sets the mode, then docker start rbpki-bao-25 and read docker logs again.

The startup log begins with warnings that ownership could not be changed. That is the image entrypoint reporting it cannot chown a mount, which happens whenever the container user is not root. It does not stop the server. Confirm the log later reports that the server started.

bao operator init reports that the instance is already initialised. A previous attempt left data in the volume. Remove the containers and all three rbpki-bao-25-* volumes as shown in Cleanup, then start Task 2 again from the top.

The dynamic credential is refused in Task 7. Its lease expired while you were reading. PostgreSQL enforces VALID UNTIL regardless of anything OpenBao is doing, so unseal, issue a fresh credential with Task 4’s last command, and repeat Task 6 and Task 7 promptly.

Cleanup

LAB="$HOME/rbpki-lab-25"

# 1. Stop and forget every container this lab started.
docker rm -f rbpki-fetcher-25 rbpki-cacher-25 rbpki-bao-25 rbpki-pg-25

# 2. Remove the network and the three named volumes.
docker network rm rbpki-net-25
docker volume rm rbpki-bao-25-config rbpki-bao-25-data rbpki-bao-25-audit

# 3. Compare the estate against the Task 1 capture.
cp "$LAB/state.pre-lab" /tmp/rbpki-25-before
{ docker ps -a --format '{{.Names}}' | sort
  docker network ls --format '{{.Name}}' | sort
  docker volume ls --format '{{.Name}}' | sort; } > /tmp/rbpki-25-after
diff /tmp/rbpki-25-before /tmp/rbpki-25-after && echo "estate restored"
rm -f /tmp/rbpki-25-before /tmp/rbpki-25-after

# 4. Remove the lab directory, which still holds the unseal shares.
# The secret manager writes its storage as root inside the container, so hand
# the tree back before removing it from the host.
if [ -d "$LAB" ]; then
  docker run --rm -v "$LAB:/w" alpine:3.22 \
    sh -c 'rm -rf /w/* /w/.[!.]* 2>/dev/null || true'
fi
rm -rf "$LAB"

Cleanup is complete when diff prints estate restored with no preceding lines, and ls "$HOME/rbpki-lab-25" reports no such directory. Step 4 is not optional: init.out contains every share and the root token, and leaving it on disk is the single worst artefact this lab produces.

Production notes

• Monitor the sealed field, and alert on it separately from availability. A sealed instance answers its status endpoint, so any check that stops at “the port is open” will call this healthy for the entire outage.
• Decide in advance how long your consumers may cache a secret. A long cache buys you time during an outage and costs you time during a compromise, because a revoked secret keeps working until the cache turns over. There is no setting that is right for both.
• Keep the server configuration in version control and deploy it with the node. In this release the audit device cannot be re-enabled through the API, so a node rebuilt without its configuration is a node with no audit trail during the very incident you will be asked about.
• Rehearse the ceremony with the real holders, not with a document. A quorum that has never been assembled is a quorum you find out about at 03:00, and the failure mode is not an outage but a permanent loss.

What You Learned

• A sealed barrier fails uniformly and immediately. Every path returns the same error, which makes the diagnosis fast once you have seen it and baffling the first time.
• Issued credentials outlive their issuer. The database enforces the expiry it was given, so an outage neither breaks a live dynamic credential nor lets you revoke one.
• Caching changes when you fail, not whether you fail. The consumer that never noticed the outage is the one that dies on its next restart, which is why restart-driven failures cluster hours after the actual event.
• Recovery is a people problem with a command attached. Two share holders and one status check is the entire technical procedure; the part that fails in practice is knowing who holds what.