Lab 19: Deploy a secret manager and complete the unseal ceremony

Objective

Almost every secret-manager tutorial starts a development server, prints a root token to the terminal and moves on. That server keeps everything in memory, unseals itself, and has no audit trail. It teaches the API and hides the operations. The two things that actually decide whether a secret manager survives a bad night are the ones a development server removes: who holds the key shares, and whether you can prove what the service was asked to do.

In this lab you run OpenBao 2.6.x the way a production instance runs it. You write a server configuration file that declares its storage, its listener and its audit device. You start it and find it uninitialised and sealed, because that is what a real server does on first boot. You then perform the initialisation ceremony that splits the unseal key into three shares with a threshold of two, you unseal with two of them, and you watch the progress counter move.

By the end you will be able to answer a question that comes up in every incident review of a secrets outage: what exactly is a quorum of key shares unlocking, and who has to be in the room before the service can serve a single request again.

Architecture

One container runs the OpenBao server. Three host directories are mounted into it: a configuration directory holding the HCL file, a data directory that becomes the file storage backend, and an audit directory that receives the audit log. Nothing is published to the host network. Every command in this lab reaches the server through docker exec, which keeps the listener private even though the lab disables TLS on it.

The thing worth drawing is not the container layout, it is the key hierarchy that the ceremony manipulates. Getting this chain the right way round is the difference between an operator who can explain a seal event and one who repeats a slogan.

flowchart LR
    A["Key shares\n2 of 3 submitted"] --> B["Unseal key\nreconstructed in memory"]
    B --> C["Root key\ndecrypted"]
    C --> D["Keyring\ndecrypted"]
    D --> E["Secret data\nencrypted on disk"]

A quorum of shares reconstructs the unseal key. The unseal key decrypts the root key. The root key decrypts the keyring, and an encryption key from the keyring decrypts the data. The shares do not reconstruct the root key directly, and the root key is never written to storage in the clear. Sealing the server discards the root key from memory, which is why a sealed server can read its own files and still serve nothing.

Requirements

• Docker with a working daemon, and permission to run containers as your user.
• OpenBao 2.6.x, pulled as the openbao/openbao container image. Nothing is installed on the host.
• About 200 MB of free disk in your home directory and roughly 70 minutes.
• No out-of-band access requirement. This lab does not touch SSH, the firewall, the primary interface, or /etc/fstab. Every resource it creates is named with an rbpki- prefix so that it cannot collide with anything you already run.

Scenario

You are the operator on call for a platform team that has decided to stop storing database passwords in configuration management. The secret manager is going in this week. Your architect has asked for one thing before any application is pointed at it: a written ceremony that shows how the service comes back after a restart, and who has to take part. You are building the rehearsal instance that the ceremony will be written against.

Tasks

Task 1 — Record the starting state and build the lab tree

LAB="$HOME/rbpki-lab19"
# The secret manager writes its storage as root inside the container, so hand
# the tree back before removing it from the host.
if [ -d "$LAB" ]; then
  docker run --rm -v "$LAB:/w" alpine:3.22 \
    sh -c 'rm -rf /w/* /w/.[!.]* 2>/dev/null || true'
fi
rm -rf "$LAB"
mkdir -p "$LAB/config" "$LAB/data" "$LAB/audit"
cd "$LAB"

# Record what Cleanup must restore.
docker ps -a --format '{{.Names}}' | sort > "$LAB/state.pre-lab"
docker network ls --format '{{.Name}}' | sort >> "$LAB/state.pre-lab"
grep -c . "$LAB/state.pre-lab"

# The server process writes into these two directories as root.
chmod 777 "$LAB/data" "$LAB/audit"

The two chmod 777 calls are a lab convenience. They exist because the container runs as root in this exercise and the host directories belong to you. A production deployment gives the OpenBao service account its own data directory with mode 0700 and never shares it.

Task 2 — Write the server configuration

cat > "$LAB/config/bao.hcl" <<'EOF'
storage "file" {
  path = "/openbao/data"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = true
}

audit "file" {
  type    = "file"
  path    = "file/"
  options = { file_path = "/openbao/audit/audit.log" }
}

disable_mlock = true
api_addr      = "http://127.0.0.1:8200"
EOF

cp "$LAB/config/bao.hcl" "$LAB/rbpki-bao.hcl"
grep -n 'audit' "$LAB/config/bao.hcl"

Four decisions are encoded in that file. The storage "file" stanza puts the encrypted barrier on disk instead of in memory, which is what makes the initialisation ceremony meaningful. The listener disables TLS, which is acceptable only because the port is never published outside the container. disable_mlock is set because the container image cannot always lock memory. The audit stanza is the important one: it declares the audit device in configuration, which is the only way to get one in this release.

Task 3 — Start the server and read its uninitialised state

docker network create rbpki-net-19

docker run -d --name rbpki-bao-19 --network rbpki-net-19 \
  --cap-add=IPC_LOCK --user root \
  -v "$LAB/config:/openbao/config" \
  -v "$LAB/data:/openbao/data" \
  -v "$LAB/audit:/openbao/audit" \
  openbao/openbao:latest server -config=/openbao/config/bao.hcl

sleep 6
docker inspect -f '{{.State.Status}}' rbpki-bao-19

$ docker exec -e BAO_ADDR=http://127.0.0.1:8200 rbpki-bao-19 bao status

Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            2.6.2
Commit Date        2026-08-18T15:48:19Z
Storage Type       file
HA Enabled         false

Read the first four rows carefully. Initialized false means no barrier exists yet, so there is nothing to unseal. Total Shares 0 and Threshold 0 are not defaults waiting to be used, they are the honest answer that no key has been split. The command exits non-zero here, which is deliberate: a health probe that only checks for exit status 0 correctly treats an uninitialised server as not ready. The storage type reads file, confirming that the configuration file was loaded rather than a development default.

Task 4 — Initialise the barrier with three shares and a threshold of two

docker exec -e BAO_ADDR=http://127.0.0.1:8200 rbpki-bao-19 \
  bao operator init -key-shares=3 -key-threshold=2 > "$LAB/init.raw" 2>&1

umask 077
grep 'Unseal Key 1:' "$LAB/init.raw" | awk '{print $NF}' > "$LAB/share-1"
grep 'Unseal Key 2:' "$LAB/init.raw" | awk '{print $NF}' > "$LAB/share-2"
grep 'Unseal Key 3:' "$LAB/init.raw" | awk '{print $NF}' > "$LAB/share-3"
grep 'Initial Root Token:' "$LAB/init.raw" | awk '{print $NF}' > "$LAB/root-token"
chmod 600 "$LAB/share-1" "$LAB/share-2" "$LAB/share-3" "$LAB/root-token"

# The deliverable keeps the prose and drops the key material.
sed -E 's/([A-Za-z0-9+/]{20,})/[REDACTED-KEY-MATERIAL]/g' "$LAB/init.raw" \
  > "$LAB/init-ceremony.txt"

The redacted transcript is the part you keep. Here is what it says once the key material is removed:

$ docker exec -e BAO_ADDR=http://127.0.0.1:8200 rbpki-bao-19 bao operator init -key-shares=3 -key-threshold=2

Unseal Key 1: [REDACTED-KEY-MATERIAL]
Unseal Key 2: [REDACTED-KEY-MATERIAL]
Unseal Key 3: [REDACTED-KEY-MATERIAL]

Initial Root Token: s.[REDACTED-KEY-MATERIAL]

Vault initialized with 3 key shares and a key threshold of 2. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 2 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key. Without at least 2 keys to
reconstruct the root key, Vault will remain permanently sealed!

Two details deserve attention. First, the message says “Vault” throughout. OpenBao is a fork of HashiCorp Vault and its command-line output still carries the original product name in several strings. Quote it as it is printed rather than correcting it in your runbooks, because an operator searching a transcript will search for what the terminal actually said.

Second, the phrase “securely distribute the key shares” is the whole design. A threshold of two out of three means no single person can bring the service up alone, and no single person losing a laptop can take it down permanently. Choose the share holders before the ceremony, not after.

Task 5 — Unseal with two shares and watch the counter

docker exec -e BAO_ADDR=http://127.0.0.1:8200 rbpki-bao-19 \
  bao operator unseal "$(cat "$LAB/share-1")" > "$LAB/unseal-1.txt" 2>&1

docker exec -e BAO_ADDR=http://127.0.0.1:8200 rbpki-bao-19 \
  bao operator unseal "$(cat "$LAB/share-2")" > "$LAB/unseal-2.txt" 2>&1

cat "$LAB/unseal-1.txt" "$LAB/unseal-2.txt" > "$LAB/unseal-progress.txt"
grep -E 'Sealed|Threshold|Unseal Progress' "$LAB/unseal-progress.txt"

$ docker exec -e BAO_ADDR=http://127.0.0.1:8200 rbpki-bao-19 bao operator unseal

Sealed             true
Threshold          2
Unseal Progress    1/2

Unseal Progress 1/2 is the row to put in the runbook. It tells the operator holding the second share that the ceremony is genuinely under way and that their share is the one that finishes it. Shares may be submitted in any order and from any number of terminals, because the server keeps the partial reconstruction in memory against a nonce until the threshold is reached.

After the second share the server reports Sealed false and prints the cluster name and cluster identifier it generated during initialisation. At that moment the root key is in memory and the service starts answering requests.

Task 6 — Prove the audit device came from configuration

BAO_TOKEN="$(cat "$LAB/root-token")"
export BAO_TOKEN

docker exec -e BAO_ADDR=http://127.0.0.1:8200 -e BAO_TOKEN="$BAO_TOKEN" rbpki-bao-19 \
  bao audit list -detailed > "$LAB/audit-device.txt" 2>&1
cat "$LAB/audit-device.txt"

# The device is already writing, because it was present before initialisation.
wc -l "$LAB/audit/audit.log"

$ docker exec -e BAO_ADDR=http://127.0.0.1:8200 -e BAO_TOKEN="$BAO_TOKEN" rbpki-bao-19 bao audit list -detailed

Path     Type    Description    Replication    Options
----     ----    -----------    -----------    -------
file/    file    n/a            replicated     file_path=/openbao/audit/audit.log

The audit log already has content, because the device was active before the initialisation request arrived. That ordering is the point. An audit device enabled after the first secret is written has a gap in it exactly where the interesting events are.

Task 7 — Try to enable an audit device through the API

$ docker exec -e BAO_ADDR=http://127.0.0.1:8200 -e BAO_TOKEN="$BAO_TOKEN" rbpki-bao-19 bao audit enable file file_path=/openbao/audit/api.log

Error enabling audit device: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/audit/file
Code: 400. Errors:

* cannot enable audit device via API; use declarative, config-based audit device management instead

You are running as root, so this is not an authorisation failure. The server option that permits API-created audit devices defaults to off in this release, and the documented reason is that the API route lets any sufficiently privileged caller create files at arbitrary paths on the host or send audit records to arbitrary network addresses. Declaring the device in configuration puts that decision back with whoever controls the configuration file.

The upstream documentation shows the stanza with two labels, as audit "file" "file/", and with a nested options block written without an equals sign. The single-label form with an explicit path key that you used in Task 2 is the form verified against this release. If your server refuses to start after editing the stanza, the header shape is the first thing to check.

Task 8 — Seal the server, prove the outage, then bring it back

docker exec -e BAO_ADDR=http://127.0.0.1:8200 -e BAO_TOKEN="$BAO_TOKEN" rbpki-bao-19 \
  bao operator seal

docker exec -e BAO_ADDR=http://127.0.0.1:8200 -e BAO_TOKEN="$BAO_TOKEN" rbpki-bao-19 \
  bao kv get kv/app/config > "$LAB/seal-outage.txt" 2>&1
cat "$LAB/seal-outage.txt"

$ docker exec -e BAO_ADDR=http://127.0.0.1:8200 -e BAO_TOKEN="$BAO_TOKEN" rbpki-bao-19 bao kv get kv/app/config

Error making API request.

URL: GET http://127.0.0.1:8200/v1/sys/internal/ui/mounts/kv/app/config
Code: 503. Errors:

* Vault is sealed

Note that kv/ was never mounted on this server, and the path still returns 503 rather than an error about an unknown mount. A sealed server refuses the request before it resolves which engine would have handled it, which is a useful diagnostic: if a caller reports 503 on a path you know exists, stop looking at the mount and look at the seal status.

Note the status code as well. A 403 would mean the caller was known and refused; 503 means the service cannot answer anyone. That distinction matters when you are reading an application’s error logs at three in the morning and deciding whether you have a policy problem or an availability problem.

Now repeat the ceremony to bring the service back, which is the rehearsal your architect asked for:

docker exec -e BAO_ADDR=http://127.0.0.1:8200 rbpki-bao-19 \
  bao operator unseal "$(cat "$LAB/share-2")"
docker exec -e BAO_ADDR=http://127.0.0.1:8200 rbpki-bao-19 \
  bao operator unseal "$(cat "$LAB/share-3")"

docker exec -e BAO_ADDR=http://127.0.0.1:8200 rbpki-bao-19 bao status | grep -E 'Sealed|Initialized'

Shares two and three work just as well as one and two. Any quorum of the threshold size is sufficient, which is what makes a 2-of-3 split survivable when one holder is unreachable.

Task 9 — Capture the deliverables

cd "$HOME/rbpki-lab19"
cat unseal-1.txt unseal-2.txt > unseal-progress.txt
ls -l rbpki-bao.hcl init-ceremony.txt unseal-progress.txt audit-device.txt seal-outage.txt

# Prove the redaction worked before you share the transcript anywhere.
grep -c 'REDACTED-KEY-MATERIAL' init-ceremony.txt

Validation

• docker exec -e BAO_ADDR=http://127.0.0.1:8200 rbpki-bao-19 bao status reports Initialized true and Sealed false, and exits 0. A non-zero exit with Sealed true means the second share was not accepted, usually because a trailing newline was captured with it.
• The same status output shows Total Shares 3 and Threshold 2. Any other pair means the initialisation flags did not reach the server, and the ceremony must be redone from an empty data directory.
• bao audit list -detailed lists exactly one device at path file/ with the file_path option pointing at /openbao/audit/audit.log. If it reports that no audit devices are enabled, the audit stanza was not parsed and the server is running without a trail.
• wc -l "$HOME/rbpki-lab19/audit/audit.log" returns a non-zero count. A zero-length file means the container could not write to the mounted directory.
• The five deliverables exist, are non-empty, and grep -c 'REDACTED-KEY-MATERIAL' init-ceremony.txt returns at least 4. A count of 0 means you copied the raw ceremony output and must delete it.

Expected Outcome

$HOME/rbpki-lab19/
├── audit/
│   └── audit.log
├── config/
│   └── bao.hcl
├── data/
├── audit-device.txt
├── init-ceremony.txt
├── init.raw
├── rbpki-bao.hcl
├── root-token
├── seal-outage.txt
├── share-1
├── share-2
├── share-3
├── state.pre-lab
└── unseal-progress.txt

You now have a rehearsed ceremony rather than a set of instructions. You can state how many people must be reachable before the service returns, you can show that the audit device is part of the server’s configuration rather than a runtime action someone might forget, and you can demonstrate the difference between a sealed server and an unauthorised caller from the status code alone.

Troubleshooting

The container exits within seconds of starting. Read docker logs rbpki-bao-19. The two common causes are a syntax error in the HCL, which the server names by line, and a permission failure on /openbao/data or /openbao/audit. Re-run the chmod 777 from Task 1 and start it again.

bao operator init reports that the server is already initialised. The data directory survived from an earlier attempt. Remove the container, delete $HOME/rbpki-lab19/data, recreate it, and start again. Initialising is a one-time operation against a given storage backend.

An unseal command is accepted but the progress counter does not move. The share you submitted was already counted. Submitting the same share twice does not advance the threshold. Check that share-1 and share-2 really differ with cmp share-1 share-2.

Every command reports a connection refused. The server binds inside the container and nothing is published to the host, so commands must run through docker exec exactly as written. Running a host-installed bao binary against 127.0.0.1:8200 will not find it.

Cleanup

LAB="$HOME/rbpki-lab19"

# 1. Stop and forget the service.
docker rm -f rbpki-bao-19 || true
docker network rm rbpki-net-19 || true

# 2. Compare what exists now with what existed before the lab.
docker ps -a --format '{{.Names}}' | sort > "$LAB/state.post-lab"
docker network ls --format '{{.Name}}' | sort >> "$LAB/state.post-lab"
diff "$LAB/state.pre-lab" "$LAB/state.post-lab" || true

# 3. Remove the lab directory, including the shares and the root token.
# The secret manager writes its storage as root inside the container, so hand
# the tree back before removing it from the host.
if [ -d "$LAB" ]; then
  docker run --rm -v "$LAB:/w" alpine:3.22 \
    sh -c 'rm -rf /w/* /w/.[!.]* 2>/dev/null || true'
fi
rm -rf "$LAB"

Confirm the host is as you found it: docker ps -a --format '{{.Names}}' | grep -c rbpki- returns 0, docker network ls --format '{{.Name}}' | grep -c rbpki- returns 0, and ls -d "$HOME/rbpki-lab19" reports that the directory does not exist. The diff in step 2 should print nothing before the directory is deleted; any line it prints is a resource this lab created and did not remove.

Production notes

• File storage is a lab choice, not a deployment choice. The file backend is deprecated for removal in a future OpenBao release and supports no high availability. A production deployment uses integrated Raft storage or the PostgreSQL backend with high availability enabled.
• A sealed node provides no standby cover. A server in a sealed state cannot act as a standby, so a cluster whose nodes restart into a sealed state has lost capacity until a human arrives with shares. This is the strongest argument for automatic unsealing in production, with recovery keys held under the same ceremony discipline as unseal shares.
• Split the shares across people and locations, and rehearse. A 2-of-3 split held by three people who all work in the same office is a 2-of-3 split with one failure domain. Rehearse the ceremony on a schedule, because the first time you find out that a share holder has left the company should not be during an outage.
• Publish the audit volume in your monitoring. Auditing is fail-closed. Free space on the audit filesystem belongs on the same dashboard as the service’s own availability.

What You Learned

• A non-development server starts uninitialised and sealed, and that is correct. The status rows Initialized false, Total Shares 0 and Threshold 0 describe a server with no barrier yet, not a broken one.
• The share quorum reconstructs the unseal key, not the root key. Unseal key decrypts root key, root key decrypts keyring, keyring decrypts data. Storage outside the barrier is ciphertext.
• Audit devices in this release come from configuration. The API route returns a 400 with the message about declarative, config-based audit device management, and that default exists because the API route can write files anywhere on the host.
• Sealed is 503, denied is 403. Reading that distinction correctly turns a confused incident into a two-minute triage.
• The root token is a break-glass credential. Use it to build a real authentication path, then put it away.