OpenBao concretely: storage, the barrier, and the unseal ceremony

OpenBao v2.6.2 is a concrete implementation of the six-part model, and the part it does most distinctively is the one no architecture diagram shows: the boundary between the process and its own persistence layer. Everything OpenBao writes is encrypted before it leaves the process, which means the disk, the database, or the Raft log is a place to keep ciphertext and nothing more.

The barrier, and what the storage backend is allowed to see

OpenBao calls its encryption layer the barrier. The barrier is responsible for encrypting and decrypting all data, and the storage backend sits outside it and is considered untrusted. That last phrase is not marketing. It is a design constraint with a testable consequence: an operator with root on the storage host, a database administrator with full access to the backing tables, or an attacker who exfiltrates a filesystem snapshot obtains encrypted blobs and learns nothing about their contents.

Four storage backends ship in v2.6.2, and only four:

If your notes say Consul, etcd, S3 or DynamoDB, those notes came from HashiCorp Vault. None of them exist in OpenBao. The lab configuration below uses the filesystem backend precisely because it is the simplest thing that persists across a restart, and it carries a deprecation you must not carry into production.

storage "file" {
  path = "/openbao/data"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = true
}

disable_mlock = true
api_addr      = "http://127.0.0.1:8200"

That tls_disable = true is a lab-only setting and an explicit anti-pattern in production: it disables transport encryption entirely, so every token and every secret value crosses the network in clear text and any host on the path can read them. The correct configuration terminates TLS on the listener with a certificate the clients already trust. Keep the deliberate weakness confined to a disposable container.

The key hierarchy, one level at a time

The hierarchy is four levels deep, and getting the order wrong is the single most common mistake in secret-manager documentation.

flowchart TD
    S["Shamir shares\nheld by separate people"] -->|"threshold reached"| U["Unseal key"]
    U -->|"decrypts"| R["Root key"]
    R -->|"decrypts"| K["Keyring\n(encryption keys)"]
    K -->|"encrypts and decrypts"| D["Every value in storage"]

Read it upwards to see why it is built this way. Data is encrypted with an encryption key held in the keyring. The keyring is encrypted by the root key. The root key is encrypted by the unseal key. The unseal key is what Shamir’s scheme splits into shares. The shares therefore reconstruct the unseal key, not the root key, and any source telling you that the shares rebuild the root key has skipped a level.

Sealing locks the data by discarding the root key from memory. Nothing on disk changes when a node seals. What changes is that the process can no longer decrypt the keyring, so it can no longer decrypt anything at all, and it refuses every request until a quorum of humans puts the unseal key back together.

The ceremony, as it actually runs

A freshly started server that has never been initialised reports its emptiness plainly, and exits non-zero while doing so:

$ bao status
Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Version            2.6.2
Storage Type       file
HA Enabled         false

Initialisation is the one irreversible moment in the life of the cluster. It generates the root key, splits the unseal key, and prints material that is never stored anywhere and never shown again:

$ bao operator init -key-shares=3 -key-threshold=2
Unseal Key 1: <REDACTED>
Unseal Key 2: <REDACTED>
Unseal Key 3: <REDACTED>
Initial Root Token: s.<REDACTED>

Vault initialized with 3 key shares and a key threshold of 2. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 2 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key. Without at least 2 keys to
reconstruct the root key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum
of existing unseal keys shares. See "bao operator rotate-keys" for more
information.

The output says “Vault” because OpenBao inherited those strings from the fork point and has not rewritten them all. Quote it as you find it rather than tidying it up, because an operator grepping logs needs to recognise the real text.

Shares are submitted one at a time, in any order, by different people from different terminals. After the first share the node is still sealed and reports its progress:

$ bao operator unseal
Sealed             true
Threshold          2
Unseal Progress    1/2

After the second share, Sealed flips to false and the node begins serving. Three shares exist and two are required, so any one holder can be on holiday, and any one share can be lost, without the cluster becoming unrecoverable. Lose two of three and the data is gone permanently, because nothing anywhere stores the root key.

The root token is break-glass, not a service credential

Initialisation also prints an initial root token. It carries the root policy, which cannot be modified or removed, and it is the only credential in a new cluster that can do anything at all. It exists to solve exactly one problem: creating the first authentication method and the first policy so that ordinary credentials can be issued.

After that bootstrap it should not survive. A root token in a configuration management repository, a CI variable, or an engineer’s shell profile is a permanent, unscoped, unattributable credential, and every audit record it produces says only that root did something. The correct end state is a cluster where no valid root token exists, and where a new one is generated on demand from a quorum of unseal-key holders when a genuine emergency requires it. That generation step is itself the control: it needs several people, it is slow on purpose, and it leaves a record.

Production discipline

1. Separate the share holders. Three shares in one password manager, under one account, is a one-of-one threshold wearing a costume. Different people, different storage, different physical locations.
2. Rehearse the ceremony before you need it. The first time an operator runs bao operator unseal should not be during an unplanned reboot at 04:00. Rehearse it quarterly and time how long it takes to assemble a quorum.
3. Record who holds which share, and revisit it on every leaver. Use bao operator rotate-keys to reissue the share set when a holder departs, which requires a quorum of the existing shares.
4. Choose Raft unless PostgreSQL is already a first-class citizen. Integrated storage removes an external dependency from your tier-zero service. Never take the filesystem backend past a lab; it is scheduled for removal in v2.7.0.
5. Plan auto-unseal deliberately. Recovery keys used with auto-unseal cannot decrypt the root key, so they are not sufficient to unseal the cluster if the unsealing mechanism itself is unavailable. Know which failure that leaves you exposed to before you rely on it.

Cross-course references

• Linux for Production Sysadmins - Part VII (systemd) covers the unit ordering and restart policy that decides whether a rebooted node sits sealed and idle or is unsealed by a documented procedure.
• Kubernetes for Production Sysadmins - Part LXV (Secrets Security) covers etcd encryption at rest, which solves a narrower version of the same problem the barrier solves here.
• Observability for Production Sysadmins - Part XI (Blackbox Monitoring) covers the synthetic probe that turns a silently sealed node into an alert before an application notices.

Quiz