Lab 21: Issue dynamic database credentials

Objective

A static database password has one property that no amount of vaulting fixes: it is the same password tomorrow. Storing it in a secret manager improves distribution and auditing, and leaves the blast radius of a leak exactly where it was. If the credential is copied into a log, a core dump or a screenshot, the only remedy is a rotation that touches every consumer at once.

A dynamic credential inverts that. The database login does not exist until an authenticated caller asks for one, it belongs to that caller alone, it carries an expiry the database itself enforces, and it can be destroyed on demand without touching any other consumer. In this lab you build one end to end against a real PostgreSQL 17 server and then you prove the four claims that make it worth the complexity.

By the end you will be able to answer, with evidence, the question that decides whether a team adopts dynamic credentials: when we revoke, what exactly disappears, and how long does an already-open session keep working.

Architecture

Two containers share a private Docker network. PostgreSQL runs with a bootstrap superuser that only OpenBao ever uses. OpenBao runs in development mode, which keeps its own state in memory and unseals itself, because the subject of this lab sits above the barrier rather than inside it. The application in the story never holds a database password at all: it holds a token, asks for a credential, and receives one that did not exist a second earlier.

flowchart LR
    A["Caller with a token"] --> B["bao read\ndatabase/creds/app-readonly"]
    B --> C["OpenBao runs the\ncreation statements"]
    C --> D["PostgreSQL creates\na new login role"]
    B --> E["Lease: id, duration,\nrenewable"]
    E --> F["bao lease revoke"]
    F --> G["OpenBao runs the\nrevocation and drops the role"]

The credential and the lease are issued together. The lease is the handle that makes the credential disposable: it carries an identifier, a duration, and a flag saying whether it can be extended. Revoking the lease is not a note in a database somewhere, it makes OpenBao connect to PostgreSQL as the bootstrap user and remove the role it created.

Requirements

Docker with a working daemon and permission to run containers as your user.
OpenBao 2.6.x and PostgreSQL 17, both pulled as container images. Nothing is installed on the host.
Roughly 75 minutes, about 500 MB of free disk for the two images, and an outbound network path to pull them.
No out-of-band access requirement. This lab does not touch SSH, the firewall, the primary interface, or /etc/fstab. Both containers and the network carry an rbpki- prefix.

Scenario

Your reporting service needs read access to the application database. It runs on twelve hosts, it restarts often, and the last time its password leaked into an exception trace the rotation took two days and an outage. You have been asked to make the next leak uninteresting. That means the reporting service must never hold a long-lived password, and you must be able to prove that withdrawing a credential really withdraws it.

Tasks

Task 1 — Record the starting state and build the lab tree

LAB="$HOME/rbpki-lab21"
# The secret manager writes its storage as root inside the container, so hand
# the tree back before removing it from the host.
if [ -d "$LAB" ]; then
  docker run --rm -v "$LAB:/w" alpine:3.22 \
    sh -c 'rm -rf /w/* /w/.[!.]* 2>/dev/null || true'
fi
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

docker ps -a --format '{{.Names}}' | sort > "$LAB/state.pre-lab"
docker network ls --format '{{.Name}}' | sort >> "$LAB/state.pre-lab"
grep -c . "$LAB/state.pre-lab"

Task 2 — Start PostgreSQL and OpenBao on a private network

LAB="$HOME/rbpki-lab21"
docker network create rbpki-net-21

docker run -d --name rbpki-pg-21 --network rbpki-net-21 \
  --network-alias db.lab.example \
  -e POSTGRES_PASSWORD=lab-bootstrap-pw \
  -e POSTGRES_DB=appdb \
  postgres:17-alpine

docker run -d --name rbpki-bao-21 --network rbpki-net-21 --cap-add=IPC_LOCK \
  -e BAO_DEV_ROOT_TOKEN_ID=rbpki-lab-root-not-a-real-token \
  -e BAO_DEV_LISTEN_ADDRESS=0.0.0.0:8200 \
  openbao/openbao:latest server -dev

sleep 8
docker inspect -f '{{.State.Status}}' rbpki-pg-21 rbpki-bao-21

docker exec -e BAO_ADDR=http://127.0.0.1:8200 \
  -e BAO_TOKEN=rbpki-lab-root-not-a-real-token rbpki-bao-21 bao version

That prints the build under test, in the form OpenBao v2.6.2 (dd9c19c37a878cf4a81b18efb8d6f0599c7da923), committed 2026-08-18T15:48:19Z. Record it. A dynamic credential lab is version-sensitive, because the list of supported database plugins changes between releases.

Task 3 — Enable the database engine and configure the connection

docker exec -e BAO_ADDR=http://127.0.0.1:8200 \
  -e BAO_TOKEN=rbpki-lab-root-not-a-real-token rbpki-bao-21 \
  bao secrets enable database

docker exec -e BAO_ADDR=http://127.0.0.1:8200 \
  -e BAO_TOKEN=rbpki-lab-root-not-a-real-token rbpki-bao-21 \
  bao write database/config/appdb \
    plugin_name=postgresql-database-plugin \
    allowed_roles=app-readonly \
    connection_url='postgresql://{{username}}:{{password}}@db.lab.example:5432/appdb?sslmode=disable' \
    username=postgres \
    password=lab-bootstrap-pw

The engine reports Success! Enabled the database secrets engine at: database/ and the configuration write reports Success! Data written to: database/config/appdb.

Three parts of that configuration decide how safe this is. connection_url is a template: the {{username}} and {{password}} placeholders are filled from the stored bootstrap credential, and keeping them as placeholders is what allows the credential to be rotated later without rewriting the URL. allowed_roles is an allow-list, so a role that is not named there cannot use this connection even if someone creates it. And sslmode=disable is a lab shortcut, acceptable only because the traffic never leaves a private Docker network; against a real database this is where the connection would present and verify a certificate.

Task 4 — Define what a dynamic credential is allowed to do

LAB="$HOME/rbpki-lab21"
cat > "$LAB/creation-statements.sql" <<'EOF'
CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
GRANT CONNECT ON DATABASE appdb TO "{{name}}";
GRANT USAGE ON SCHEMA public TO "{{name}}";
GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}";
EOF
cat "$LAB/creation-statements.sql"

CREATE_SQL=$(cat "$LAB/creation-statements.sql")

docker exec -e BAO_ADDR=http://127.0.0.1:8200 \
  -e BAO_TOKEN=rbpki-lab-root-not-a-real-token rbpki-bao-21 \
  bao write database/roles/app-readonly \
    db_name=appdb \
    creation_statements="$CREATE_SQL" \
    default_ttl=2m \
    max_ttl=10m

Passing the SQL through a shell variable rather than typing it inline keeps the quoting honest. The statements contain both double quotes, around the generated role name, and single quotes, around the generated password, so any attempt to inline them into one command line ends in an escaping mistake that is difficult to see and easy to ship.

The write reports Success! Data written to: database/roles/app-readonly.

Read the SQL as a security boundary rather than as plumbing. Everything the credential will ever be able to do is in those four statements. There is no INSERT, no UPDATE, no CREATE, and no membership of another role, so a leaked reporting credential cannot write to the database no matter who holds it. Adding a privilege here is a change that belongs in review, because it silently widens every credential the role has ever issued and every one it will issue in future.

The two lifetimes are deliberately short. default_ttl=2m is what an unrenewed credential gets; max_ttl=10m is the ceiling that renewal cannot pass. Two minutes is a teaching value that lets you watch an expiry inside a lab session. Production values are chosen from how long a unit of work takes, not from how long the process runs.

What the three templates expand to, and who enforces the expiry

OpenBao substitutes three values into the creation statements at issue time. {{name}} becomes a generated login name built from the role name and a timestamp, which is why issued usernames start with v-token- and end in digits. {{password}} becomes a freshly generated secret. {{expiration}} becomes a timestamp derived from the lease duration, formatted for the database.

That third substitution is what makes the arrangement fail safe. VALID UNTIL is a PostgreSQL constraint, checked by PostgreSQL at authentication time. If OpenBao is unreachable, unresponsive or sealed when the lease should have been revoked, the login still stops working at the time the database was told, because the database is the one enforcing it. Leave VALID UNTIL out of a creation statement and you have moved the entire expiry guarantee into the availability of your secret manager.

Task 5 — Request a credential that did not exist a second ago

LAB="$HOME/rbpki-lab21"
docker exec -e BAO_ADDR=http://127.0.0.1:8200 \
  -e BAO_TOKEN=rbpki-lab-root-not-a-real-token rbpki-bao-21 \
  bao read -format=json database/creds/app-readonly > "$LAB/credential.json"

DYN_USER=$(grep -o '"username": *"[^"]*"' "$LAB/credential.json" | head -1 | cut -d'"' -f4)
DYN_PASS=$(grep -o '"password": *"[^"]*"' "$LAB/credential.json" | head -1 | cut -d'"' -f4)
LEASE_ID=$(grep -o '"lease_id": *"[^"]*"' "$LAB/credential.json" | head -1 | cut -d'"' -f4)

printf 'user  : %s\n' "$DYN_USER"
printf 'lease : %s\n' "$LEASE_ID"

Configuration changethe credential response, trimmed to the fields an application uses— Each read of this path mints a new login; nothing here is reusable.

$ docker exec -e BAO_ADDR=http://127.0.0.1:8200 -e BAO_TOKEN=rbpki-lab-root-not-a-real-token rbpki-bao-21 bao read -format=json database/creds/app-readonly

{
"lease_id": "database/creds/app-readonly/xoHI541EXoFgKn1OTiusOdd9",
"lease_duration": 120,
"renewable": true,
"data": {
  "password": "[REDACTED]",
  "username": "v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423"
}
}

Illustrative output

The username encodes where it came from, which is the property that makes an incident tractable: a login name in a PostgreSQL log tells you which OpenBao role issued it and roughly when. The lease_duration of 120 is the two minutes you configured, expressed in seconds. The lease identifier begins with the path the secret was requested from, which is what allows a whole class of credentials to be revoked by prefix later.

Read the path a second time and you get a different username and a different lease. There is no caching and no reuse. An application that calls this on every request creates a login on every request, which is why real clients cache the credential in memory for the life of its lease.

Task 6 — Prove the credential authenticates against the real database

LAB="$HOME/rbpki-lab21"
DYN_USER=$(grep -o '"username": *"[^"]*"' "$LAB/credential.json" | head -1 | cut -d'"' -f4)
DYN_PASS=$(grep -o '"password": *"[^"]*"' "$LAB/credential.json" | head -1 | cut -d'"' -f4)

docker exec -e PGPASSWORD="$DYN_PASS" rbpki-pg-21 \
  psql -h 127.0.0.1 -U "$DYN_USER" -d appdb -c "SELECT current_user, now();"

Read-only / SafePostgreSQL 17 answering as the freshly minted role— The database, not OpenBao, is confirming the identity.

$ docker exec -e PGPASSWORD="$DYN_PASS" rbpki-pg-21 psql -h 127.0.0.1 -U "$DYN_USER" -d appdb -c "SELECT current_user, now();"

                   current_user                   |              now
--------------------------------------------------+-------------------------------
v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423 | 2026-08-26 21:23:43.940523+00
(1 row)

Illustrative output

This is the proof that matters. current_user is PostgreSQL reporting the identity it authenticated, so the credential is not a token that some middleware translates, it is a real database login with a real password.

Task 7 — Prove the role exists in PostgreSQL, with its expiry

docker exec -e PGPASSWORD=lab-bootstrap-pw rbpki-pg-21 \
  psql -h 127.0.0.1 -U postgres -d appdb \
  -c "SELECT rolname, rolvaliduntil FROM pg_roles WHERE rolname LIKE 'v-%';" \
  > "$HOME/rbpki-lab21/pg-roles-before.txt" 2>&1
cat "$HOME/rbpki-lab21/pg-roles-before.txt"

Read-only / Safethe catalogue view, read as the bootstrap superuser— rolvaliduntil is the database enforcing the expiry OpenBao asked for.

$ docker exec -e PGPASSWORD=lab-bootstrap-pw rbpki-pg-21 psql -h 127.0.0.1 -U postgres -d appdb -c "SELECT rolname, rolvaliduntil FROM pg_roles WHERE rolname LIKE 'v-%';"

                     rolname                      |     rolvaliduntil
--------------------------------------------------+------------------------
v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423 | 2026-08-26 21:25:48+00
(1 row)

Illustrative output

Compare the two timestamps from Tasks 6 and 7. The session authenticated at 21:23:43 and the role is valid until 21:25:48, which is a little over the two minutes of the lease. The margin exists because OpenBao gives the database a slightly later boundary than the lease, so a credential does not become unusable a fraction of a second before its owner expects. The important observation is that the constraint is present at all: this login has an end date recorded inside PostgreSQL.

Task 8 — Inspect the lease

Read-only / Safethe lease behind the credential— The lease is the handle you will use to destroy the credential.

$ docker exec -e BAO_ADDR=http://127.0.0.1:8200 -e BAO_TOKEN=rbpki-lab-root-not-a-real-token rbpki-bao-21 bao lease lookup database/creds/app-readonly/xoHI541EXoFgKn1OTiusOdd9

Key             Value
---             -----
expire_time     2026-08-26T21:25:43.891514518Z
issue_time      2026-08-26T21:23:43.891514368Z
last_renewal    <nil>
path            database/creds/app-readonly
renewable       true
ttl             1m59s

Illustrative output

Substitute your own lease identifier from credential.json. renewable true means the holder may ask for more time, up to the role’s maximum. Two properties of renewal surprise people: the increment is measured from the moment of the renewal rather than added to the end of the current term, and it is advisory, so the engine may grant less than was asked for. A client that assumes it received what it requested will occasionally find its credential expiring earlier than planned.

Task 9 — Revoke the lease and prove the credential is gone

LAB="$HOME/rbpki-lab21"
LEASE_ID=$(grep -o '"lease_id": *"[^"]*"' "$LAB/credential.json" | head -1 | cut -d'"' -f4)
DYN_USER=$(grep -o '"username": *"[^"]*"' "$LAB/credential.json" | head -1 | cut -d'"' -f4)
DYN_PASS=$(grep -o '"password": *"[^"]*"' "$LAB/credential.json" | head -1 | cut -d'"' -f4)

{
  docker exec -e BAO_ADDR=http://127.0.0.1:8200 \
    -e BAO_TOKEN=rbpki-lab-root-not-a-real-token rbpki-bao-21 \
    bao lease revoke "$LEASE_ID" 2>&1
  echo '--- the credential used again, after revocation ---'
  docker exec -e PGPASSWORD="$DYN_PASS" rbpki-pg-21 \
    psql -h 127.0.0.1 -U "$DYN_USER" -d appdb -c "SELECT 1;" 2>&1
  echo '--- how many dynamic roles remain ---'
  docker exec -e PGPASSWORD=lab-bootstrap-pw rbpki-pg-21 \
    psql -h 127.0.0.1 -U postgres -d appdb \
    -c "SELECT count(*) AS leftover_dynamic_roles FROM pg_roles WHERE rolname LIKE 'v-%';" 2>&1
} > "$LAB/revocation-proof.txt"

cat "$LAB/revocation-proof.txt"

The revoke command reports All revocation operations queued successfully!. The reused credential then fails in a very specific way:

psql: error: connection to server at "127.0.0.1", port 5432 failed: FATAL:  role "v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423" does not exist

Read that error closely, because the wording is the whole lesson. PostgreSQL does not say the password is wrong or that the account is locked. It says the role does not exist. Revocation dropped the login rather than disabling it, so there is nothing left to re-enable, nothing to un-expire, and no residue for an attacker to work with. The final query in the capture confirms it from the other direction, returning a leftover_dynamic_roles count of 0.

One honest caveat: dropping a role does not terminate a session that is already open. A connection authenticated before revocation continues until the client or the server closes it. If your threat model requires cutting live sessions, revocation must be paired with terminating the backends for that role.

Task 10 — Capture the deliverables

cd "$HOME/rbpki-lab21"
ls -l creation-statements.sql credential.json pg-roles-before.txt revocation-proof.txt
grep -c 'does not exist' revocation-proof.txt
grep -c 'lease_id' credential.json

Validation

The query in Task 6 returns one row and current_user matches the username in credential.json. If psql reports that authentication failed, the credential was consumed from a second, different bao read and the username and password belong to different leases.
pg-roles-before.txt contains exactly one role beginning v- and a non-empty rolvaliduntil. An empty rolvaliduntil means VALID UNTIL '{{expiration}}' is missing from the creation statement and the database is enforcing no expiry at all.
grep -c 'does not exist' revocation-proof.txt returns 1. If it returns 0, look for a line saying the password authentication failed instead, which means the role survived and only its password changed, and the revocation statements need reviewing.
The final count in revocation-proof.txt is 0. A non-zero count means an earlier bao read left an orphaned credential, which you can clear with bao lease revoke -prefix database/creds/app-readonly.
All four deliverables exist and are non-empty.

Expected Outcome

$HOME/rbpki-lab21/
├── creation-statements.sql
├── credential.json
├── pg-roles-before.txt
├── revocation-proof.txt
└── state.pre-lab

You can now show a sceptical team the full life of a database credential in under three minutes: issued on request, authenticated by the database itself, constrained by a VALID UNTIL that survives an OpenBao outage, and destroyed on command with nothing left behind. You can also state the limit accurately, which is that an established session outlives its credential.

Troubleshooting

The configuration write fails with a connection error. OpenBao is dialling db.lab.example:5432 over the Docker network. Confirm the alias with docker inspect -f '{{.NetworkSettings.Networks}}' rbpki-pg-21 and confirm PostgreSQL finished starting; the image needs several seconds after the container reports as running.

bao read database/creds/app-readonly fails with a permission error from PostgreSQL. The bootstrap user cannot create roles. Confirm you configured username=postgres and that POSTGRES_PASSWORD matches the password you stored.

The credential works but the query returns no permission on a table. The creation statements grant SELECT ON ALL TABLES at the moment the role is created, so a table created afterwards is not covered. Production roles add a ALTER DEFAULT PRIVILEGES statement, or grant through a group role.

Every credential expires while you are still typing. default_ttl=2m is doing its job. Raise it for exploration, but put it back before drawing any conclusions about renewal behaviour.

Cleanup

LAB="$HOME/rbpki-lab21"

# 1. Revoke anything this lab may have left outstanding, before the server goes.
docker exec -e BAO_ADDR=http://127.0.0.1:8200 \
  -e BAO_TOKEN=rbpki-lab-root-not-a-real-token rbpki-bao-21 \
  bao lease revoke -prefix database/creds/app-readonly || true

# 2. Count what remains inside PostgreSQL, for the record.
docker exec -e PGPASSWORD=lab-bootstrap-pw rbpki-pg-21 \
  psql -h 127.0.0.1 -U postgres -d appdb \
  -c "SELECT count(*) FROM pg_roles WHERE rolname LIKE 'v-%';" || true

# 3. Stop and forget both services and the network.
docker rm -f rbpki-bao-21 rbpki-pg-21 || true
docker network rm rbpki-net-21 || true

# 4. Compare against the Task 1 capture, then remove the directory.
docker ps -a --format '{{.Names}}' | sort > "$LAB/state.post-lab"
docker network ls --format '{{.Name}}' | sort >> "$LAB/state.post-lab"
diff "$LAB/state.pre-lab" "$LAB/state.post-lab" || true
# The secret manager writes its storage as root inside the container, so hand
# the tree back before removing it from the host.
if [ -d "$LAB" ]; then
  docker run --rm -v "$LAB:/w" alpine:3.22 \
    sh -c 'rm -rf /w/* /w/.[!.]* 2>/dev/null || true'
fi
rm -rf "$LAB"

Confirm restoration by running docker ps -a --format '{{.Names}}' | grep -c rbpki- and docker network ls --format '{{.Name}}' | grep -c rbpki-; both must return 0. ls -d "$HOME/rbpki-lab21" must report that the directory no longer exists, and the diff in step 4 must print nothing before the removal.

Production notes

Check the plugin list before designing. This OpenBao release ships database plugins for Cassandra, InfluxDB, MySQL and MariaDB, PostgreSQL, and Valkey. Engines that exist in other products, including MSSQL, MongoDB, Oracle and Redis, are not built in here. Design against what the binary actually contains.
Choose the lifetime from the work, not the process. A batch job that runs for four minutes wants a four-minute credential. A long-lived service wants a short lease it renews, so that a process which stops renewing loses access quickly.
Static roles are the answer for accounts you cannot create. Where an application must connect as a fixed username, a static role keeps that one username and rotates its password on a schedule instead of minting new logins.
Alarm on orphaned dynamic roles. A count of v-% roles that only ever rises means revocation is failing somewhere. It is a two-line query and it is the cheapest early warning you will get.
Pair revocation with session termination when it matters. Dropping a role does not close an open connection. Write that step into the compromise runbook rather than discovering it during one.

What You Learned

The login is created on demand and belongs to one caller. The username records the role and the moment it was issued, which makes a database log traceable back to a request.
The creation statements are the privilege boundary. Everything a dynamic credential can ever do is in that SQL, and widening it retroactively widens every credential the role issues.
VALID UNTIL moves expiry enforcement into the database. The credential stops working on time even if the secret manager is unavailable.
Revocation drops the role. PostgreSQL reports role ... does not exist, not a password failure, and the leftover count returns to zero.
A lease is a handle, and its prefix is a fleet-wide control. bao lease revoke -prefix retires a whole class of credentials in one command.

Objectives

Objective

Architecture

Requirements

Scenario

Tasks

Task 1 — Record the starting state and build the lab tree

Task 2 — Start PostgreSQL and OpenBao on a private network

Task 3 — Enable the database engine and configure the connection

Task 4 — Define what a dynamic credential is allowed to do

Task 5 — Request a credential that did not exist a second ago

Task 6 — Prove the credential authenticates against the real database

Task 7 — Prove the role exists in PostgreSQL, with its expiry

Task 8 — Inspect the lease

Task 9 — Revoke the lease and prove the credential is gone

Task 10 — Capture the deliverables

Validation

Expected Outcome

Troubleshooting

Cleanup

Production notes

What You Learned

Deliverables

Verification status