Dynamic credentials: a database login that did not exist a minute ago

A dynamic credential is one that the secrets engine manufactures at the moment a workload asks for it, binds to a lease, and destroys when that lease ends. Nothing is pre-provisioned. The username, the password and the grants behind them all come into existence on request and leave no residue afterwards. That single property changes what a stolen credential is worth, and it changes how long a credential incident lasts.

What the engine does when a workload asks for a credential

The engine holds two things. The first is a privileged bootstrap account on the target database, stored inside the barrier and used for nothing except administering other accounts. The second is a role definition: a template of SQL that says what a credential of this kind is allowed to do. Neither of those is a credential a workload ever sees.

bao secrets enable database

bao write database/config/appdb \
    plugin_name=postgresql-database-plugin \
    allowed_roles=app-readonly \
    connection_url="postgresql://{{username}}:{{password}}@db.lab.example:5432/appdb?sslmode=verify-full&sslrootcert=/etc/pki/internal-root.crt" \
    username=postgres \
    password=lab-bootstrap-pw

The connection_url carries {{username}} and {{password}} as templates rather than literals, so the engine substitutes whatever bootstrap credential it currently holds. That indirection is what makes rotating the bootstrap account possible without rewriting the mount. allowed_roles is the gate that stops an operator from later attaching an unrelated, over-privileged role definition to this connection.

Read the sslmode before either of them. Every credential this mount ever issues, and the bootstrap superuser password itself, travels this connection. sslmode=verify-full with an sslrootcert naming the internal root is the only setting that both encrypts the session and proves the host on the other end is the database you meant. sslmode=disable sends all of it in clear, and require encrypts without authenticating the server, so an attacker who can answer for db.lab.example collects the superuser password on the first connection. You will see disable in lab material, including this course’s own labs, where the traffic never leaves a private container network. It has no place in an estate.

The role definition is the interesting half, because it is the only place where the privileges of every future credential are decided:

CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}'
    VALID UNTIL '{{expiration}}';
GRANT CONNECT ON DATABASE appdb TO "{{name}}";
GRANT USAGE ON SCHEMA public TO "{{name}}";
GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}";

CREATE_SQL=$(cat app-readonly-creation.sql)

bao write database/roles/app-readonly \
    db_name=appdb \
    creation_statements="$CREATE_SQL" \
    default_ttl=2m \
    max_ttl=10m

bao read -format=json database/creds/app-readonly

{
  "lease_id": "database/creds/app-readonly/xoHI541EXoFgKn1OTiusOdd9",
  "lease_duration": 120,
  "renewable": true,
  "data": {
    "password": "[REDACTED]",
    "username": "v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423"
  }
}

Read that response as three separate facts. The data object is the credential itself. The lease_id is the handle that lets anyone with authority destroy it later, and it is not a secret. The lease_duration of 120 seconds is the engine agreeing to keep the credential alive for two minutes, matching the default_ttl set on the role. A workload that stores only the username and password and throws the lease identifier away has kept the dangerous half and discarded the useful half.

flowchart LR
    A["Workload with a policy\ngranting read on the creds path"] --> B["Secrets engine"]
    B --> C["Connects as the\nbootstrap account"]
    C --> D["Executes CREATE ROLE\nwith generated values"]
    D --> E["Lease recorded\nusername and password returned"]
    E --> F["Workload connects\nto PostgreSQL directly"]

Notice where the traffic goes after issuance. The workload talks to PostgreSQL directly, not through the secrets engine. The engine is on the issuance path and the revocation path, never on the query path. That is why a dynamic-credential outage looks like new connections failing while established ones keep working, and it is the single most useful thing to know when triage starts.

Proving the credential is a real database principal

A credential that only the issuer believes in is worth nothing. The value of the design comes from the fact that the generated username is an ordinary PostgreSQL role, subject to every rule the database already enforces. Connect with it and ask the database who it thinks you are:

                   current_user                   |              now
--------------------------------------------------+-------------------------------
 v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423 | 2026-08-26 21:23:43.940523+00

                     rolname                      |     rolvaliduntil
--------------------------------------------------+------------------------
 v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423 | 2026-08-26 21:25:48+00

Two independent expiry boundaries now exist for the same credential. The secrets engine holds a lease that expires at 21:25:43. PostgreSQL holds a rolvaliduntil of 21:25:48, five seconds later, set by the VALID UNTIL '{{expiration}}' clause in the creation statement. The database-side boundary sits slightly beyond the lease boundary so that the two do not race each other, and it is a genuine backstop: even if the secrets engine were unreachable when the lease expired, PostgreSQL would stop accepting that password on its own. Note precisely what VALID UNTIL governs, because it is easy to overstate. It expires the password, not the role. A role whose password has expired can no longer log in, but it still owns objects and still appears in pg_roles.

Revocation as an operation, not a cleanup job

Expiry is the normal path. Revocation is the incident path, and it is a first-class operation rather than a side effect:

$ bao lease revoke database/creds/app-readonly/xoHI541EXoFgKn1OTiusOdd9
All revocation operations queued successfully!

$ psql -U v-token-app-read-... -d appdb -c "SELECT 1;"
psql: error: connection to server at "127.0.0.1", port 5432 failed: FATAL:  role "v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423" does not exist

 leftover_dynamic_roles
------------------------
                      0

The word to notice in the first line is queued. Revocation is asynchronous: the engine has accepted the instruction, not proved it finished. The proof is the second and third results, both read from PostgreSQL rather than from the issuer. The credential does not fail with an authentication error, which would mean the principal still exists with a different password. It fails because the principal is gone entirely, and a count of leftover dynamic roles confirms nothing was orphaned. Verifying revocation inside the issuer is the classic mistake here, and it hides exactly the failure mode you care about: a revocation that could not reach the database.

What changes about incident response

Consider the ordinary version of this incident. A credential appears in a log aggregator, or a laptop is stolen, or a contractor leaves. With a shared static database password, the response is a rotation: change the password, then find and restart every process that held it, in an order nobody has written down, during which the service is partly broken. The work is proportional to the number of consumers, and the exposure window runs from whenever the credential leaked to whenever the last consumer was restarted.

With dynamic credentials the response is a revocation, and it is proportional to nothing. Revoke the single lease if you know which one leaked. Revoke the prefix if you do not, which cancels every outstanding credential of that class and forces each holder to request a fresh one on its next connection attempt. Revoking the token that requested the credentials cascades to every lease that token created, and revoking a parent token cascades through its children as well, so one compromised workload identity can be unwound in a single action.

Three things follow that are worth stating plainly, because teams usually discover them during the first real incident.

• The blast-radius question becomes answerable. With a shared password, “who has this credential” is an archaeology exercise across configuration management, container images and someone’s shell history. With leases it is a query: the outstanding leases under a path, each one attributable through the audit record to the identity that requested it.
• Recovery is self-healing rather than coordinated. After a prefix revocation nobody restarts anything. Each consumer fails its next connection, requests a new credential, and continues. The failure is loud, brief and localised, instead of quiet and estate-wide.
• The dependency moves rather than disappearing. The issuer is now in the path of every new connection an application makes. If it is sealed, unreachable, or refusing the workload’s authentication, no new credentials are issued, and the incident you get is a slow starvation as existing leases expire one by one.

That last point deserves respect rather than a footnote. Dynamic credentials trade a large, permanent exposure for a small, recurring availability dependency. That is usually the right trade, but it is a trade, and the operational work it creates is the subject of the rest of this part.

Production discipline

1. Give the engine its own database account and rotate it. The bootstrap credential is the one static secret in this design. Isolate it to a dedicated login, rotate it at mount time, and treat its loss as a break-glass event with a documented recovery path.
2. Grant in the creation statement, never afterwards. Every privilege a dynamic credential will ever hold has to be written into the role definition. A manual GRANT applied to one generated principal disappears with it and leaves the next one broken.
3. Verify revocation in the target system. Treat the queued response as an instruction accepted, not an outcome. Count dynamic principals in the database on a schedule and alert when the count exceeds the number of outstanding leases.
4. Keep the lease identifier with the credential. A client that discards it cannot revoke early, cannot renew, and cannot be audited against its own consumption. Log the identifier, never the password.
5. Size the role’s max_ttl against the longest legitimate unit of work. A batch job that runs for an hour against a role capped at ten minutes will fail halfway through, and the failure will be blamed on the database.

Cross-course references

• Kubernetes for Production Sysadmins - Part LXV (SecretsSec) covers why a Secret object holding a static database password is readable by anyone who can create a Pod in the namespace, which is the exposure a dynamic credential is designed to remove.
• Linux for Production Sysadmins - Part LXXII (Secrets) covers how credentials reach a process on a host, which is where the username and password from the response above actually land.
• Observability for Production Sysadmins - Part XVIII (AlertingRules) covers writing the rule that turns a growing count of orphaned database principals into a page rather than a discovery during the next audit.

Quiz