Static secrets, and why central storage does not solve rotation

A static secret is a value that some other system already accepts: a database password, an API token, a webhook signing key. Moving it into a secret manager is worth doing, and it is routinely oversold. Being precise about which problems the move solves, and which it leaves entirely untouched, is the difference between a migration that improves security posture and one that produces a very well-audited copy of the same risk.

What centralising a static secret actually fixes

Four things genuinely improve the moment the value stops living in a configuration file.

• Distribution. One authoritative copy replaces a value pasted into an Ansible variable file, a Kubernetes manifest, a CI variable and two engineers’ laptops. The copies were never all updated together; now there is nothing to keep in step.
• Access control. Reading the value becomes a request that a policy evaluates, instead of a filesystem permission that anyone with a shell can bypass by reading the process environment.
• Auditability. Every read leaves a record naming the identity, the path and the time. A configuration file leaves no record of ever having been read.
• Blast radius on host compromise. A host that no longer stores the value on disk stops leaking it in backups, snapshots and support bundles.

Three things do not improve at all. The value itself is unchanged, so an old or weak credential stays old and weak. The downstream system still accepts it, because nothing told the database anything. And every workload that has read it holds a plaintext copy in memory for as long as it runs, which no storage-side control can withdraw.

KV version 2 is a versioned store, not a rotation engine

Enabling the engine and writing a value is deliberately unremarkable:

# Mount the key-value engine at kv/ with versioning enabled.
bao secrets enable -path=kv -version=2 kv

# Write the first version of an application's configuration.
bao kv put kv/app/config db_user=appuser db_password=lab-only-not-real

# Write it again with a different value.
bao kv put kv/app/config db_user=appuser db_password=lab-only-rotated

The second write does not overwrite the first. It creates version 2 and leaves version 1 intact and readable. That behaviour is the point of KV version 2, and it is worth understanding exactly, because it is also the source of two recurring surprises.

$ bao kv metadata get kv/app/config
==== Metadata Path ====
kv/metadata/app/config

============ Metadata ============
Key                         Value
---                         -----
cas_required                false
created_time                2026-08-26T21:23:42.212427564Z
current_version             2
custom_metadata             <nil>
delete_version_after        0s
max_versions                0
oldest_version              0
updated_time                2026-08-26T21:23:42.393137309Z

====== Version 1 ======
Key              Value
---              -----
created_time     2026-08-26T21:23:42.212427564Z
deletion_time    n/a
destroyed        false

====== Version 2 ======
Key              Value
---              -----
created_time     2026-08-26T21:23:42.393137309Z
deletion_time    n/a
destroyed        false

Note the path in the header. The metadata lives at kv/metadata/app/config while the values live under kv/data/app/config, and the command line hides that split from you. max_versions at 0 means no automatic pruning, so every historical value is retained indefinitely until somebody explicitly destroys it. cas_required at false means a blind write always wins, so two concurrent writers silently produce two versions and the loser never learns it lost.

Two of those settings deserve a decision rather than a default. cas_required turns every write into a check-and-set: the writer must state which version it believes is current, and a write against a stale version is refused instead of silently becoming the newest one. Enable it wherever more than one system writes a path, because the alternative is a last-writer-wins race whose loser gets a success response. delete_version_after sets a duration at which each version self-deletes, which is the closest KV version 2 comes to expiring a value and is useful for paths that hold a short-lived bootstrap credential rather than a long-lived configuration.

The other thing the metadata tells you is who could read what. Version 1 is still present, still undeleted and still undestroyed, and any identity whose policy grants read on the data path can ask for it by number. A version history is therefore part of your access surface, not merely a convenience for rollback, and the retention decision belongs with whoever owns the credential rather than with whoever happened to mount the engine.

Delete, undelete, destroy and metadata deletion

Four different removal operations exist and they differ by an order of magnitude in consequence.

# Soft delete: version stays, marked with a deletion time.
bao kv delete kv/app/config

# Bring it back, because nothing was actually removed.
bao kv undelete -versions=2 kv/app/config

# Permanent for the named versions only.
bao kv destroy -versions=1 kv/app/config

The dangerous one is not on that list because it is not a kv subcommand people reach for casually: granting delete on kv/metadata/* in a policy destroys all versions and the metadata for anything under that prefix. Operators who think of delete as the mild permission grant it freely. In KV version 2 it is the most destructive capability in the engine.

The rotation gap, stated precisely

A rotation is a change in two systems that must be ordered correctly. The store holds a value; the downstream system holds whatever it will accept. Writing a new version changes exactly one of those.

flowchart LR
    subgraph M["Secret manager"]
        V1["version 1"] --> V2["version 2\nwritten by an operator"]
    end
    subgraph DB["Downstream system"]
        P["password unchanged\nstill accepts version 1"]
    end
    V2 -.->|"no connection exists"| P
    A["Running application"] -->|"read once at startup"| V1

The diagram carries the whole argument. Nothing links the write in the manager to the credential the database accepts, so version 2 is a value that authenticates against nothing until somebody separately changes the database. Meanwhile the application read version 1 at startup and holds it in memory, so it is not affected by the write either, in any direction. Three copies of the truth now exist and none of them agrees with the others.

That is the rotation gap, and it produces two classic outages. Change the database first and every running application breaks immediately. Change the store first and nothing happens at all, which is worse, because the change looks successful and the failure arrives at the next restart, possibly weeks later, at the hands of someone who was not involved.

The gap is also why a leaked static secret is not contained by anything the store can do. Destroying every version removes the value from the manager and leaves the downstream system accepting it exactly as before. Containment always requires an action in the system that grants access, and the store’s job in that operation is only to stop handing the value to anything new. Read the two systems as two independent inventories that happen to hold the same string, and the ordering problems stop being surprising.

Production discipline

1. Set max_versions when you mount the engine. Ten is a defensible history for rollback. Unbounded is an archive of credentials nobody remembers creating.
2. Never treat a new version as a completed rotation. Rotation is complete when the downstream system rejects the previous value. Verify that, from the downstream side, or you have not rotated anything.
3. Order the change: add, deploy, remove. Where the downstream system supports two valid credentials at once, write the new one, let workloads pick it up, then invalidate the old one. Where it does not, you have a maintenance window, and you should know that before you plan the work.
4. Prefer a generating engine wherever one exists. A static value that a human chose is a fallback for systems that cannot issue credentials on demand, not a target state.

Cross-course references

• Ansible for Production Sysadmins - Part XXI (Secrets Management) covers the variable files that a key-value engine replaces, and the no_log gaps that survive the move.
• Terraform for Production Sysadmins - Part XI (State Security and Lifecycle) covers why a value read into state is recorded there regardless of how it was marked, which is the same problem of uncontrolled copies described here.
• Git, CI/CD & GitOps for Infrastructure Engineers - Part LXXXII (GitOps Secrets) covers referencing a stored secret from a manifest instead of committing its value.

Quiz