Reference architecture: workload secrets

The test of a workload secrets design is not how short the credentials are. It is what happens when you trace them backwards. Follow each credential to the thing that authorised it, and keep going. A design that works terminates in something a platform mints and nobody can copy. A design that does not terminates in a file, and every short lifetime above that file is decoration.

The chain, read backwards

Start at the far end, with the database connection an application has open right now, and walk towards the origin.

• The password it is using was created moments ago and is stored nowhere. It came into existence when the credential was requested and will be destroyed when the lease ends.
• The right to request it came from a token that carries a policy and a lease of its own, not from membership of a group or a line in a configuration file.
• The token was issued by an authentication method after the workload presented a credential it did not choose and cannot edit.
• That credential was minted by the platform the workload runs on, which observed properties of the running process rather than reading a value the process supplied.
• The platform’s authority to make that statement rests on its own key material, which is the control plane’s problem and is governed by the PKI architecture of the previous lesson.

At that last step the chain stops, because there is nothing an operator copied onto the host. That is what secret zero termination means in practice, and it is a property you can check by walking the list rather than a claim about a product.

Now walk the same list for a design where the workload authenticates with a role identifier and a secret identifier written into its environment. The chain terminates at whatever placed that value there: an image layer, a configuration management run, a provisioning script. Whatever that thing is, it is the credential, its lifetime is the real lifetime, and the fifteen-minute tokens further up merely limit how long each individual hop lasts.

The four hops in the issuance path

flowchart LR
    PLAT["Platform\nattests the running workload"]
    WL["Workload"]
    SM["Secret manager\nauth method plus policy"]
    DB["Database or API"]
    PLAT -->|"assertion, minutes to an hour"| WL
    WL -->|"login with the assertion"| SM
    SM -->|"token bound to a policy and a lease"| WL
    SM -->|"creates a principal with an expiry"| DB
    SM -->|"username and password for one lease"| WL
    WL -->|"connects as a distinct principal"| DB
    SM -->|"on revoke, the principal is dropped"| DB

Four hops, and the interesting one is the fourth. The manager does not hand out a stored credential; it creates a new principal in the target system for this consumer and this lease, and the response carries both the identity and its expiry:

{
  "lease_id": "database/creds/app-readonly/xoHI541EXoFgKn1OTiusOdd9",
  "lease_duration": 120,
  "renewable": true,
  "data": {
    "password": "[REDACTED]",
    "username": "v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423"
  }
}

The username is the whole architecture in one string. It is unique to this lease, so every query the database logs is attributable to one consumer during one window. It did not exist two minutes ago. And because it was created rather than shared, withdrawing it is a real operation with a real effect rather than a policy change nobody can verify:

$ bao lease revoke database/creds/app-readonly/xoHI541EXoFgKn1OTiusOdd9
All revocation operations queued successfully!

$ psql -U v-token-app-read-... -d appdb -c "SELECT 1;"
psql: error: connection to server at "127.0.0.1", port 5432 failed: FATAL:  role "v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423" does not exist

Compare that with revoking a shared application password, where the honest verification is a survey of every consumer and a hope. Here the principal is gone from the database and the proof is a connection refusal from the database itself, which is an independent channel from the manager that performed the revocation.

What it costs to put a manager on the request path

The design has one uncomfortable property, and it should be stated plainly rather than discovered. A workload that needs a fresh credential every few minutes has made the secret manager a hard dependency of its own availability. A sealed manager answers everything the same way:

Code: 503. Errors:

* Vault is sealed

The clock that follows is the same shape as the issuing CA in the previous lesson, with a very different scale. There, ninety-day leaves gave an estate a month of grace. Here, a two-minute lease gives it two minutes. Lease length is not a security setting to be minimised; it is a dial between two failures, and both ends of it are real.

• Short leases buy revocation speed and small attribution windows, and they spend availability. The estate cannot survive an outage longer than the shortest lease in it.
• Long leases, or client-side caching, buy tolerance of an outage and spend revocation speed. Caching a credential for ten minutes means a revoked credential keeps working for up to ten minutes, by construction, no matter what the audit log says.

Recovery from the sealed state is a ceremony rather than a restart, and it is deliberately not automatable by one person. The unsealing material is split into shares with a threshold, the manager reports its progress as shares are supplied, and it refuses service until the threshold is met. The root key is not stored anywhere, so an estate that cannot assemble the threshold does not have a slow recovery, it has no recovery. That fact belongs in the disaster recovery plan next to the names of the custodians.

The platform assertion at the top of the chain has failure modes of its own that are easy to state wrongly. A projected service account token in Kubernetes is bound to its Pod, defaults to one hour, and is rotated by the kubelet at eighty per cent of its lifetime or after twenty-four hours. That is the good path. Tokens injected by admission rather than requested properly are extended by default, up to a year, by a control plane flag that is enabled unless someone turned it off, so the one-hour figure describes the design and not necessarily your cluster. And there is no interface for revoking such a token at all: the documented remedy is to delete the Pod. Both facts argue the same way, which is that the credential a workload actually uses should be a lease the manager can revoke, with the platform token used only to obtain it.

Production discipline

1. Walk the chain backwards in every design review. Ask what authorised each hop until the answer is a platform attestation or a file. Write the answer down; it is the design’s real trust root.
2. Make the target system enforce the expiry too. A credential whose only expiry lives in the issuing manager becomes permanent the moment that manager is unavailable.
3. Choose the lease length against your outage budget, not against a guideline. State the maximum tolerable secret manager outage first, then set the shortest lease longer than it.
4. Treat caching as an explicit extension of revocation time. Where a cache exists, publish the number: this credential can survive revocation by up to that long, and incident response must assume it.

Cross-course references

• Kubernetes for Production Sysadmins - Part LVII (Authentication) covers how the cluster mints and validates the assertion that starts this chain, including what the token is bound to and what it is not.
• Observability for Production Sysadmins - Part XVIII (Alerting Rules) covers alerting on the health of a dependency that sits on the request path, which is what a secret manager becomes in this design.
• Ansible for Production Sysadmins - Part XXI (Secrets) covers the delivery mechanisms that place a bootstrap value on a host, the step that decides whether the chain terminates or merely pauses.

Quiz