The secret-zero problem: how a machine gets the credential that gets its credentials

Secret zero is the name for a question that sounds like a riddle and is actually an architecture decision. If an application obtains its database password from a secret manager, it needs a credential to reach the secret manager. If that credential is delivered by an automation system, the automation system needs a credential too. Each step is sensible on its own, and the chain has to end somewhere. Where it ends is the only part of the design that matters.

The regress, walked one step at a time

Take a concrete estate and follow it downwards without flinching.

A reporting service on web-01 needs to read from db-03. It holds no database password; it authenticates to the secret manager and receives a dynamic credential. To authenticate it presents an AppRole SecretID, which is written to /etc/reporting/secret-id at deployment time. The deployment is performed by a configuration management run, which fetched that SecretID from the secret manager moments earlier, using a token of its own. That token was issued to the automation controller when it started, from a credential stored in its own configuration. And that credential was placed there by whoever built the controller.

flowchart TD
    A["Reporting service\nneeds a database credential"] --> B["Secret manager\nissues it dynamically"]
    B --> C["Service must present\nan AppRole SecretID"]
    C --> D["Automation run\nwrote the SecretID to disk"]
    D --> E["Automation holds\nits own token"]
    E --> F["Token came from a credential\nin the controller's configuration"]
    F --> G{"Who placed that\nand how is it protected?"}

Notice what has and has not been achieved. Something real was gained at every step. The database password no longer exists anywhere on disk. A compromise of web-01 yields a credential that expires in minutes and is scoped to one read-only role. The number of long-lived secrets in the estate has gone from many to one. That is not nothing, and teams who claim the pattern is theatre are wrong.

But the chain did not terminate. It converged. One credential now sits in the automation controller’s configuration, and it is more powerful than the credential it replaced, because it can obtain the credentials of everything else. Convergence without termination is the classic outcome, and it is worth naming precisely: the estate traded many small exposures for one very large one, and then stopped.

Why AppRole moves the problem rather than ending it

AppRole is a good mechanism used badly more often than it is used well, and it is worth being fair to it before criticising it. Splitting the credential into a RoleID and a SecretID means the two halves can travel separately. A SecretID can be issued single-use, given a short TTL, bound to a source address range, and delivered response-wrapped so that any interception is detectable rather than silent. A SecretID constrained that way is genuinely not a password.

The problem is structural rather than a matter of configuration quality. Something has to put the SecretID where the workload can read it, and that something must already be trusted enough to obtain SecretIDs. Response wrapping does not remove that party; it narrows what the party learns. A single-use SecretID does not remove it either; it shortens the window during which the party’s compromise matters. Every improvement operates on the size and duration of the exposure, and none of them operate on its existence.

The give-away in a design review is a sentence of the form “and then the trusted delivery agent places it”. Ask what the delivery agent authenticates with. If the answer is another stored credential, the regress is still running and has simply been pushed one layer further from the reviewer’s attention.

What actually terminates the regress

A chain terminates when the workload presents something it did not have to be given and could not have copied from elsewhere. Four mechanisms achieve that, and they differ in what they ask you to trust.

The first is platform attestation, where the credential is the workload’s observable position in the system rather than anything it holds. An agent on the node inspects the calling process through the kernel or the local orchestrator and determines its properties: its user, its binary, its control group, its service account. The specification for this pattern is emphatic that the workload must not participate in the check, because a check the workload can influence is a check it can lie to. Nothing is stored, so nothing can be stolen from storage.

The second is cloud instance identity, where the platform that created the machine will sign a statement about it on request. The machine did not store the statement; the infrastructure produced it because the machine is running where the infrastructure put it. The identity is therefore as strong as the boundary around the metadata channel that delivers it, which is a real and sometimes weak boundary, but it is not a secret on disk.

The third is the Kubernetes projected token, which is the same idea implemented by the kubelet. The token is written into the pod because the API server acknowledges the pod exists, it is bound to that pod, it expires, and the kubelet replaces it before it does. Nothing was distributed to the workload by an automation system.

volumes:
  - name: bao-token
    projected:
      sources:
        - serviceAccountToken:
            path: bao-token
            audience: openbao
            expirationSeconds: 7200

kubectl create token reporting \
    --namespace analytics \
    --audience openbao \
    --duration 10m

The audience is the part to look at. A projected token requested with an audience of openbao is not accepted by the API server as an API credential, and an API credential is not accepted by the secret manager. Requesting a distinct audience per relying party is what stops one compromised recipient from replaying the token at another, and it costs a single line of manifest.

There is a quick way to tell a projected credential from a distributed one on a running host, and it is worth doing during a review rather than taking the manifest’s word for it:

TOKEN_PATH=/var/run/secrets/tokens/bao-token

# Who may read it, and when it was last replaced.
stat -c 'file=%n owner=%U:%G mode=%a modified=%y' "$TOKEN_PATH"

A file whose modification time moves on its own, without a deployment or a configuration management run touching the host, is being maintained by the platform. A file whose modification time matches the last deployment was placed by something, and that something is the next link in the chain.

The fourth is a hardware root of trust. A key generated inside a device and never exported cannot be copied by anyone with only filesystem access, and possession is proved by a signing operation rather than by handing a value over. This is the only terminator on the list that is physical, which makes it the strongest and the least convenient: it binds identity to a specific piece of hardware, which is exactly what you want for a certificate authority and exactly what you do not want for a workload that is rescheduled twenty times a day.

Where the trust really lands

None of the four mechanisms create trust from nothing. Each moves it to a place where it can be established once, out of band, and then reused indefinitely. A hypervisor knows which instance it started. A kubelet knows which pod it is running. A device manufacturer vouches for a key that never left the chip. In each case the verifier and the platform already share a relationship formed at provisioning or manufacture time, and the workload inherits it by being where it is.

That has a consequence people skip. The platform operator is now inside your trust boundary in a way they were not before. Anyone who can convince the kubelet that an arbitrary process is the reporting pod can obtain the reporting pod’s credentials, without ever touching a secret. Terminating the regress does not remove risk; it converts a risk you manage with file permissions and rotation into a risk you manage with node hardening, admission control and platform access review. Whether that is an improvement depends entirely on which of those two things your organisation is better at.

Judging a proposed answer

Four questions separate a terminated chain from a longer one, and they can be asked in a design review in about five minutes.

• Does the workload store anything that survives a reboot? If a file must be present before the workload can authenticate, something put it there and the chain continues through that something.
• Could the identity be presented from a different machine? A bearer value copied to another host works there. An attestation based on the process’s position does not, and a key inside a hardware module cannot leave at all.
• Who could forge the attestation, and what would that require? Answer it as an attack, not as a policy. Root on the node, control of the kubelet, and access to the hypervisor are all different answers with different likelihoods.
• How narrow is the audience? An identity accepted by every relying party lets the weakest of them impersonate the workload to the strongest. One audience per relying party is the cheapest control on this list.

Production discipline

1. Draw the chain before defending it. Write every hop from the application down to the last thing a human placed. A design that cannot be drawn on one page has not been thought through.
2. Name the last stored secret out loud. If one exists, put it in the design document with its location, its rotation procedure and the blast radius of its loss.
3. Use a distinct audience for every relying party. It is one field in a manifest and it removes an entire class of replay between systems that trust the same issuer.
4. Audit the node attestor, not just the workload attestor. A pasted join token underneath a sophisticated identity system reintroduces exactly the problem the system was bought to remove.
5. Budget for the trust you moved. Terminating the regress transfers assurance onto node hardening and platform access control. Fund those, or the termination is nominal.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) covers the automation controller that, in most estates, ends up holding the last stored credential in the chain described here.
• Kubernetes for Production Sysadmins - Part LXXIV (Kubeadm) covers how a cluster’s own trust is established at bootstrap, which is the same problem solved one layer lower down.
• Terraform for Production Sysadmins - Part XIX (Security) covers how provisioning code handles credentials, which is where a bootstrap secret is most often written into a repository by accident.

Quiz