External secret managers - the bootstrap and availability problems

A central secret store gives you the things files cannot: one place to rotate, an audit trail of every read, access policies per identity, and dynamic credentials that expire on their own. Those are real and they are worth the move.

This lesson is about the two things the move does not solve and the one thing it makes worse, because those are what determine whether the store improves your operations or becomes the most interesting single point of failure you own.

The bootstrap problem

To read a secret from the store, the host needs a credential for the store. Where does that credential come from?

If the answer is “a file on the host”, you have moved every secret behind a single file-based secret, and the security of the whole system is the security of that file. That is not necessarily wrong - one well-protected file is better than forty scattered ones - but it must be a decision rather than an accident.

The recursion terminates when authentication is based on something the host is rather than something it has:

The cloud instance identity is the cleanest: the host asks the metadata service, which is reachable only from that instance, for a signed document proving what it is. Nothing was placed on disk by anyone.

# The instance metadata service lives on the link-local address
# 169.254.169.254 on every major cloud. On AWS, IMDSv2 requires a
# token first, which is what makes it resistant to SSRF.
TOKEN=$(curl -sS -X PUT 'http://169.254.169.254/latest/api/token' \
  -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')
curl -sS -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/info

On hardware and on TPM-backed VMs, systemd-creds does the same job locally:

# Seal to this machine AND this boot state, with an expiry
sudo systemd-creds encrypt --with-key=host+tpm2 --name=dbpw \
     --not-after='2027-01-01 00:00:00' \
     plaintext.pw /etc/credstore.encrypted/dbpw
sudo shred -u plaintext.pw

--with-key=host+tpm2 requires both the host key and the TPM, so the ciphertext is useless on any other machine. --not-after= embeds an invalidation time, so a credential that outlives its intended window stops working rather than quietly persisting.

Availability: the store is now in your boot path

A file-based secret has the availability of the local disk. A store-based secret has the availability of the store, the network to the store, DNS for the store’s name, and the certificate chain the client validates. That is four new dependencies on a path that previously had none, and they are all in the start-up path, which is exactly when you least want new dependencies.

The design decision is explicit and it has no universally right answer:

Fail closed. The service refuses to start without a current secret. Correct for anything where operating with a stale credential is a compliance or safety problem. The cost is that a store outage becomes a fleet-wide outage the moment anything restarts.

Fail open with a cached value. The agent keeps the last rendered secret on tmpfs and the service keeps running with it. Correct for most workloads: a store outage degrades rotation rather than availability. The cost is that a revoked credential can stay in use until the cache is refreshed.

vault-agent implements the second by default in the templating pattern from the first lesson in this part: the rendered file stays in place, and the agent retries in the background. Whether that is what you want is a decision to make deliberately, per service, and to record.

Circular dependencies

A secret store is infrastructure, so it has dependencies, and those dependencies frequently want secrets. That is how a recovery ends up unable to start:

• The store’s clients resolve its name through DNS. If the DNS servers need a credential from the store to start, neither can start after a full power loss.
• The store validates clients with TLS certificates issued by an internal CA whose key is… in the store.
• Auto-unseal uses a cloud KMS, so a restart of the store during a cloud control-plane incident cannot complete.
• The store runs on a cluster whose nodes fetch their join credentials from the store.

None of these is exotic and all of them are invisible until a cold start. The test is a specific one, and it is a tabletop exercise rather than a command: starting from everything off, in what order does this come up, and does any step require something that is not yet up?

Write the answer down as an ordered list, and keep a break-glass path that does not depend on the store at all - a sealed envelope, an offline copy of the unseal material, a local credential on the DNS servers. See linux-dependencies-dns-certs-identity for the general form of this analysis.

Scoping: one identity, one path

The audit trail and the access policy are the reasons to have a store, and both are worthless if every host authenticates as the same identity with a policy granting secret/*.

$ vault token lookup -format=json

{
"data": {
  "display_name": "approle",
  "meta": { "role_name": "app01-prod" },
  "policies": ["default", "app01-prod-read"],
  "ttl": 3512,
  "renewable": true
}
}

The properties worth checking on any token in production: its policies name one service and one environment, its ttl is hours rather than years, and it is renewable so the short TTL does not cause an outage. A token with "ttl": 0 is a root-equivalent problem waiting to be discovered.

The audit log is the other half. It only helps if it leaves the box:

• Forward the store’s audit device to the central log store, so an attacker who compromises the store cannot erase the record of what they read.
• Alert on a read of a credential outside its normal pattern - a production database secret read by an identity that is not the production database client.
• Alert on any use of the root or emergency identity, every time, with no exceptions and no tuning.

Migrating without a big bang

Moving forty credentials from files to a store in one change is how the store gets blamed for the next three outages. Move one at a time, and make each move individually reversible:

1. Write the credential into the store while the file remains the source of truth. Nothing reads the store yet.
2. Add a read from the store that compares against the file and logs a mismatch. Run that for a week. It finds the credentials that were already inconsistent between hosts.
3. Switch one non-critical consumer to read from the store, keeping the file in place as a rollback.
4. Switch the remaining consumers, one group at a time.
5. Remove the file - and only then, because a file left behind is a stale credential that still authenticates.
6. Rotate the credential through the store, which is the first proof that the whole path works.

Step 2 is the one people cut and the one that pays. Every estate has credentials that differ between hosts because somebody fixed one host in an incident three years ago and never propagated it. Discovering that during a comparison week is free; discovering it during the cutover is an outage.

Step 6 matters because a migration that is never followed by a rotation has not demonstrated anything: the value in the store is being read successfully, but the whole reason for the store - being able to change the value in one place - is still untested.

Knowledge check