Identity, secrets and PKI: the bootstrap problem

Names resolving and addresses being handed out again, where the previous lesson ended, is the moment an estate stops being a set of unreachable hosts and becomes a set of hosts that cannot prove who they are. Between those states sits a small group of systems — a directory, an identity provider, a secret manager, an internal certificate authority — each normally a prerequisite of everything else, and each quietly a prerequisite of the others. That is this lesson’s subject, because a graph in which A cannot start without B and B cannot start without A has no recovery order to get wrong.

Three nodes, three edges, and no place to start

Consider the arrangement a competent team reaches after three years of doing the right things, one improvement at a time. The secret manager serves its API over TLS, and the certificate it presents is issued by the estate’s internal certificate authority, because self-signed certificates in production were an audit finding. The CA’s signing key is not sitting on a filesystem; it is sealed in the secret manager, because unprotected private keys on disk were the previous finding. Administrators reach both through the identity provider, because shared local accounts were the finding before that. And the identity provider reads its own database password from the secret manager during start-up, because hard-coded credentials were the very first finding.

Three nodes, three edges, and every edge was an improvement. Together they close a loop, which is not a scheduling difficulty to be solved with more people: no ordering of those three exists, so no amount of effort produces one. The team learns this by watching the first start-up attempt fail, then proving that the other two choices of what to start first fail in exactly the same way.

In steady state the loop is undetectable. All three processes are running, each already holds what it needed from the others, and the edges fire only at process start. There is nothing to monitor, because the shape exists only when nothing is running — a condition production is designed never to be in. It appears on the day the estate is cold.

A fourth participant turns an awkward loop into an estate-ending one. If the repository passphrase is held in the secret manager — a thoroughly sensible place for it — and the secret manager is itself restored from that repository, the two are mutually downstream and every other system hangs off them.

Detecting the cycle instead of discovering it

Detection is mechanical and costs almost nothing. It is rarely done because recovery dependency graphs are held as narrative — a numbered list, an architecture diagram — and narrative cannot be checked for loops. Written as data it can be, by any topological sort: an edge list is one line per edge, naming a prerequisite and the thing that needs it, and a sort over that file either produces an order or reports the loop it found.

GRAPH=/srv/dr/graph/edges.txt
mkdir -p "$(dirname "$GRAPH")"

# one line per edge: prerequisite first, dependent second, both bare tokens
printf '%s\n' \
  'internal-ca secret-manager' \
  'secret-manager identity-provider' \
  'identity-provider internal-ca' > "$GRAPH"

tsort "$GRAPH"

Feed it every node the identity estate contains, which is more than the four named above: the directory and the identity provider are often separate systems, the secret manager may be fronted by a KMS or an HSM with its own authentication path, and there is usually an SSH certificate authority, a signing key for session tokens, and a multi-factor service without which no operator can log in at all. The operator’s own access path — VPN, bastion, console — is a node like any other.

The edges that matter have a particular signature: consumed once, during initialisation, and never again. A service that fetches its database password from the secret manager while starting and caches it for the life of the process depends totally on that manager, and no amount of steady-state observation shows it. Configuration is a better source than memory, but the decisive one is a cold start on an isolated host: bring the node up with nothing else reachable and read what it tries to open before it gives up.

When the sort reports a loop, the nodes it names form a strongly connected component: every member is reachable from every other, which is exactly why none can be first. The remedy is never reordering. It is deleting an edge, and an edge is deleted by supplying what it demanded from somewhere the component cannot reach. Every proposed remedy has to survive one question, asked literally: is the thing that satisfies this edge itself inside the component? Plausible bootstrap plans fail it on first reading — the CA root key is in the vault, the unseal shares are in the password manager, and the password manager authenticates against the identity provider.

Because the edge list is a file, the check belongs in the pipeline that reviews infrastructure changes: the question a reviewer cannot answer from a paragraph — did this change close a loop? — the sort answers.

The identity store is a database of credentials, and restoring it is a rollback

Suppose the cycle is broken and the directory can be restored. The restore introduces a second problem, less discussed and, where an intruder is involved, considerably more dangerous.

An identity system is not a configuration file. It is a database whose contents are credentials and authorisations: password verifiers, key versions, enabled flags, group memberships, multi-factor enrolments, issued certificate serial numbers, and the revocation state attached to them. Restoring it to a point in time restores every one of those to the value it held then, and nothing inside the restore separates routine changes made since from deliberate security actions, because at that level they are the same kind of write.

Work an example with dates on it. The recovery point is six days old. Four days ago an administrator disabled a departing employee’s account, removed a contractor from an administrative group, rotated a service account password that had turned up in a log file, and revoked a certificate issued to a decommissioned host. The restore returns the directory to a known-good state and, in the same operation, re-enables the account, restores the group membership, reinstates the leaked password, and drops the revoked certificate off the revocation list.

The correlation is what makes this dangerous rather than untidy: the incidents forcing an identity restore are disproportionately the incidents during which things were revoked. A compromise response disables accounts for hours, and the recovery then rolls the directory back to a point before that work began.

An identity restore is therefore two pieces of work: the restore, and then a reconciliation pass, performed before the system serves any authentication request, replaying every credential and authorisation change made since the recovery point. Its only source is the audit log — which is the argument for shipping identity audit logs somewhere the identity system does not own, and for treating “the directory is restored” as a step in the middle of the runbook rather than its end.

Credentials that must exist before the services that use them start

The dependency the graph exposes is not on the identity systems as processes but on the material they issue. A service needs its credential to exist, to be valid, and to be reachable at the instant it starts, which puts the issuer strictly upstream of every consumer during recovery even where the relationship is invisible in normal running.

Four kinds of material behave differently. Static service account passwords survive a rebuild of the consumer but not a rollback of the store. Machine and workload identities — a joined machine account, a node’s client certificate — are bound to a host, so a rebuilt host is a new identity needing an enrolment step over an authenticated path. Short-lived tokens do not exist at rest and must be minted before anything can use them. And mutual-TLS certificates fail closed at both ends, so a mismatch leaves two services healthy in isolation and unable to talk.

Dynamic, short-lived credentials are simultaneously the correct steady-state answer and the hardest cold-start case. An estate that has eliminated every long-lived credential has, by construction, made the issuer a total dependency of everything it runs. The conclusion is not to keep static secrets around; it is that the issuer needs a bootstrap path of its own, and that a small, inventoried set of break-glass credentials independent of it is a recovery control rather than a lapse in hygiene.

Two properties are missed when planning from a backup rather than from a running system. The first is expiry: a recovery from a six-week-old image finds material that expired four weeks ago, so the plan must include issuing new material — and issuing needs the CA, which was in the cycle. The second is the clock: validity and token expiry are wall-clock judgements, so time synchronisation sits upstream of all of it.

Possessing the repository is not possessing the ability to open it

All of that assumes the backups can be read, which is the assumption the escrow capture measured. A repository’s passphrase lived inside the directory the repository was protecting; the site was destroyed; the repository survived complete and unreadable.

$ restic snapshots

Fatal: wrong password or no key found
>>> exit code: 12

The key directory explains why copying does not help: the key file travels with the repository, so every offsite copy carries the same single lock.

$ ls /work/repo/keys/

d5f39ef5517fab2e4d1e6dc1d9d5b4b1f1ca4eeacecde523af8b5a0c3c3d120e

The capture then added a second passphrase held by a recovery group and destroyed production again. The same repository answered.

$ restic --password-file /work/recovery-pass snapshots

ID        Time                 Host          Tags        Paths        Size
--------------------------------------------------------------------------
b96ba7cf  2026-08-28 14:04:52  17dffded9807              /work/prod2  38 B
--------------------------------------------------------------------------
Timestamps shown in local time
1 snapshots

>>> exit code: 0

Read the pair as a statement about bootstrap material rather than about restic. Holding the repository is not holding the ability to open it, and the entire difference between exit code 12 and exit code 0 was where one secret had been kept. The escrowed passphrase is bootstrap material, subject to the same test as the CA root key.

What is held offline, and what “held” means

The edge a cycle cannot satisfy from inside is satisfied from a small, deliberately boring set of material held outside the estate. It has three parts, and the third is the one most often missing.

Root key material. The offline root CA key, kept off any system that touches the component and used only to sign intermediates. The material that reopens the secret manager: for a Shamir-sealed Vault the unseal shares, which the seal documentation describes as the split of the unseal key into shares of which a threshold must be supplied before Vault can reconstruct it; for an instance that auto-unseals against an HSM or KMS, the recovery keys that the same document says are returned in their place. And the repository’s escrow passphrase and exported key block. NIST SP 800-57 Part 1 defines split knowledge as exactly this arrangement — a key divided into shares, none of which conveys knowledge of the key, recombined when it is needed — and the honest price of adopting it is that the number of custodians is a term in the recovery time of everything downstream.

Break-glass credentials. A local administrator on the directory that does not authenticate through the identity provider. An out-of-band management credential per site. A cloud account credential outside the federated path. A repository credential in a different trust domain from the estate it reads. Each sealed, inventoried, and alarmed on use, so retrieval is an event rather than a discovery.

The written bootstrap procedure. An ordered document stating which piece of material opens which door, in what sequence, with the observable expected at each step, and how to reach those systems when the usual access path is part of the outage. Held on media beside the material it describes.

“Held offline” then means four things, and dropping any one converts a control into a belief. The material sits outside every failure domain in the component. Its custodians cannot deploy production. It is inventoried with expiry dates, because offline root certificates expire, escrowed passphrases get rotated in production and not in the safe, and break-glass accounts get caught by password policies. And it is exercised on a schedule.

That exercise is not a check that the envelope can be found. It is a bootstrap performed from the offline material alone, estate assumed gone and production credentials withheld, carried out by the custodians rather than for them, because half of what is tested is whether those people can use the material under time pressure. It produces a dated record: who retrieved what, how long each step took, and what was observed. The elapsed time from “we need this” to “the CA signs again” is a term in the recovery time objective of every service downstream, and until the exercise is run that term is unknown rather than small.

Production discipline

1. Write the identity graph as an edge list and sort it on every change. A paragraph cannot be checked for loops; a file of prerequisite dependent pairs can, in a pipeline, on the change that introduces the loop.
2. Break each cycle with material the component cannot reach, and test that claim literally. Ask of every bootstrap item whether the thing supplying it sits inside the strongly connected component; the CA root in the vault, the unseal shares in the password manager and the runbook in the wiki all fail.
3. Treat an identity restore as a rollback of revocation, paired with a reconciliation pass. It re-enables accounts disabled since the recovery point, reinstates credentials rotated since, and drops certificates revoked since off the revocation list; the audit log is the only source for replaying them, so it must live outside the identity system.
4. Hold the bootstrap kit in three parts: root key material, break-glass credentials, and the written procedure. The measured cost of getting custody wrong is the escrow capture’s — an intact repository, Fatal: wrong password or no key found, exit code 12, against exit code 0 for the same repository opened with a passphrase kept elsewhere.
5. Rehearse the bootstrap from the offline material alone and record the elapsed time. The custodians perform it with production credentials withheld; the interval from decision to a working certificate authority is a term in every downstream recovery time objective and is unknown until it has been measured.

Cross-course references

• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XVIII (Incidents and Recovery) treats recovery of a secret management platform and an internal CA as its own discipline, and Part XV (KMS, HSM and Key Protection) supplies the custody rules for the root key material this lesson puts in the offline kit; together they are the detailed form of the bootstrap edge that breaks the cycle described here.
• Kubernetes for Production Sysadmins — Part LXXVI (Cluster Certificates) and Part XCIX (Complete Cluster Loss) are this lesson’s cycle instantiated on one platform: the cluster CA must exist before the control plane will admit a node, so its material has to be recoverable independently of the cluster that normally holds it.
• Linux for Production Sysadmins — Part XXVII (Authentication and Enterprise Identity) covers how hosts consume a directory, including cached credentials and enrolment, which is the consumer side of the machine identity dependencies this lesson places upstream of every service start.

Quiz