Distributing a private trust anchor across a fleet, and the ordering rule

Issuing a certificate from a private CA is a five second operation. Making a thousand machines willing to accept it is a project. Those two facts are the whole reason private PKI rollouts go wrong: the fast half is visible and satisfying, the slow half is invisible plumbing, and the order in which they are done decides whether the change is boring or an outage measured in hours.

The ordering rule

Trust must arrive before anything relies on it. Distribute the anchor to every consumer and prove it landed, and only then begin serving certificates that chain to it. Reversed, the outage lasts exactly as long as it takes to reach the final consumer, and it starts on every machine simultaneously.

The reversal is easy to fall into because the two halves belong to different people. Someone owns the CA and can issue in an afternoon. Someone else owns the fleet and needs a change window. The certificate change is a deliverable with a ticket, and the trust change is infrastructure nobody demos. So the certificate lands first and the plumbing catches up, which works right until it does not.

flowchart LR
    A["Publish anchor to the source of truth"] --> B["Distribute to every consumer"]
    B --> C{"Coverage proven by a live probe?"}
    C -- "no" --> B
    C -- "yes" --> D["Switch issuance to the new hierarchy"]
    D --> E["Wait out the longest certificate issued under the old anchor"]
    E --> F["Remove the old anchor and re-verify"]

The loop in the middle of that diagram is the part that gets skipped. Distribution is not a step that completes; it is a step that converges, and the gate out of it is evidence rather than elapsed time. Everything downstream of the gate is safe precisely because the gate was honest.

The harder question is what counts as a consumer. It is not a list of hosts. It is every runtime on every host, every container image in every registry, every build agent, every monitoring probe, every partner system outside your administrative boundary, and every appliance whose trust store is firmware. That last category decides the shape of the whole programme. If a consumer cannot be updated at all, then a private anchor is the wrong answer for that path, and the correct response is a publicly trusted certificate on that specific endpoint rather than a heroic attempt to reach a device you do not control.

Making coverage measurable

There are two grades of evidence that an anchor arrived, and they are not close in value.

The weak grade is file evidence: the anchor exists at the expected path with the expected content. It is worth collecting, and it must compare identity rather than filename, because a file with the right name and the wrong bytes passes every check that looks at paths.

ANCHOR=/usr/local/share/ca-certificates/runbook-lab-root.crt
REFERENCE=/opt/pki/expected/runbook-lab-root.crt

# Compare identity, not location: both must print the same digest.
openssl x509 -in "$ANCHOR" -noout -fingerprint -sha256
openssl x509 -in "$REFERENCE" -noout -fingerprint -sha256

The strong grade is functional evidence: a client on that machine, in that runtime, completes a real path to a real endpoint. Stand up a canary service early, before you need it, serving a certificate issued under the new hierarchy and doing nothing else. Then a single probe proves the entire chain of preconditions at once, that the anchor is present, the generator ran, the correct store was updated for that runtime, and the process was restarted.

CANARY=canary.internal.example.com

# One probe covers the anchor, the generator, the store and the restart.
curl --silent --show-error --fail "https://$CANARY/healthz" > /dev/null

Coverage is a fraction, and a fraction needs a denominator you can defend. “It is on all the hosts Ansible knows about” is a denominator with a hole in it the size of every machine that has drifted out of inventory. If you cannot state what the denominator is and how it was built, you do not have coverage, you have a sample.

The channel that delivers the anchor

An anchor certificate contains only public information. Nothing about it needs to be kept secret, and treating it as a secret produces awkward handling for no benefit. Its integrity is a different matter entirely. Whoever can substitute the file you are distributing gains the ability to issue certificates that your entire fleet will accept for any name they choose. Installing a CA certificate is the most powerful single change you can make to a host’s security posture, and it habitually goes through lighter review than a log rotation setting.

That produces a bootstrap constraint worth checking before the rollout rather than during it: the channel that delivers the anchor must not depend on the anchor. If configuration management reaches hosts over a TLS connection validated against the very root you are installing, the first host cannot be reached at all. Most tooling avoids this by accident rather than design, because SSH host keys and agent enrolment sit outside the web PKI, but verify it rather than assume it. Where the circularity is real, the break is an image build or a signed package, with the fingerprint confirmed out of band on first install.

Retiring an anchor without breaking anything

Removal is the same rule read backwards, and it is where estates usually get caught. An anchor may only be removed once nothing depends on it, and “nothing” includes every certificate already issued under it that is still inside its validity window. The bound is arithmetic: the earliest safe removal date is the moment the last certificate issued under the old hierarchy has been replaced, which cannot be sooner than the longest lifetime you ever issued.

Cross-certification is the mechanism that shortens the wait. The same intermediate, with the same subject name and the same public key, can be packaged in two certificates: one signed by the old root and one signed by the new. A server that sends the variant signed by the old root satisfies clients that hold only the old anchor, while clients that hold the new anchor can still build a path, because path building searches rather than walks. That is what lets issuance move to the new hierarchy before coverage reaches every last consumer.

The cross certificate has its own validity window, and that window becomes the real deadline for the migration. It is also why a client holding both anchors may legitimately build either path, and the two paths expire on different days. Record which path each class of client is expected to build, because an outage that arrives when a cross certificate expires looks exactly like an outage caused by a leaf expiring and gets debugged in the wrong place for the first hour.

Production discipline

1. Write the order down as a gate, not a preference. Issuance from the new hierarchy is blocked until the coverage probe passes, and the block is enforced by whoever holds the CA.
2. Build the canary before the rollout. An endpoint that exercises the new anchor, and nothing else, converts a fleet-wide question into a single boolean per machine.
3. Defend the denominator. State how the consumer list was built and what would make a consumer invisible to it.
4. Review the anchor change like a privilege grant. It is one, and the reviewer should be able to name who holds the corresponding CA key and where it lives.
5. Plan the removal on the day of the addition. Record the longest lifetime issued under the outgoing hierarchy, because that number is the earliest date the old anchor can safely leave.

Cross-course references

• Ansible for Production Sysadmins - Part XVI (Handlers) covers running a command only when a file actually changed, which is the shape an anchor installation and its generator run must take to be safe to re-run.
• Linux for Production Sysadmins - Part LI (Fleet) covers managing change across many machines at once, including the inventory question that decides whether your coverage figure means anything.
• Observability for Production Sysadmins - Part LXIII (Synthetic) covers synthetic probes, which is exactly what the canary endpoint becomes once the rollout is finished.

Quiz