Root and trust anchor migration

A trust anchor is the one thing in a public key infrastructure that the infrastructure cannot deliver. Leaves are presented by the peer. Intermediates travel in the handshake. The anchor has to already be there before any of that means anything, which is why replacing one is a distribution problem wearing a cryptography costume.

Why the anchor is different from everything else

The asymmetry is worth stating bluntly: the certificate authority does not know who trusts it. There is no registration, no callback, no list. A trust store is a local configuration decision made by whoever operates the client, and the only way the authority learns that a store exists is when something breaks.

Those stores are also plural and heterogeneous within a single estate. An operating system bundle is updated by the package manager. A Java installation carries its own keystore. Python programs frequently use a bundle shipped inside a library rather than the system one. Go programs compile in their own loading rules. A container image carries a frozen copy of whatever the base image had at build time. Network appliances, printers, payment terminals and partner systems each have their own mechanism, and some have none at all.

# The operating system bundle is one store among many. Adding to
# it is necessary and nowhere near sufficient.
find / -xdev \( -name cacerts -o -name ca-bundle.crt -o -name cert.pem \) -print 2>/dev/null
python3 -c 'import certifi; print(certifi.where())'

Run that on a host that has been in production for a few years and the result is usually uncomfortable. Each path is an independent store with an independent update path and an independent owner, and the migration has to reach every one of them before anything changes on the issuing side.

The strict ordering

The sequence below is not a recommendation about tidiness. Every step exists because performing it later than the step after it causes an outage that cannot be fixed quickly.

flowchart TD
    A["1. Distribute NEW anchor to every trust store\nadd, never replace"] --> B{"2. Coverage proven by canary?"}
    B -- "no" --> A
    B -- "yes" --> C["3. Both anchors trusted in parallel\nissuance unchanged"]
    C --> D["4. Cut issuance over in stages\nnew leaves chain to the NEW root"]
    D --> E{"5. Served chains validate\non every client class?"}
    E -- "no" --> F["Point issuance back at the OLD root\ncheap: both anchors still trusted"]
    F --> C
    E -- "yes" --> G["6. Soak until the last\nOLD-root leaf has expired"]
    G --> H["7. Withdraw the OLD anchor\nno fast rollback from here"]

Read the diagram as a claim about reversibility. Steps one to six are all cheap to undo, because the estate trusts both anchors throughout and issuance can be pointed back at the old hierarchy in minutes. Step seven is different: undoing it requires the same distribution campaign that steps one and two took, which is the thing you spent months on.

The single most common way to get this wrong is to combine steps one and seven, replacing the old anchor with the new one in the same operation because that is what a configuration management change naturally looks like. Adding a file is a different action from replacing a directory’s contents, and only the first is safe.

# Correct: add. The store now contains both anchors.
cp new-root.crt /usr/local/share/ca-certificates/example-root-2026b.crt
update-ca-certificates

The command rebuilds the combined bundle from everything in the directory, so an anchor is added by adding a file and removed by removing one. Nothing about the old anchor is touched, and a client that had been working continues to work whichever hierarchy it encounters.

Proving coverage before the cutover

Step two is where migrations are actually won or lost. It is not possible to prove coverage by inspecting the distribution job, because the entire difficulty is the stores the distribution job does not know about. Coverage has to be measured from the client side, by clients, against something real.

The instrument is a canary endpoint: a service, reachable by every class of client, presenting a certificate that chains to the new root and nothing else. Any client that can complete a verified handshake against it holds the new anchor. Any client that cannot is a client that would have failed at cutover.

CANARY=trust-canary.example.com
openssl s_client -connect "$CANARY:443" -servername "$CANARY" </dev/null 2>&1 \
  | grep 'Verify return code'

A reading of Verify return code: 0 (ok) from that probe is a positive result from that host, in that runtime, with that store. The value of the canary is that it can be embedded in every client class you have: a start-up check in the application, a step in the container build, a scheduled task on the appliance, a request the partner’s integration makes. Each one reports for itself.

What a failure looks like is worth recognising, because it is the same string in both of its common forms and it says nothing about anchors:

curl: (60) SSL certificate problem: unable to get local issuer certificate

error 2 at 1 depth lookup: unable to get issuer certificate

The first is a client that has no anchor for the chain it was sent. The second is a local verification given an intermediate as its anchor rather than a root, which is the shape of the same mistake made in a configuration file. Neither message contains the word trust, which is why anchor problems are so often investigated as chain problems.

Cross-signing removes the hard dependency

The ordering above works, but it makes issuance wait on the slowest trust store in the estate. Cross-signing removes that constraint by making a single chain validate under either anchor.

A cross-certificate is a certificate whose subject and public key are those of the new root, signed by the old root’s private key. The new root therefore exists in two forms: the self-signed one that new clients install as an anchor, and the cross-signed one that servers can send in the chain. A server presenting leaf, intermediate and cross-certificate satisfies both populations. A client holding the new anchor stops as soon as it reaches the new root. A client holding only the old anchor continues one step further, verifies the cross-certificate with the old root’s key, and validates.

The caveats are real and worth planning for. Path building is a search, so a client holding both anchors may build either path, and which one it picks is an implementation detail. If the old root reaches its own expiry while the cross-certificate is still being sent, some validators reject the longer path and others ignore the expired anchor entirely, producing a failure that appears on some clients and not others on a date nobody associated with the migration. The cross-certificate is also removable from chain files at any time, which makes retiring it a serving-side change of the kind covered two lessons ago rather than another distribution campaign.

Withdrawal is the one step with no cheap rollback

Removing the old anchor is the last step and the only irreversible one. Two conditions gate it, and both are dates rather than opinions.

• Every leaf chaining to the old root has expired. Until then, some server somewhere is presenting a chain that only the old anchor validates. This is the condition teams misjudge, because the population of such leaves includes certificates issued by systems nobody remembers.
• The withdrawal has soaked in a canary first. Reverse the canary: a second endpoint presenting a chain to the old root only. When the migration is complete, every client should fail against it. Any client that still succeeds has not migrated, and finding that out on a canary is very different from finding it out on the payment gateway.

Even then, withdraw in stages and keep the anchor file rather than deleting it, so that restoring it on a single host is a copy and a bundle rebuild rather than a retrieval from an archive. This is the failure the fleet breaks on: the anchor is removed estate-wide in one change, a client class nobody enumerated stops working, and the fix requires touching every one of them again.

Production discipline

1. Add, never replace. The change that installs the new anchor must leave the old one in place, and must be a separate change from the one that removes it.
2. Measure coverage from the clients. A canary endpoint under the new root, probed by every class of client including inside container images, is the only evidence that distribution finished.
3. Cut issuance over only after coverage is proven, and in stages. Rolling issuance back is cheap while both anchors are trusted; it stops being cheap the moment one is withdrawn.
4. Watch the old root’s own expiry. If a cross-certificate is in use, that date is a hazard for the whole estate, not a housekeeping detail.
5. Withdraw last, in stages, and keep the file. Restoring an anchor on one host should be a copy and a bundle rebuild, not an archive retrieval under incident pressure.

Cross-course references

• Linux for Production Sysadmins - Part XII (RepoSecurity) covers the package signing keyring, a second anchor set on the same hosts that has to be migrated by exactly this sequence.
• Git, CI/CD & GitOps for Infrastructure Engineers - Part LIII (ContainerCI) covers the image build pipeline, which is what decides whether a workload ever receives the new anchor.
• Kubernetes for Production Sysadmins - Part LXIII (LinuxSec) covers the container runtime security context, including how a workload sees the host’s certificate material or fails to.

Quiz