← All runbooks in Secrets, PKI & Certificates

critical riskcluster affecting~480 min

Runbook: Perform a Root Trust Transition

1 · Prerequisites

Confirm every item is in place before any state change.

secrets-pki-rb-22-rotate-intermediate-ca
secrets-pki-lab-11-install-private-trust-anchor
secrets-pki-cl-15-pki-disaster-recovery
A named owner for every trust store class in the estate, including the classes owned by other organisations, and a written commitment from each that they can add and later remove an anchor on request.
A completed key ceremony for the new root, witnessed and recorded, with the private key already under its intended protection.
Confirmation that the outgoing root remains valid for longer than the whole transition programme, with margin.
At least one client of every class that can be configured to hold only the new anchor, for the negative test that no other check can substitute for.
An agreed programme schedule with named soak periods, approved by the platform owner and by every team whose service changes chain.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm this is a planned transition and not an incident. If the outgoing root key is suspected of being exposed, the ordering in this runbook is wrong for your situation: it assumes both anchors may safely coexist while certificates drain. A suspected root compromise removes that assumption and belongs to the CA compromise procedure instead.
· Enumerate every trust store class in the estate and give each one an owner. The operating system bundle is one class. A runtime that ships its own bundle is another. A container image that bakes the bundle at build time is another. An appliance updated on a firmware cycle is another. The inventory of classes, with an owner and an update mechanism recorded against each, is the real deliverable of this step.
· Establish, per class, how an anchor is added and how it is removed. Removal is the harder half and it is the one people forget to check. A class where you can add an anchor but not remove one turns the last phase of this runbook into a permanent state.
· Confirm no client pins the outgoing root public key. Pinning defeats a correctly executed transition, and it fails at the moment issuance moves rather than at the moment the anchor is distributed, so it will be blamed on the wrong step. Ask the owning teams explicitly; do not infer it from configuration.
· Verify the new root before it is mentioned to anyone. Check its self-signature, its validity dates and its constraints with openssl x509 -in root-new.crt -noout -subject -issuer -dates -ext basicConstraints,keyUsage. A root whose extensions are wrong is cheapest to discard while nobody has installed it.
· Confirm monitoring can tell the two chains apart. Every probe that reports on certificates must be able to answer which root a served chain terminates at. A programme measured by a probe that reports only pass or fail cannot tell you how far it has progressed.
· Confirm the outgoing chain keeps working throughout. Nothing in the distribution phase should change what any server presents. Record a baseline reading from every listener now so that any change during distribution is visible as a regression rather than as noise.
· Agree the abort criteria in advance. Write down what level of distribution failure stops the programme, who declares it, and how long the estate may safely sit with both anchors installed. Deciding this under pressure produces the outage this runbook exists to prevent.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Publish the schedule and the abort criteria before touching anything. Every team that owns a trust store class or a service needs the dates, the soak periods and the name of the person who can stop the programme. A transition that surprises a team is a transition that gets a trust store rolled back at the worst moment.
2Create the new root offline and leave it inert. Generate the key under the agreed ceremony, self-sign the certificate with basicConstraints = critical,CA:TRUE and keyUsage = critical,keyCertSign,cRLSign, and issue nothing under it. An inert root that is installed everywhere is harmless; a root that has issued certificates before it is trusted anywhere is an outage waiting for a deployment.
3Publish the new anchor to the distribution channels each class consumes. The configuration repository, the image build inputs, the appliance management system, the partner integration package. Publication is not distribution, but nothing can be distributed until it is published.
4Add the new anchor to every trust store, keeping the outgoing anchor in place. On a Debian-family host this is cp root-new.crt /usr/local/share/ca-certificates/ followed by update-ca-certificates. Every other class has its own mechanism, which is why the inventory exists. Both anchors must be present simultaneously; this is the parallel-trust state the whole design depends on.
5Prove distribution independently, per class, by reading the store rather than the deployment report. A configuration management system reporting converged is a statement about its own run, not about the file. Read back the anchors the store holds on a sample of hosts in every class, and record the result as a fraction of the class.
6Hold the estate in parallel trust for the agreed soak period. The soak exists to surface the hosts that were switched off, the images that had not been rebuilt, and the appliance that reverted its configuration overnight. Nothing is issued under the new root during the soak.
7Confirm distribution has reached every class before proceeding. This is the gate the entire runbook is built around. If any class is short, the correct action is to fix that class, not to start issuing. Cutting over issuance while any store lacks the new anchor produces unable to get local issuer certificate on every client in that class at once.
8Issue an intermediate under the new root, and still change nothing that is served. Verify it with openssl verify -CAfile root-new.crt srv-ca-new.crt, publish it as chain material, and stop. The estate continues to present the old chain.
9Cut over the lowest blast radius service first, and test it from both directions. Reissue its leaf under the new intermediate, deploy the new fullchain, reload, then verify from a client holding only the new anchor and from a client holding both. The single-anchor client is the test that proves the path terminates where you think it does.
10Cut over the remaining services in waves, smallest blast radius first. Each wave ends with an off-host verification of every listener in it and a bake period long enough for infrequent clients to reconnect. Do not compress the bake to meet a date; the clients you have not seen yet are the reason it exists.
11Track the residual population on the old chain until it reaches zero. Probe every listener and record which root its chain terminates at. A count that stops falling means a deployment path nobody has mapped, and that path will still be unmapped when you withdraw the old anchor.
12Hold a second soak with the whole estate on the new chain and both anchors still installed. This period is your free rollback window: any service can be put back to its recorded chain in one reload because every client still trusts the old root.
13Withdraw the outgoing anchor last, class by class, in the reverse order of distribution. Remove it from the least critical class first and wait. Withdrawal is the only step in this runbook that can break a client which was working a moment earlier, so it gets the same care as the cutover did.
14Record the transition. The new root subject, serial and validity dates, the classes and their distribution fractions at each gate, the wave dates, the residual counts, the date the outgoing root was withdrawn, and the date its key may be destroyed. Root transitions happen rarely enough that the record is the only training material the next operator will get.

4 · Verification

Confirm the procedure actually fixed the problem.

✓A client configured with only the new anchor completes a request against every cut-over service and reports a verify return code of 0.
✓A client configured with only the outgoing anchor still succeeds during the parallel phase and fails only after the deliberate withdrawal, which confirms the two phases were genuinely separate.
✓For every trust store class, a read-back from a sample of hosts confirms the anchors actually present, expressed as a fraction of the class rather than as a deployment status.
✓From off-host, openssl s_client -connect app.lab.example:443 -servername app.lab.example -showcerts shows a chain whose top entry is issued by the new root for every cut-over listener.
✓The residual probe across the whole inventory reports zero listeners whose chain still terminates at the outgoing root.
✓Certificate expiry and chain monitoring reports the new root by name, proving the probes were updated with the estate rather than left describing the old world.
✓One service is deliberately rolled back to its recorded chain during the parallel phase and still validates, which proves the rollback path is real and not theoretical.
✓After withdrawal, a host in a withdrawn class no longer holds the outgoing anchor, confirmed by reading its store rather than by inspecting the removal job.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶During distribution, rollback is removing the new anchor from the affected class. Nothing depends on it yet, so this is safe at any point and costs only the redistribution effort.
↶During the parallel phase, rollback for a service is restoring its recorded chain file and reloading. Every client still holds both anchors, so the old chain validates immediately and the service is back within one reload.
↶If a wave fails, restore every service in that wave before diagnosing, and diagnose out of rotation. A half-cut wave is the hardest state to reason about under pressure.
↶If withdrawal of the outgoing anchor breaks a class, reinstall the anchor in that class first and investigate afterwards. The anchor is the cheapest thing in the system to put back.
↶If the new root itself proves defective before anything is issued under it, withdraw it from every class and discard it. This is the case the distribute-first ordering was designed to make survivable.
↶If a client is discovered to pin the outgoing root, roll that service back to the old chain and leave it there until the pin is removed by its owner. Do not attempt to work around a pin from the server side.
↶If rollback is needed after the outgoing anchor has been withdrawn from a class, that class must have the anchor redistributed before its services can be reverted, which is why withdrawal is last and why the second soak exists.

6 · Escalation

When the runbook isn't enough, contact:

· A trust store class cannot accept the new anchor at all, or its owner cannot commit to a date: escalate to the platform owner immediately. The programme cannot proceed past the distribution gate with a known gap, and pretending otherwise converts a schedule problem into an outage.
· A class can add an anchor but has no supported way to remove one: escalate to the platform owner and the risk owner before distribution starts, not before withdrawal. It changes the shape of the whole programme.
· A client is found to pin the outgoing root public key: escalate to the owning application team and pause every wave that touches a service it consumes.
· The outgoing root will expire before the programme can complete: escalate to the platform owner. The remaining options are a compressed programme with reduced soak, an emergency reissue, or an accepted outage, and none of those is an operator decision.
· Evidence emerges that the outgoing root key was exposed rather than merely being replaced: stop the programme and escalate to the security owner. The ordering constraints invert, and this runbook no longer applies.
· A partner or customer organisation owns a trust store in scope and will not commit to the schedule: escalate to the relationship owner and the platform owner, and treat that class as a hard constraint on the programme dates.

Replacing a root is the hardest routine change in a private PKI, and the difficulty has almost nothing to do with cryptography. A root is a trust anchor, which means its certificate is not validated by anything: it is believed, because somebody put it in a store. Changing what is believed means reaching every store in the estate, including the ones inside images you rebuild quarterly, appliances on a vendor firmware cycle, and systems owned by organisations that are not yours.

There is one ordering that survives contact with that reality, and one that does not. Distribute the new anchor everywhere first, run both anchors in parallel, then move issuance, then withdraw the old anchor. Every step in that order leaves the estate working. The inverse order, issuing under the new root and then racing to distribute the anchor, fails on every client that has not yet received it, simultaneously, at the moment of the first deployment.

Hold on to that asymmetry for the rest of this page. Installing an anchor nobody uses costs nothing. Serving a chain that terminates at an anchor a client does not hold costs a total failure for that client. The whole procedure is built to keep you permanently on the cheap side of that trade.

When this runbook applies, and when it does not

It applies when a private root must be replaced as a planned programme, the outgoing root is still valid and uncompromised, and you can identify an owner for every trust store in the estate.

It does not apply when:

The outgoing root key is compromised or suspected of being compromised. This procedure deliberately runs both anchors side by side for weeks. If the old key is in someone else’s hands, that parallel period is an open door, and the correct response is the CA compromise procedure with its very different ordering.
Only the intermediate needs replacing. If clients anchor on the root, a new intermediate under the same root reaches nobody’s trust store and is a far smaller change.
The certificates are publicly trusted. You do not operate the root of a public chain and you cannot distribute anchors to browser and operating system vendors. Changing public CA is a procurement and reissuance exercise, not a trust transition.
You cannot enumerate the trust stores. An estate where nobody can say which systems trust the root is not ready for this runbook. Building the inventory is the prerequisite, and it is usually the longest part.

The ordering, as a single picture

flowchart TD
    A["New root created offline, issues nothing"] --> B["New anchor added to every trust store"]
    B --> C{"Every class proved to hold both anchors?"}
    C -- "no" --> B
    C -- "yes" --> D["Issue intermediate under the new root"]
    D --> E["Cut issuance over in waves"]
    E --> F{"Residual on the old chain is zero?"}
    F -- "no" --> E
    F -- "yes" --> G["Withdraw the old anchor, class by class"]
    C -- "issue anyway" --> Z["unable to get local issuer certificate"]

The loop between distribution and its gate is where the time goes, and it is the only part of the programme with no deadline pressure attached, because nothing is broken while it runs. The third branch out of that gate is the outage: issuing before every class holds the new anchor makes every client in the short class fail at once, with an error that points at the server rather than at the store. Note where the failure lands. The client reports that it cannot build a path, which reads as a server problem, so the first hour of the incident is usually spent inspecting a server that is doing exactly what it was told to do.

Blast radius

Action	Reversible?	What it costs if wrong
Create the new root offline	Yes	Nothing; an unissued root is inert
Publish the new anchor	Yes	Nothing; publication is not installation
Add the new anchor to a trust store	Yes	Nothing, provided the old anchor stays
Issue an intermediate under the new root	Yes	Nothing, while the old chain is still served
Cut a service over to the new chain	Only if you kept the old chain file	Total failure for any client whose store lacks the new anchor
Withdraw the old anchor from a class	Only if that class can be redistributed	Total failure for every service still on the old chain
Destroy the outgoing root key	No	No further CRL can be signed for anything beneath it

Step 1 - Create the new root and prove it while it is still inert

Read-only / Safeon the offline CA host, before the anchor is published anywhere— Confirms the new root is self-signed, in date, and carries the constraints a trust anchor needs.

$ openssl x509 -in root-new.crt -noout -subject -issuer -dates
openssl x509 -in root-new.crt -noout -ext basicConstraints,keyUsage
openssl verify -CAfile root-new.crt root-new.crt

Subject and issuer must be identical, basicConstraints must be critical with CA:TRUE, and keyUsage must be critical with keyCertSign and cRLSign. The verify call, with the certificate supplied as both the anchor and the subject, must print OK. Discovering a defect here costs one ceremony. Discovering it after ten thousand hosts have installed the anchor costs the entire programme.

Step 2 - Distribute the new anchor to every trust store, keeping the old one

Configuration changeDebian-family operating system store, one class of many— Adds the new anchor beside the existing one. The old anchor is not touched.

$ sudo cp root-new.crt /usr/local/share/ca-certificates/runbook-lab-root-2027.crt
sudo update-ca-certificates
openssl storeutl -noout -certs /etc/ssl/certs/ca-certificates.crt

That is one class. A JVM keeps its own keystore and will not read the operating system bundle. A runtime that ships a bundle with its HTTP client will not read it either. A container image bakes the bundle at build time, so the update is a rebuild and a redeploy rather than a command. An appliance takes a firmware or a management-system push. This is why the inventory of classes, with an owner and an update mechanism against each, is the first deliverable.

Step 3 - Hold the gate

Do not proceed while any class is short. This is the step that everything else depends on, and it is the step that schedule pressure attacks first. If a class is at ninety-five per cent, the missing five per cent is not a rounding error; it is the list of clients that will fail the instant a service they use changes chain.

Soak in the parallel state for the agreed period. The soak is what finds the laptop that was off, the autoscaling group whose image predates the change, and the appliance that reverted overnight.

Step 4 - Issue under the new root, and change nothing that is served

Create the issuing intermediate under the new root and verify it, exactly as in the intermediate rotation procedure. Publish it as chain material. Then stop. Every listener continues to present the old chain, and every client continues to validate against the old anchor it has always had.

The estate is now in the state the whole design was aiming for: two anchors installed, one chain in service, and a free choice about when each service moves.

Step 5 - Cut over in waves, and test with a single-anchor client

Read-only / Safefrom a client whose store holds ONLY the new anchor— The negative test that no other check substitutes for: it proves the path terminates at the new root.

$ HOST=app.lab.example
CANEW=/etc/pki/trust-new/root-new.crt

curl --cacert "$CANEW" "https://$HOST/healthz"
openssl s_client -connect "$HOST:443" -servername "$HOST" -CAfile "$CANEW" -no-CAfile -no-CApath -no-CAstore </dev/null

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
Verify return code: 0 (ok)

Illustrative output

A client that holds both anchors will succeed on either chain, which makes it useless for proving that the cutover worked. The single-anchor client is the only observation that distinguishes “the new chain is being served” from “the old chain still validates”. Run it after every wave.

Step 6 - Withdraw the old anchor last

Common pitfalls

Symptom	Cause	Action
A client class fails immediately after a cutover wave	That class never received the new anchor, despite a green deployment report	Roll the wave back, then read the store on that class directly
Distribution reaches one hundred per cent and then falls	An image or appliance is redeploying an older bundle	Fix the build input, not the running hosts
One service fails while its neighbours succeed	A second copy of the chain, typically at a load balancer or sidecar	Find every place the chain is stored, not every place it is configured
A client still fails with both anchors installed	The client pins a public key rather than validating a path	Roll that service back and escalate to the pin owner
Withdrawal breaks a class that had been fine for weeks	A residual listener on the old chain was never found	Reinstall the anchor first; investigate afterwards
Monitoring shows all green throughout, including during a known failure	The probe reports reachability, not which root the chain terminates at	Fix the probe before continuing the programme

Verification

Verification for this runbook is deliberately adversarial: every check is designed so that the comfortable answer cannot produce a pass. A client holding only the new anchor must succeed against every cut-over service, and a client holding only the outgoing anchor must succeed during the parallel phase and fail only after the deliberate withdrawal, which proves the two phases were genuinely separate rather than accidentally simultaneous. For every trust store class, the evidence is a read-back from a sample of real hosts expressed as a fraction of the class, never a deployment status. From off-host, every cut-over listener must present a chain that terminates at the new root, and the residual probe across the whole inventory must reach zero. One service is rolled back on purpose during the parallel phase to prove the rollback path exists. After withdrawal, a host in a withdrawn class must no longer hold the outgoing anchor when its store is read directly.

Rollback

Rollback changes character at each phase, and knowing which phase you are in is most of the skill. During distribution, rollback is removing an anchor nothing depends on, and it is free. During the parallel phase, rollback for any service is restoring its recorded chain file and reloading, because every client still holds both anchors; this is the window in which the programme is genuinely safe, and it is why the second soak exists before withdrawal. If a wave fails, restore every service in it before diagnosing anything, out of rotation. If the new root proves defective before it has issued anything, withdraw it from every class and discard it. If a client is found to pin the outgoing root, roll its service back and leave it there until the owner removes the pin. After withdrawal, rollback for a class requires redistributing the anchor to that class first, which can take as long as the original distribution did.