The new root went live at 22:04 and the payment path stopped four minutes later

Reported symptoms

A new internal root exists because the previous hierarchy had to be replaced, which means this change has been carrying a deadline and a certain amount of impatience for six weeks. The certificates are issued, the chains are staged, the change record is written, and the window is 22:00 to 23:00 UTC on 25 August.

At 22:04 UTC the last service picks up its new chain. Every step in the runbook has reported success. The change is marked complete in the record at 22:05.

• 22:06 UTC. The reconciliation job fails. It cannot open a connection to the payments service, and it retries, and it fails again.
• 22:11 UTC. The containerised workers start failing as their connection pools recycle. They had been running happily on connections opened before 22:04, which is why the first five minutes looked fine.
• 22:14 UTC. Somebody points out that most of the estate is completely unaffected. Four fifths of internal traffic is moving normally.
• 22:15 UTC. Two engineers say, independently, that they cannot reproduce it from their laptops. Both are correct. Both laptops have the new anchor, delivered by the device management system hours earlier.
• The ingress health checks are green and stay green for the whole incident, because the host they run on is an ordinary Linux host in the fleet.

The incident is declared at 22:19 UTC. It takes until 22:34 for somebody to run the same request from inside a container rather than from a shell on a host, and the failure reproduces immediately.

The chain is rolled back at 22:41 UTC and everything recovers within a minute. Total impact: thirty-seven minutes, all of it on the payment path. And the awkward detail that shapes the rest of the week is that the certificates were never wrong. Every one of them was correct, complete and properly issued.

Evidence provided

The fastest way into this incident is two identical requests from two places, and the pair is worth reading before anything else.

$ curl -sS -o /dev/null -w '%{http_code}' https://payments.internal.example.com/healthz

200

$ docker run --rm curl-runner:2026-07 curl -sS https://payments.internal.example.com/healthz

curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)

Same URL, same service, same second. The difference is not the certificate and not the network; it is which certificate authority bundle the process is reading. Everything after this is working out which processes read which bundle.

The chain the server presents is not in question, and it is worth proving that early so that nobody spends the night editing nginx.

$ echo | openssl s_client -connect payments.internal.example.com:443 -servername payments.internal.example.com 2>/dev/null | grep -E 'Verify return code|^ [0-9] s:|^   i:'

 0 s:CN=payments.internal.example.com
 i:O=Example Internal, CN=Example Internal Issuing CA G3
1 s:O=Example Internal, CN=Example Internal Issuing CA G3
 i:O=Example Internal, CN=Example Internal Root CA G2
Verify return code: 0 (ok)

Now the same certificate examined against each of the two anchors in turn. This pair is where the diagnosis becomes unambiguous.

$ openssl verify -CAfile /etc/pki/roots/root-g1.crt -untrusted /etc/pki/chains/issuing-g3.crt /run/incident/payments-leaf.pem

error 2 at 1 depth lookup: unable to get issuer certificate

$ openssl verify -CAfile /etc/pki/roots/root-g2.crt -untrusted /etc/pki/chains/issuing-g3.crt /run/incident/payments-leaf.pem

/run/incident/payments-leaf.pem: OK

Read those two together and the fault has an address. The error is reported at depth 1, which is the intermediate, and it says the issuer of the intermediate could not be found. The client built as much of the path as it could and then ran out of trust anchors. The leaf is fine, the intermediate is fine, the chain the server sends is complete. The client simply does not have the certificate at the top.

The last reading is the one that should have been taken before the change window opened.

$ openssl x509 -in /etc/pki/roots/root-g2.crt -noout -fingerprint -sha256

sha256 Fingerprint=2F:1B:C4:90:7A:35:E8:6D:04:B2:9C:71:AE:38:F5:60:D1:8C:47:2A:B3:09:E6:74:5F:C8:12:AD:30:6B:99:E4

Counting that fingerprint across the estate, by class of trust store rather than by host, gives the coverage nobody had measured.

Ninety-five and a half per cent. The seventy-five stores in the remaining four and a half per cent include every service on the payment path.

Work the evidence before reading on

Nothing here is broken in the sense the team spent its first fifteen minutes assuming. The certificates are correct and so is the server configuration.

1. openssl verify reports the failure at depth 1 rather than depth 0. Say what that single number tells you about where to look, and what it rules out.
2. The identical request succeeds from a host and fails from a container on that same host. Name the thing that differs, and name when it was decided.
3. Rolling back the served chain restored service in under a minute. What property of the estate made that possible, and which step in the original plan would have destroyed it?
4. Coverage was 95.5 per cent of trust stores. Explain why that number is not a useful measure of readiness, and propose the number that is.

Before continuing: the plan had distribution before cutover, in the right order, and it still failed. State precisely what was being treated as evidence of distribution, and what should have been.

Root cause

The anchor changed before the trust stores did

The invariant a trust anchor migration has to preserve is short enough to fit on one line: at every instant, every client must trust the anchor of the chain being served. This change broke it at 22:04 UTC, and the four-minute delay before the first failure is just how long it took for a connection pool to recycle.

The plan was not naive. It listed distribution as step three and the chain cutover as step seven, in that order, which is correct. What it did not do was define what finishing step three meant, so the step was ticked when the configuration management run went green.

That is the whole ordering failure, and it is a small one. The steps were in the right sequence; the gate between them was a report rather than a measurement.

Configuration management defined the estate, and the estate was larger

The push succeeded on 1,340 of 1,340 Linux hosts. That statement is true and it was never the question. Configuration management writes the operating system bundle on machines it manages, and there are four kinds of trust store in this estate that it has no relationship with at all.

The JVM services read a truststore file chosen at process start, which is a separate file with a separate format and a separate update tool. Container images carry a bundle baked in at build time, so an image built in July contains July trust regardless of what happens on the host it runs on. The CI runner image is the same problem with a different owner. The appliances have no shell and accept configuration only through their own interface.

None of this was hidden. It had simply never been load-bearing, because no trust anchor had changed since the hierarchy was built, and a fact that has never mattered does not get written down. The inventory that existed was an inventory of hosts, and hosts were exactly the wrong unit: 1,340 identical Linux stores counted 1,340 times, while a container image serving forty workloads counted zero.

Percentage coverage hid the shape of the gap

The coverage figure, had anyone computed it, was 95.5 per cent, and that number would have been reassuring. It is the wrong statistic in a way worth naming, because the instinct to compute it is strong.

Trust is not a proportion. A client with the anchor works and a client without it fails, so the meaningful measure is how many classes are complete, not how many stores are covered. By that measure the estate was at two classes out of seven, or 29 per cent, which nobody would have signed off.

The missing seventy-five stores were also not a random sample. They were the JVM services, which is where the payments and reconciliation code lives, and the container images, which is where the workers live. A gap in an estate correlates with something, because the reason a store was missed is usually that it belongs to a different platform generation, and platform generations map onto workloads.

Resolution

1. Roll the served chain back before diagnosing anything else. Revert each service to the old chain and reload. This is safe because the old anchor is still installed in every trust store, including the ones that received the new one, and it should take a reload rather than a rebuild. Restoring service first also gives the investigation the thing it most needs, which is time.
2. Confirm recovery from a client of each class that failed, not from the host you happen to be logged into. The container that reproduced the failure at 22:34 is the best available witness, so use it.
3. Leave the new anchor in place wherever it was successfully installed. It is harmless alongside the old one, it is progress you do not want to repeat, and removing it would turn a partial distribution into a fresh one.
4. Rebuild the inventory by class rather than by host. Operating system bundles, JVM truststores, container base images, the CI runner image, network appliances, managed laptops, and anything else that terminates or initiates TLS. Give each class a named owner, an update mechanism and a command that reads the store.
5. Distribute the new anchor to every remaining class while the old chain continues to be served. For the JVM services this is a truststore update and a restart; for the images it is a rebuild and a redeploy; for the appliances it is a firmware release with a date you do not control.
6. Decide the appliances explicitly rather than leaving them to the end. Either the cutover waits for the firmware release, or a cross-signed certificate for the new intermediate signed by the old root is served alongside the new chain so that both anchor populations can build a path. Write down which was chosen and why, because this is the decision the next reviewer will ask about.
7. Gate the second attempt on measured coverage. Every class complete, verified by reading the store rather than by reading the exit status of whatever wrote to it. Not a percentage of endpoints, a count of classes, and the count must be all of them.
8. Re-cut the served chain service by service rather than in one window, starting with a service that has no dependants and watching a client of each class after each step. A staged cutover turns an estate-wide failure into a single-service failure with an obvious cause.
9. Set the soak at 14 days minimum before removing the old anchor, and choose it deliberately to cover anything that connects weekly or monthly. Batch jobs, reporting runs and quarterly reconciliation are the connections that will not appear during the first night.
10. Remove the old anchor as a separate, scheduled change with its own rollback plan. It is the only irreversible step in the whole transition and it deserves its own window rather than a line at the bottom of somebody else's change record.

Verification

1. Read the trust store itself for a sample of each class, and compare against the recorded fingerprint of the new anchor. The operating system bundle on a Linux host, the truststore file each JVM service is genuinely started with rather than the one in the documentation, the bundle inside a freshly pulled image, the CI runner image, the appliance interface and a managed laptop. Seven classes, seven readings, seven pieces of evidence.
2. Treat a successful push as an intent and the store as the state. Configuration management reporting success proves that a file was written where it was told to write one; only reading the store proves the anchor is where a client will look for it.
3. Connect from one representative client of every class to a service serving the new chain and confirm the handshake validates. Presence of the anchor and a working handshake are different claims, and the second is the one users experience.
4. Confirm the old anchor is still present everywhere until its scheduled removal. This reads backwards during a verification pass and it is the property that keeps the rollback available, so it belongs in the checklist rather than in somebody's memory.
5. Exercise the rollback in staging and record how long it takes. A rollback nobody has timed is a paragraph, and the number is what lets the incident commander choose it at 22:19 rather than at 22:41.
6. After each stage of the re-cut, run the failing case from the class that failed first. Evidence from the population that already broke once is worth more than evidence from the population that never had a problem.
7. Before removing the old anchor, scan for any client still building a path through it and require a count of zero. A weekly job that has not run since the cutover is not evidence of success, it is evidence of not having been asked yet.

Prevention

• Make distribution the gate, and make the gate a measurement. The invariant is one sentence: at every instant, every client trusts the anchor of the chain being served. Distribute, verify by reading stores, then cut over, then soak, then remove. Every incident of this shape is that ordering violated in one place.
• Inventory trust stores by class, not by host. Hosts are what configuration management knows about; classes are what actually behave differently. Add a class to the inventory the day a new runtime enters the estate, because the gap is always the platform generation nobody has migrated yet.
• Count classes, not endpoints. Ninety-five and a half per cent of stores sounds like readiness and 29 per cent of classes sounds like what it was. Require every class complete before a cutover, and refuse to average across them, because trust is not a proportion and the missing few per cent is the outage.
• Cross-sign when a class cannot be updated in time. A certificate for the new intermediate public key signed by the old root lets one served chain satisfy both anchor populations. Build the plan around the firmware release date for the appliances rather than discovering it in the change window.
• Health check from a client of every class. A check that runs on an ordinary managed host will stay green through exactly this failure. Put a probe inside a container, inside a JVM and behind each appliance, so that the monitoring population resembles the client population.
• Publish a tested, timed rollback in the change plan. For a chain change the rollback is a reload, it takes seconds, and it is only available while the old anchor is still installed everywhere. Writing that down is what makes it the first action rather than the twenty-second minute.
• Rehearse in staging with the production class mix. A staging estate of Linux hosts rehearses one seventh of this change. Give it a JVM service, a container image and an appliance stand-in, or accept that the rehearsal is measuring the easy part.