Half the fleet cannot reach the service and the other half never noticed

Reported symptoms

app.lab.example is an internal platform service on the private two-tier PKI: an offline root, an issuing CA that signs everything, and a ninety-day leaf that is rotated by a scheduled job. Every host and every container image in the estate carries the root as a trust anchor. Roughly forty internal services call it. The rotation job has run without incident for two years.

It ran at 21:19 UTC on 26 August and the estate came apart fifteen minutes later:

• 21:20, callers begin failing at connection time. By 21:24 roughly nine in ten of them are failing continuously, and none of them ever sends a request.
• Two hosts, batch-07 and batch-08, keep calling the service at their normal rate with a zero error count. The first responder spends ten minutes looking for a network partition that would explain a split like that, and finds none.
• The service is fine. It accepts connections, its local health endpoint answers, and the error rate for requests it actually received is zero, because the requests that fail never arrive.
• 21:38, the post-rotation canary is re-run by hand. It passes, as it did automatically at 21:21.
• 21:45, a freshly built container is deployed to test a theory about image drift. It fails identically to everything else.
• Every failing client reports the same verification failure, naming an issuer it cannot locate. The wording does not vary between languages, runtimes or images.

At 21:52 somebody notices that the previous certificate is still on disk and is nine days from its own expiry, and proposes rolling the rotation back. It is the only suggestion in the channel that would have restored service, and it is dismissed because the certificate that was just installed is obviously the newer one.

Evidence provided

$ openssl s_client -connect app.lab.example:443 -servername app.lab.example </dev/null 2>&1 | grep -E 'depth=|verify error|verify return'

depth=0 CN=app.lab.example
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=app.lab.example
verify error:num=21:unable to verify the first certificate
verify return:1

$ openssl s_client -connect app.lab.example:443 -servername app.lab.example -showcerts </dev/null 2>/dev/null | grep -E 'Certificate chain|^ [0-9]+ s:|^   i:'

Certificate chain
0 s:CN=app.lab.example
 i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA

$ curl -sS https://app.lab.example/status

curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)

$ openssl verify -CAfile /usr/local/share/ca-certificates/runbook-lab-root.crt -untrusted srv-ca.crt app.crt

app.crt: OK

$ openssl verify -CAfile /usr/local/share/ca-certificates/runbook-lab-root.crt app.crt

error 20 at 0 depth lookup: unable to get local issuer certificate

$ grep -c 'BEGIN CERTIFICATE' /etc/nginx/tls/app.crt /etc/nginx/tls/app-fullchain.crt

/etc/nginx/tls/app.crt:1
/etc/nginx/tls/app-fullchain.crt:2

Work the evidence before reading on

The service is healthy and the certificate is correct. Neither of those facts is in tension with an estate-wide outage, and working out why is the whole exercise.

1. Two openssl verify runs against one file, one anchor and one difference produce OK and error 20. Which components does that pair exonerate, and what is the only thing left?
2. The chain listing has one entry. The leaf says it was issued by the issuing CA and the trust store holds the root. Count the links the client needs and the links it was given.
3. batch-07 succeeds against the same URL, the same DNS answer and the same clock as a host that fails. Something on batch-07 substitutes for what the server stopped sending. What is it, and which command in the evidence names it?
4. The canary passed twice. Say what a canary running on batch-07 is actually capable of detecting, and what it has never been able to detect.

Before continuing: state the difference between a certificate the client cannot trust and a certificate the client cannot connect to an anchor it already trusts, and name the single command that tells the two apart.

Root cause

The server stopped sending the intermediate

Chain validation is a path problem. The client holds the root. The leaf declares that it was issued by the issuing CA. Between those two facts there has to be a certificate showing that the root signed the issuing CA, and the client does not have one. The server is the component expected to supply it, and after the rotation the server had only the leaf to send.

That is exactly what the chain listing shows: one entry, index 0, issued by an authority that appears nowhere else in the exchange. It is also what the two openssl verify runs prove. Supplied with the issuing CA the leaf verifies cleanly. Denied it, the identical leaf against the identical anchor fails at depth 0 with error 20. Nothing about the certificate is wrong. Something about the transport is incomplete.

The two errors reported by the diagnostic client describe the same gap from two angles. Error 20 says a local issuer for this certificate could not be found. Error 21 says the first certificate in the presented path could not be verified, which is the summary the client produces once path building has failed and there is nothing left to try.

The deployment picked up the file one word away

The issuance job writes two files into the same directory: the leaf alone, and a bundle of the leaf followed by the issuing CA. The rotation copied the leaf. The certificate count makes the mistake visible in a single line, one certificate in the file the server was reading and two in the file beside it.

No component could have objected. A TLS server does not validate its own chain, and there is no reason it should: a server has no idea what its clients trust, and an incomplete path is a perfectly legal thing to send. The consequence is that a server will start, log nothing, report healthy, and refuse every connection from every client that does not already hold the missing link.

The canary was measuring the wrong host

batch-07 was built before the trust-store clean-up in 2025 and still carries the issuing CA in its local anchor directory alongside the root. A client with the intermediate locally does not need the server to send it, so batch-07 completes the path from its own store and succeeds. batch-08 is its twin. Those two hosts are the entire population that kept working.

The canary runs on batch-07, and has since it was written, because it was the host with the tooling. That makes it structurally unable to detect a missing intermediate. It passed honestly, twice, and it would have passed the next time as well.

Resolution

1. Separate a path failure from a trust failure before changing anything. Verify the served leaf against the root twice, once with the issuing CA passed as an untrusted intermediate and once without. OK followed by error 20 means the anchor is right and only the transport is incomplete, which is a server-side repair.
2. Confirm what the server is actually sending, by counting the entries in the chain listing rather than by reading the configuration. One entry against a two-tier PKI is the fault, stated in a single number.
3. Point the server at the bundle. Concatenate the leaf first and the issuing CA second into one file, and reference that path from ssl_certificate. Order matters: the leaf must come first, and every certificate after it must be the issuer of the one before.
4. Leave the root out of the bundle. The client already holds it, sending it adds bytes to every handshake, and some clients object to a self-signed certificate appearing in a presented path. Two certificates is the correct answer here, not three.
5. Validate the configuration before applying it, then reload rather than restart so that connections drain instead of dropping. Confirm the reload was accepted rather than assuming it.
6. Read the chain back from a host that carries only the root, ideally a container started fresh for the purpose. Reading it from batch-07 proves nothing, because batch-07 is the reason nobody noticed.
7. Keep the previous certificate on disk until the repair is proven. It is nine days from expiry and is not a solution, but it is a rollback, and a rollback you have already deleted is not one.
8. Do not install the issuing CA into client trust stores to clear the errors. It works, and it permanently removes the estate ability to detect this fault while widening what an online key is trusted to do.
9. Do not disable verification on the callers, and do not tell anyone to pass an insecure flag while the fix is prepared. The repair is one file and one reload, and a flag added under time pressure outlives the incident by years.

Verification

1. From a container started fresh with the root as its only private anchor, open a TLS connection and read the chain the server presents. It must contain two entries: the leaf at index 0 issued by the issuing CA, and the issuing CA at index 1 issued by the root.
2. That same connection must report a verify return code of 0 and the word ok, which is the observation that was returning error 20 before the repair.
3. Confirm there is no third entry in the presented chain. Adding the root is the standard overcorrection and it is worth ruling out explicitly rather than assuming the bundle was built correctly.
4. Request the application endpoint with verification enabled from at least three callers that were failing during the incident, including the container that was rebuilt at 21:45, and confirm HTTP 200 with the expected body rather than a successful handshake alone.
5. Confirm batch-07 still works. It is not at risk, but a repair that breaks the two hosts that were fine would be a poor trade and takes one command to exclude.
6. Confirm the certificate count in the file the server now reads is two, so that the next rotation copying the same path cannot silently reintroduce the fault.
7. Prove the canary now detects the fault: run it from a rebuilt minimal environment against the pre-repair configuration and require it to fail. A canary that has never failed has never been tested.

Prevention

• Make the deployable artefact the bundle. If issuance writes one file containing the leaf and every intermediate, and the server configuration references only that path, there is no leaf-only file in the directory for a deploy step to reach for. The fix is naming, not vigilance.
• Count the presented certificates after every rotation. A post-deploy check that opens a connection and asserts the chain length is exactly two runs in under a second and catches this entire class of fault the moment it is introduced.
• Build every canary environment from scratch, with the minimum trust. A canary on a long-lived host slowly becomes a test of that host. Rebuild it per run, give it the root and nothing else, and it starts failing for the same reasons your least privileged client does.
• Audit local trust stores quarterly. The two hosts that masked this were carrying an anchor nobody had authorised, and nobody had ever looked. Alert on any host holding an anchor outside the approved set.
• Alert on client-side verification failures, not only server errors. A server with a broken chain reports a perfectly healthy error rate, because it is only counting the requests that reached it. The signal lives at the callers.
• Rotate with margin, so rollback stays real. Replacing a certificate nine days before it expires means the previous file buys you a week. Rotating at a third of the lifetime remaining keeps a genuine fallback on disk for the whole change window.