The missing intermediate, the most common TLS failure in production

Of all the ways TLS fails in production, one accounts for more incident channels than the rest combined: the server holds a correct, unexpired, correctly named certificate, and sends only that one certificate. The material is faultless. The delivery is incomplete. Because the symptom is a certificate error, the first hour is usually spent re-issuing a certificate that was never wrong.

The fingerprint of the failure

Point openssl s_client at an endpoint that is serving its leaf alone and the diagnosis is written out in full before the handshake even finishes.

depth=0 CN=app.lab.example
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=app.lab.example
verify error:num=21:unable to verify the first certificate
verify return:1
Certificate chain
 0 s:CN=app.lab.example
   i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA

Three things in that block settle the question together. The Certificate chain section lists one entry. Entry 0’s issuer line names RunBook Lab Server Issuing CA, and no certificate with that subject appears anywhere in the block. And the depth on both errors is 0, which means the validator never got off the leaf.

The two numbers are not duplicates and they are not alternatives. Error 20 says the issuer of the certificate at this depth could not be found in any pool available to the validator, neither the certificates the peer sent nor the local anchor store. Error 21 follows because the chain that was assembled has length one, so the signature on that single certificate could never be checked against anything. Seeing 20 and 21 together at depth 0 is close to conclusive: the peer sent one certificate and it was not self-signed.

The same condition reaches an application as a much blunter message.

curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)

That capture is worth dwelling on, because the host that produced it already held the root in its trust store. The anchor was present, the anchor was correct, and the connection still failed. Nothing about adding roots would have helped, because the gap was one level below the anchor. Exit status 60 is curl’s generic peer verification failure and carries no further detail on its own; the verify result in the text is where the diagnosis lives.

Proving the certificate is innocent

Before touching the server, establish which half of the problem you have. Take a copy of the leaf and a copy of the intermediate you believe should be in the bundle, and run the validation offline with the roles made explicit.

ROOT=root.crt
ISSUING=srv-ca.crt
LEAF=app.crt

# Model the server sending the intermediate.
openssl verify -CAfile "$ROOT" -untrusted "$ISSUING" "$LEAF"

# Model the server sending nothing but the leaf.
openssl verify -CAfile "$ROOT" "$LEAF"

app.crt: OK
error 20 at 0 depth lookup: unable to get local issuer certificate

One input changed between those two runs, and it was not the certificate. The leaf’s signature, its validity window, its names and its extensions are identical in both cases. Supplying the intermediate as untrusted path material turns a failure into app.crt: OK, so the issued material is sound and the entire defect is that the server never put the intermediate on the wire. That single pair of commands is the difference between a fifteen minute fix and an unnecessary emergency re-issuance.

Why it works in a browser and fails in the pipeline

The most costly property of this fault is that it is invisible to the people most likely to look first. Someone opens the site, sees a padlock and declares the certificate fine, while a batch job and a Kubernetes readiness probe are both failing against the same address.

Verifiers differ in how hard they work to complete a path that arrived incomplete. Several browser and platform verifiers cache intermediates they have seen on earlier connections and reuse them as candidates later, and some will fetch the missing certificate over plain HTTP from the caIssuers URL in the leaf’s authorityInfoAccess extension. Those behaviours repair the chain silently at the client. OpenSSL does none of it. It builds a path from the certificates in front of it and stops. So curl, a Go binary, a Python worker and a Java service will all report a failure that a desktop browser never shows.

flowchart TD
    A["Client reports verify error 20"] --> B{"Does showcerts list more than one entry?"}
    B -- "no" --> C["Server sent the leaf alone: fix the served bundle"]
    B -- "yes" --> D{"Does offline verify with -untrusted pass?"}
    D -- "yes" --> E["Anchor absent on this client: fix the trust store"]
    D -- "no" --> F["Chain material is wrong: wrong or stale intermediate"]

The flow encodes the order that keeps you out of trouble. Count the entries the server actually sent first, because that answer is a property of the server and is identical from every vantage point. Only when the transmitted chain is complete does the question become one about the client’s trust store, and only then is a trust store change a sensible response.

Where the intermediate goes missing

Almost every occurrence traces back to a file, and usually to a file that was correct once. Certbot is a good illustration because it places both a right and a wrong answer in the same directory, under names that are easy to confuse in a configuration template.

A service configured against cert.pem starts cleanly, passes a configuration syntax test, serves traffic and fails verification for a large fraction of clients. Nothing in the service logs will call that out, because from the server’s point of view nothing went wrong. Those four names are symlinks into an archive directory, and the symlink target is repointed on each renewal, which is what allows a service to keep one path in its configuration for years.

Choosing the replacement intermediate is a question with exactly one correct answer, and downloading a plausible looking certificate from a search result is not it. The one you need has a subject that matches the leaf’s issuer field character for character, and the authoritative pointer to it is inside the leaf itself: the authorityInfoAccess extension carries a caIssuers location published by the CA that signed it. Read the extension, fetch from the CA, and confirm the match offline before deploying anything.

The other recurring causes are worth naming so you recognise them quickly. A deployment script that copies the leaf and forgets the chain. A load balancer pool where one member was updated by hand and the others were not, producing a fault that appears on roughly one connection in three. A CA that rotated its issuing certificate, leaving a bundle whose intermediate no longer matches the leaf’s issuer. And a bundle whose second certificate is present but unparseable, which is the case worth measuring rather than eyeballing.

BUNDLE=/etc/ssl/private/app.lab.example-fullchain.pem

# Count what the file really contains, rather than what it looks like.
openssl storeutl -noout -certs "$BUNDLE"

# Rebuild it deliberately: leaf first, then each issuer above it.
cat app.crt srv-ca.crt > "$BUNDLE"

storeutl enumerates every certificate it can parse out of the file, so a bundle that should hold two and yields one has told you where to look without any guesswork about stray whitespace or a truncated copy.

Production discipline

1. Point services at the full bundle by convention. Make the deployment artefact a file that always contains leaf plus issuers, and never let a leaf-only file exist in a directory a service reads.
2. Assert the count, do not trust the name. A check that the served file parses to the expected number of certificates catches the whole class before a reload.
3. Probe with a client that does not repair chains. A curl or OpenSSL probe from a machine with no browser history is the honest test; a padlock in a browser is not evidence.
4. Re-verify after every renewal. The intermediate can change underneath you when the CA rotates, so a bundle that was complete last quarter is not evidence about the one deployed this morning.

Cross-course references

• Linux for Production Sysadmins - Part LXXIX (Troubleshooting) covers narrowing a fault to one layer before changing anything, which is exactly what separating delivery from material achieves here.
• Kubernetes for Production Sysadmins - Part CXIX (PodTroubleshoot) covers readiness probes that fail while a browser succeeds, a symptom pattern this failure produces almost perfectly.
• Observability for Production Sysadmins - Part XI (Blackbox) covers probing endpoints from a defined vantage point, which turns an incomplete chain into an alert rather than a customer report.

Quiz