Identify the layer before you touch the configuration

A TLS failure is not one failure. It is any of at least five, each with its own evidence, and at three in the morning they are routinely mistaken for one another. The discipline that turns a three hour incident into a ten minute one is refusing to change a single line of configuration until you can say which layer stopped, on which host, observed by which client.

Five layers, and the one that stopped

A client reaching an HTTPS endpoint passes through five stages in strict order. Each stage can only fail after the previous one has succeeded, which is what makes the ordering diagnostic rather than academic.

• Name resolution and transport. The name becomes one or more addresses; a TCP connection opens to a port. Nothing here is TLS. A refused connection or a timeout is a routing, firewall or service-availability problem wearing a TLS costume because the URL happened to start with https.
• Protocol negotiation. ClientHello and ServerHello agree a version and a cipher suite. If the two ends share no version the connection dies before a certificate is ever transmitted, so no amount of certificate work will help.
• Chain delivery and path building. The server sends a certificate list. The client tries to build a path from the leaf up to something in its own trust store, using the certificates the server sent plus whatever it already holds.
• Identity and constraint checking. Only once a path exists does the client ask the questions people think of as certificate validation: is this the right name, is it inside its validity window, is it permitted for this purpose.
• The application exchange. The request is sent. A 502 here is not a TLS failure even though TLS carried it.

flowchart TD
    A["name resolves to an address"] --> B["TCP connects on port 443"]
    B --> C["ClientHello and ServerHello agree a version"]
    C --> D["server sends its certificate list"]
    D --> E["client builds a path to a trusted anchor"]
    E --> F["client checks name, dates and purpose"]
    F --> G["application request is sent"]
    B -- "refused or timeout" --> X1["not a TLS problem at all"]
    C -- "no protocols available" --> X2["version floor, on one side or the other"]
    E -- "num=20 or num=2" --> X3["chain delivery or trust store"]
    F -- "error 62" --> X4["the name, not the trust"]

The diagram is a checklist read downwards. Work out how far the connection travelled before it stopped, and you have already excluded every layer below that point. A hostname mismatch, for example, is proof that negotiation succeeded, that the server sent a complete chain, and that the chain reached a trusted anchor. Everything about the trust configuration is working; the certificate simply carries a different name.

Evidence before change, always

The single most expensive habit in TLS troubleshooting is editing a configuration file to see whether it helps. It destroys the evidence, it changes the endpoint for every other client at the same moment, and when the symptom moves you can no longer tell whether you fixed the fault or merely relocated it.

Capture first, into a directory named after the host and the timestamp, so that the record survives the shift change.

HOST=api.example.com
PORT=443
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
OUT="$HOME/tls-evidence/$HOST-$STAMP"
mkdir -p "$OUT"

date -u > "$OUT/clock.txt"
getent hosts "$HOST" > "$OUT/resolution.txt"
openssl s_client -connect "$HOST:$PORT" -servername "$HOST" \
  -showcerts </dev/null > "$OUT/s_client.txt" 2>&1
echo "$?" > "$OUT/s_client.exit"
curl -sS -o /dev/null "https://$HOST/" > "$OUT/curl.txt" 2>&1
echo "$?" > "$OUT/curl.exit"

Five files, thirty seconds, and every subsequent argument in the incident channel becomes checkable. The clock capture is not padding: a validity window is only meaningful relative to a clock, and the host you are standing on is one of the two clocks that decide the outcome.

Let the difference do the diagnosis

One capture tells you that something is wrong. Two captures, one from a client that works and one from a client that does not, tell you where. Run the identical command from both and compare the certificate list the server sent.

Here is a real capture from a server configured to send only its leaf certificate:

depth=0 CN=app.lab.example
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=app.lab.example
verify error:num=21:unable to verify the first certificate
verify return:1
Certificate chain
 0 s:CN=app.lab.example
   i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA

And here is the same server after the intermediate was added to the file it serves:

Certificate chain
 0 s:CN=app.lab.example
   i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
 1 s:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
   i:O=RunBook Academy Lab, CN=RunBook Lab Root CA
Verify return code: 0 (ok)

The trust store did not change between those two captures. The client did not change. The only difference is that the second list has an entry 1, and its issuer line names the root. That single structural difference localises the defect to the server and identifies the fix as a file change on the server, not a trust change on every client that will ever connect.

Four variables, one at a time

When two observers disagree about the same endpoint, exactly four things can differ between them. Hold three fixed and vary the fourth.

• The client. Its TLS library, its version, and above all its trust store. A browser, a Java service and a container running curl do not share one set of trust anchors.
• The endpoint actually reached. A name may resolve to several addresses, only one of which is misconfigured. A terminating proxy, not the origin, may be the real TLS peer.
• The path. An intercepting middlebox re-signs traffic with its own certificate, so the certificate you inspect from inside the office is not the one the internet sees.
• The clock. Two hosts disagreeing by minutes will disagree about whether a freshly issued certificate is valid yet.

When the report and the evidence disagree

Incident reports arrive as sentences like “the API is down” or “certificates have expired again”. Both are hypotheses. The evidence you captured either supports them or it does not, and the mismatch itself is informative.

If a browser succeeds where curl fails on the same laptop, the two are consulting different trust stores, and the defect is a missing anchor on the failing side. If the failure appears once in every four attempts, you are talking to a pool and one member is wrong; repeat the capture against each address individually rather than against the name. If every client on one host fails and every client elsewhere succeeds, suspect the host, starting with its clock and its bundle of trust anchors.

Resist the pull of the most recent change. It is often the cause, but “we deployed yesterday” is not evidence, and a certificate that expired at a fixed instant will fail on a service nobody has touched for a year.

There is one structural asymmetry that shapes every one of these investigations. Certificate validation is a decision the client makes, alone, about material the server has already finished sending. The server does not learn the verdict. From its side a rejection looks like a peer that opened a connection, received the certificate message and then went away, which most web servers record as an ordinary aborted connection if they record it at all. This is why demanding evidence from the service owner so often produces the answer that nothing is wrong: from their vantage point, nothing visible is. The client-side capture is not one perspective among several, it is where the decision was actually taken, and it is the artefact the investigation has to be built on.

Production discipline

1. Capture before you touch anything. The failing observation is unrepeatable once the configuration changes, and it is the only thing that will let you prove causation afterwards.
2. Say the layer out loud before proposing a fix. If you cannot name which of the five stages stopped, you are not ready to change a file.
3. Change one variable per experiment. Client, endpoint, path, clock. Two changes at once produce a result you cannot attribute.
4. Prefer the fix that is nearest the defect. A server that sends an incomplete list is a server-side bug; distributing extra anchors to clients to compensate spreads the fault instead of removing it.
5. Record the exit status, not only the message. Wording varies between builds and versions; the numeric status is the stable part of the interface.

Cross-course references

• Linux for Production Sysadmins - Part LXXIX (Troubleshooting) covers the general evidence-first investigation loop that this lesson specialises to TLS, and Part XXII (NetTroubleshoot) covers proving the transport layer before blaming anything above it.
• Kubernetes for Production Sysadmins - Part CXVIII (Methodology) covers the same discipline applied to cluster incidents, where the number of candidate layers is larger and the temptation to guess is correspondingly stronger.
• Observability for Production Sysadmins - Part CIX (InvestigationWorkflows) covers turning a repeated manual capture into an automated probe, so the next occurrence is detected rather than reported.

Quiz