← All runbooks in Secrets, PKI & Certificates

low riskinformational~30 min

Runbook: Troubleshoot a TLS Validation Failure

1 · Prerequisites

Confirm every item is in place before any state change.

The verbatim error text from the client that is failing, copied rather than paraphrased, including any error code in brackets
A shell on a host that reaches the endpoint by the same network path as the failing client, or as close to it as the estate allows
OpenSSL 3.x, so that s_client and the verify sub-command with its -untrusted, -verify_hostname and -purpose options are available
The identity of the trust anchor the failing client is expected to use: the operating system bundle, a language runtime bundle, or a private root shipped with the application
Agreement on which client is authoritative for the incident, because two clients with different trust stores will disagree and both readings will be correct
A scratch directory you can write capture files into, outside any path the service reads

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Establish that the failure is validation and not connectivity. A refused connection, a reset, or a timeout is a different incident. Confirm the TCP handshake completes before spending any time on certificates.
· Establish which client is authoritative. Curl on your laptop, a Java service, a Python worker and a browser each carry different trust stores and different defaults. Pick the one whose failure defines the incident and state it.
· Copy the error string exactly. Almost every cause in this runbook is distinguished by wording rather than by symptom, so a paraphrase such as "certificate error" destroys the discriminating evidence before the investigation starts.
· Note whether the failure is universal or partial. One client failing while others succeed points at a trust store or a name; every client failing points at the server or at the certificate itself.
· Note when it last worked. A failure that starts at a precise moment with no deployment usually means something expired. A failure that starts immediately after a change usually means the change.
· Check that name resolution agrees with expectation. Run getent hosts "$HOST" and confirm you and the failing client are talking to the same address, because a stale record sends you to a server holding a different certificate.
· Refuse the shortcut before anyone offers it. Agree explicitly that verification will not be disabled to see whether that fixes it, because doing so destroys the only signal that distinguishes the six causes below.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Record the exact client error and the exact target. Host, port, the server name the client sends, and the error verbatim. These four values are the input to every branch and are worth more than any hypothesis offered in the incident channel.
2Reproduce the handshake once and capture everything. Run openssl s_client -connect "$HOST:$PORT" -servername "$HOST" -showcerts with stdin redirected from /dev/null and keep the whole transcript. One capture answers most of the questions that follow, and repeated live connections during an incident add noise to the server logs.
3Read the verify lines at the top of the transcript before anything else. They report the depth at which validation stopped and the numeric reason. Depth 0 is the server certificate, depth 1 its issuer, and the depth alone eliminates several causes.
4Confirm whether a certificate was received at all. If the transcript contains no certificate chain, the failure happened before certificate validation, and this is a protocol, cipher or reachability question rather than a PKI question.
5Split the captured chain into the leaf and the rest. openssl x509 -in served.pem -out leaf.pem writes the first certificate; the full capture remains available as the untrusted pool for the next steps.
6Test expiry first, because it is cheap and unambiguous. openssl x509 -in leaf.pem -noout -checkend 0 reports Certificate will not expire while the certificate is inside its window. A certificate outside its window explains the failure completely and ends the branch.
7Test the chain with the captured intermediates offered as untrusted. openssl verify -CAfile "$ROOT" -untrusted served.pem leaf.pem. Passing here while the live client failed is the proof that the server did not send a complete chain, because you supplied what it withheld.
8Test the trust anchor separately from the chain. Repeat the verify with the trust store the failing client actually uses. A result of error 2 at 1 depth lookup: unable to get issuer certificate means the root is unknown to that client, which is a trust distribution problem and not a server problem.
9Test the name explicitly rather than reading the SAN and judging by eye. Add -verify_hostname "$HOST" to the verify invocation. Matching rules for wildcards and trailing dots are not intuitive, and delegating the comparison to the library removes an entire class of wrong conclusion.
10Test the purpose. Add -purpose sslserver. A client certificate installed on a server listener, or a certificate whose extended key usage omits server authentication, fails here with error 26 at 0 depth lookup: unsuitable certificate purpose and passes every other check.
11Only now consider protocol and cipher policy. A modern OpenSSL build refuses to offer obsolete protocol versions from the client side, so a failure to negotiate is not by itself evidence of a server policy. Read which side refused before blaming either.
12Name the single cause the evidence supports and hand off to the matching procedure. Chain, name, trust store, expiry, purpose and protocol each have their own remediation, and the whole value of this runbook is that it stops the call from attempting more than one of them at once.
13Attach the capture and the verify results to the incident record. The next responder should be able to reach the same conclusion offline, without reconnecting to a production endpoint.

4 · Verification

Confirm the procedure actually fixed the problem.

✓Exactly one cause is named in the incident record, supported by a quoted verify result rather than by a summary of it.
✓The capture file and the verify command lines are attached, so the conclusion can be reproduced offline by someone who was not on the call.
✓The client that defined the incident has been re-tested after the remediation and now completes the handshake with verification fully enabled.
✓A second, independent client on a different host and trust store has been tested, so success is not an artefact of one machine.
✓No step in the resolution disabled certificate verification, and nothing in the incident record recommends doing so as a workaround.
✓If the cause was a trust store gap, the fix is recorded as a change to the client estate, not as a change to the server.
✓The verify lines from a fresh capture show the chain validating at every depth rather than stopping partway.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶This procedure is diagnostic and writes nothing to the endpoint, so rollback covers the diagnostic footprint and any experiment a responder ran.
↶Delete the capture files from any shared or world-readable location once they are attached to the incident record.
↶If a certificate was added to a client trust store to test a hypothesis, remove it unless the addition is the agreed remediation and has been reviewed. A test anchor left behind is an unowned trust decision.
↶If a hosts file entry or a resolver override was used to reach a specific backend, remove it, because a stale override outlives the incident and produces baffling failures weeks later.
↶If verification was disabled anywhere during the call despite the agreement not to, treat it as a security event: revert it, record it, and tell the service owner.
↶If the branch turned out to be wrong, do not unwind the diagnostic evidence. Return to the capture, take the next unresolved check in order, and record why the first conclusion was discarded.

6 · Escalation

When the runbook isn't enough, contact:

· The transcript contains no certificate at all and the port is answering: escalate to the network and load balancer owners, because something is terminating or intercepting the connection before the certificate is offered.
· The served certificate is valid and complete but a single client still fails: escalate to the owner of that client platform, since the trust store, the pinning configuration or the runtime version is the remaining variable.
· The certificate presented is not one the service owner recognises: escalate to security immediately and treat it as a possible interception or misissuance rather than a configuration fault.
· Two clients on the same host disagree about the same endpoint: escalate to the platform team, because one of them is using a bundle nobody is maintaining.
· The evidence points at the certificate authority rather than at the estate, such as an intermediate that no longer validates: escalate to the CA owner or the vendor before making any local change.

A TLS validation failure is a verdict delivered by a client, and the client is usually the only party that knows why. The server sees a handshake that stopped; the customer sees a red page; the incident channel sees six plausible theories arrive in the first two minutes. This procedure exists to replace the theories with a sequence of checks, each of which can only fail for one reason.

Six causes account for nearly all of it: the certificate is outside its validity window, the server sent an incomplete chain, the client does not trust the root, the name the client asked for is not in the certificate, the certificate is not permitted for server authentication, or the two sides never agreed on a protocol version. They produce different evidence. The discipline is to read the evidence in an order that eliminates causes rather than confirms favourites.

Capture once, then work offline. Every check after the first is run against a file, which means it is repeatable, attachable and free of the risk that the endpoint changes under you halfway through the call.

When this runbook applies, and when it does not

It applies when a client refuses a TLS connection on validation grounds, the TCP connection itself succeeds, and nobody yet knows which of the six causes is responsible.

It does not apply when:

The cause is already known. If the chain is demonstrably short, or the name is demonstrably absent, go straight to the specific procedure. This runbook is a discriminator, and running it on a settled diagnosis only delays the fix.
The connection never completes at the TCP layer. A timeout, a reset or a refused port is a reachability incident. No certificate has been exchanged, so no certificate evidence exists.
The failure is in mutual TLS and the rejected party is the client. The evidence lives in the server logs and in the client certificate, neither of which s_client shows you by default.
The failure is an SSH host key or SSH certificate rejection. SSH does not use X.509, has no chain and no trust store in this sense, and none of these commands apply to it.

Blast radius

Action	Reversible?	What it costs if wrong
One `s_client` capture per endpoint	Yes	A handshake in the server log and a few kilobytes on disk
Offline `openssl verify` runs against the capture	Yes	Nothing, which is precisely why the work belongs offline
Adding a root to a client trust store to test a theory	Only if you remove it	An unowned trust decision that survives the incident and widens what that host will accept
A hosts file or resolver override to reach one backend	Only if you remove it	Baffling failures weeks later on a host nobody associates with this incident
Disabling verification to confirm the fault is TLS	No	The one signal that separates six causes is destroyed, and a workaround that nobody removes becomes the permanent configuration

Step 1 - Capture the handshake once

Read-only / SafeOne capture, taken from the network path the failing client uses— Every later check runs against this file rather than against production.

$ # Substitute your own values before running:
HOST=app.lab.example
PORT=443

openssl s_client -connect "$HOST:$PORT" -servername "$HOST" -showcerts </dev/null > handshake.txt 2>&1
grep -E 'depth=|verify error|verify return code|Protocol|Cipher' handshake.txt

Keep handshake.txt whole. The verify lines at the top, the chain listing in the middle and the summary at the bottom each answer a different question, and discarding any of them means reconnecting.

Step 2 - Read the verify lines and the depth

The header of a failed capture states where validation stopped. This is what an incomplete chain looks like from the client side, taken from a server configured to send only its leaf:

Read-only / SafeCaptured on the course lab host against nginx serving the leaf alone

$ sed -n '1,12p' handshake.txt

depth=0 CN=app.lab.example
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=app.lab.example
verify error:num=21:unable to verify the first certificate
verify return:1
Certificate chain
0 s:CN=app.lab.example
 i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA

Illustrative output

Two facts are visible before any further command runs. Validation stopped at depth 0, the server certificate. The chain listing contains one entry whose issuer is not itself present, so the server offered nothing that could certify it. A healthy capture ends instead with Verify return code: 0 (ok) and lists every certificate up to, but not including, the root.

Step 3 - Split the capture and take the checks apart

ROOT=/etc/ssl/certs/ca-certificates.crt

# The captured PEM blocks, extracted from the transcript.
sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p' handshake.txt > served.pem

# The leaf on its own; served.pem stays as the untrusted pool.
openssl x509 -in served.pem -out leaf.pem

openssl x509 -in leaf.pem -noout -checkend 0
openssl verify -CAfile "$ROOT" -untrusted served.pem leaf.pem
openssl verify -CAfile "$ROOT" -untrusted served.pem -verify_hostname app.lab.example leaf.pem
openssl verify -CAfile "$ROOT" -untrusted served.pem -purpose sslserver leaf.pem

Run them in that order and stop at the first failure. Each invocation changes exactly one variable, so the first one that fails names the cause without ambiguity. Running them all and reading the last result is how two independent faults get reported as one.

Step 4 - Match the evidence to the cause

Evidence	Cause	Where the fix lives
`Certificate will expire` from `-checkend 0`	The certificate is outside its validity window	Renewal and deployment
`error 20 at 0 depth lookup: unable to get local issuer certificate`	The issuer of the leaf is not available to the client	The chain the server sends
`verify error:num=21:unable to verify the first certificate` in the capture	The same fault seen from the handshake rather than from a file	The chain the server sends
`error 2 at 1 depth lookup: unable to get issuer certificate`	The root is not in this client’s trust store	The client estate, not the server
`error 62 at 0 depth lookup: hostname mismatch`	The requested name is not in the subject alternative name	The certificate, or the name the client asks for
`error 26 at 0 depth lookup: unsuitable certificate purpose`	The certificate is not valid for server authentication	Issuance policy and the extended key usage
No certificate in the capture at all	The connection failed before certificates were exchanged	Protocol, cipher or interception

flowchart TD
    A["Client refuses the connection"] --> B{"Certificate present in the capture?"}
    B -- "no" --> P["Protocol, cipher or interception"]
    B -- "yes" --> C{"checkend 0 reports it will expire?"}
    C -- "yes" --> E["Expiry"]
    C -- "no" --> D{"verify with -untrusted passes?"}
    D -- "error 20 or 21" --> F["Server sent an incomplete chain"]
    D -- "error 2" --> G["Root unknown to this client"]
    D -- "passes" --> H{"-verify_hostname passes?"}
    H -- "error 62" --> I["Name not in the SAN"]
    H -- "passes" --> J{"-purpose sslserver passes?"}
    J -- "error 26" --> K["Wrong certificate purpose"]
    J -- "passes" --> L["Not a certificate fault"]

The diagram is the same sequence as the commands: presence, then validity window, then chain, then anchor, then name, then purpose. Each branch leaves the tree, so the procedure cannot return two answers. If every check passes and the client still fails, the fault is not in the certificate and the investigation moves to the client runtime or to whatever sits between the two endpoints.

Step 5 - Separate a client refusal from a server policy

When no certificate appears in the capture, the tempting conclusion is that the server rejects old protocol versions. Confirm which side actually refused, because a modern OpenSSL build declines to offer obsolete versions at its own default security level:

Read-only / SafeA client-side refusal, produced before any packet reached the server

$ openssl s_client -tls1_1 -connect app.lab.example:443 </dev/null

error:0A0000BF:SSL routines:tls_setup_handshake:no protocols available

Illustrative output

That message is your own library declining to build the ClientHello. It is not evidence of what the server would have done. To learn the server’s policy you need a client that is willing to offer the version in question, or the server’s own configuration.

Common pitfalls

Symptom	Cause	Action
Browser works, application fails	Different trust stores, and some clients repair a missing intermediate on their own by fetching the issuer named in the certificate	Test with the client that defines the incident and treat the browser as anecdote
The failure moves when you retry	Several backends behind one address hold different certificates	Capture from each backend by address, with the correct server name, and compare serials
Verify passes locally but the service still fails	Your trust store is not the client’s trust store	Re-run the verify with `-CAfile` pointed at the bundle that client actually loads
The error mentions a name nobody configured	The connection was answered by a default virtual host because no server name was sent	Re-capture with `-servername` and compare
Everything passes but the handshake still fails	The fault is below the certificate: protocol version, cipher selection, or an interception device	Read the protocol and cipher lines in the capture and involve the network owners

Verification

The investigation is complete when the incident record names exactly one cause and quotes the verify result that supports it. The proof is not a repeat of your own capture, which will happily reproduce your own mistake: it is the client that defined the incident completing the handshake with verification fully enabled, plus a second client on a different host and a different trust store doing the same. Attach the capture and the exact verify command lines so the conclusion can be reproduced offline. If the cause turned out to be a trust store gap, the record should show the change landing in the client estate rather than on the server, because a server change would have fixed nothing.

Rollback

Nothing here modifies the endpoint, so the rollback is about the footprint of the investigation. Remove capture files from shared or world-readable locations once they are attached to the record. Remove any root you added to a trust store to test a theory, unless that addition is the reviewed remediation. Remove hosts file entries and resolver overrides used to reach a particular backend, because those outlive the incident and produce failures nobody can explain later. If anyone disabled verification during the call, revert it and record it as a security event rather than as a diagnostic step. If the branch you took proves wrong, keep the evidence, return to the captured file, and work the next unresolved check in order.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

When this runbook applies, and when it does not

Blast radius

Step 1 - Capture the handshake once

Step 2 - Read the verify lines and the depth

Step 3 - Split the capture and take the checks apart

Step 4 - Match the evidence to the cause

Step 5 - Separate a client refusal from a server policy

Common pitfalls

Verification

Rollback

References