TLS troubleshooting - diagnosing failures from the wire

Almost every TLS investigation that goes badly starts the same way: somebody checks the certificate file on the server, sees a notAfter a long way in the future, and concludes the certificate is not the problem.

The file on disk is not evidence about what the service serves. Between the two sit a reload that did not happen, a container with a stale bind mount, a reverse proxy terminating TLS with its own certificate, a second vhost that answers when SNI is absent, and a load balancer with an older copy. Start from the wire.

Step 1: what is actually served

$ echo | openssl s_client -connect example.com:443 -servername example.com 2>&1 | head -25

depth=3 C=US, O=SSL Corporation, CN=SSL.com TLS ECC Root CA 2022
verify return:1
depth=0 CN=example.com
verify return:1
CONNECTED(00000003)
---
Certificate chain
0 s:CN=example.com
 i:C=US, O=SSL Corporation, CN=Cloudflare TLS Issuing ECC CA 3
 a:PKEY: EC, (prime256v1); sigalg: ecdsa-with-SHA256
 v:NotBefore: Jul 29 22:10:08 2026 GMT; NotAfter: Oct 27 22:17:21 2026 GMT
1 s:C=US, O=SSL Corporation, CN=Cloudflare TLS Issuing ECC CA 3
 i:C=US, O=SSL Corporation, CN=SSL.com TLS Transit ECC CA R2
---
Server certificate

Three facts come out of that block immediately:

• The chain is ordered and linked. Entry 0 is the leaf. Each i: must equal the next entry’s s:. A break in that sequence is a chain problem, not a trust problem.
• The dates are the served dates, not the dates of a file you hope is in use.
• verify return:1 at every depth means this client, with this trust store, was satisfied. A different client with a different trust store may not be.

The compact form for scripting and for a first look:

HOST=service.example.com
echo | openssl s_client -connect "$HOST:443" -servername "$HOST" -brief 2>&1

# Just the identity and validity of what is served
echo | openssl s_client -connect "$HOST:443" -servername "$HOST" 2>/dev/null |
    openssl x509 -noout -subject -dates -ext subjectAltName

Step 2: chain or trust?

Two verify error codes cover most chain complaints and they mean different things:

Codes 20 and 21 are server-side: the server did not send enough. Codes 18 and 19 are client-side: the client does not trust the anchor. Getting this distinction right is most of the diagnosis, because the two are fixed on different machines by different people.

Count what the server sends:

HOST=service.example.com
echo | openssl s_client -connect "$HOST:443" -servername "$HOST" -showcerts 2>/dev/null |
    grep -c 'BEGIN CERTIFICATE'

A public certificate normally gives 2 or 3. A result of 1 with a public CA means the intermediate is missing, and the server config is concatenating only the leaf.

The bundle a server sends should be leaf first, then each intermediate, and normally not the root - the client already has the root or does not trust it, so sending it adds bytes and no information.

Step 3: name matching

A chain can verify perfectly and still be the wrong certificate for this name. The OpenSSL lesson in this part established that clients match on subjectAltName and that openssl verify will pass a CN-only certificate that Go and Chrome reject. Ask explicitly:

HOST=service.example.com
echo | openssl s_client -connect "$HOST:443" -servername "$HOST" \
    -verify_hostname "$HOST" -verify_return_error 2>&1 | head -5

# What names it actually covers
echo | openssl s_client -connect "$HOST:443" -servername "$HOST" 2>/dev/null |
    openssl x509 -noout -ext subjectAltName

-verify_return_error makes s_client close the connection on a verification failure instead of continuing, which is what turns it into a usable test rather than a report.

Step 4: SNI

If the server hosts several names on one address, the certificate it returns depends on the SNI value in the ClientHello. Omit it and you get the default vhost.

HOST=service.example.com
# With SNI - what a normal client sees
echo | openssl s_client -connect "$HOST:443" -servername "$HOST" 2>/dev/null |
    openssl x509 -noout -subject

# Without SNI - what an old client or a bad health check sees
echo | openssl s_client -connect "$HOST:443" -noservername 2>/dev/null |
    openssl x509 -noout -subject

Different subjects between those two is the explanation for a whole family of symptoms: a monitoring probe that alerts while users are fine, a client library that does not send SNI, a health check configured against an IP address rather than a name.

Step 5: dates and the clock

Expiry is obvious. Two variants are not:

• Not yet valid. A certificate issued by a CA whose clock is ahead has a notBefore in the future, and clients reject it with a message that sounds like expiry to a tired reader.
• The client clock is wrong. A host with no working NTP rejects perfectly valid certificates, and the error is on the certificate rather than on the clock.

HOST=service.example.com
echo | openssl s_client -connect "$HOST:443" -servername "$HOST" 2>/dev/null |
    openssl x509 -noout -dates
date -u
timedatectl status | grep -E 'System clock|NTP service'

# Will it still be valid in 30 days? Exit status answers.
echo | openssl s_client -connect "$HOST:443" -servername "$HOST" 2>/dev/null |
    openssl x509 -noout -checkend 2592000

The time part of this course covers why a host drifts. The point here is to check the clock on both ends before concluding anything about a certificate.

Step 6: protocol and cipher

140234...:error:0A000102:SSL routines::unsupported protocol

or a handshake that reports Cipher is (NONE). The server and client have no protocol or cipher suite in common, usually because the server disabled TLS 1.0 and 1.1 and something old still speaks only those.

HOST=service.example.com
for proto in -tls1_2 -tls1_3; do
    printf '%s: ' "$proto"
    echo | openssl s_client -connect "$HOST:443" -servername "$HOST" \
        "$proto" -brief 2>&1 | grep -m1 -E 'Protocol|error' || echo unsupported
done

This is a compatibility decision rather than a fault. Find the client, upgrade it, and do not re-enable a deprecated protocol fleet-wide to accommodate one caller.

Step 7: client certificates

When mutual TLS is in play, the server tells you what it will accept:

HOST=service.example.com
echo | openssl s_client -connect "$HOST:443" -servername "$HOST" 2>&1 |
    sed -n '/Acceptable client certificate CA names/,/^---/p'

# Then present one
echo | openssl s_client -connect "$HOST:443" -servername "$HOST" \
    -cert client.crt -key client.key -verify_return_error 2>&1 | head -5

An empty acceptable-CA list with a server that still rejects you usually means the server is configured to request client certificates but its CA file is missing or unreadable.

Verifying a chain before you deploy it

Every check above is post-deployment. The same questions can be answered offline, which is where they belong:

# Does the bundle chain to the CA, and does the leaf cover the name
openssl verify -CAfile ca-root.pem -untrusted intermediates.pem \
    -verify_hostname service.example.com -purpose sslserver \
    -show_chain service.crt

# Does the key match the certificate
openssl x509 -noout -pubkey -in service.crt | openssl sha256
openssl pkey -noout -pubout -in service.key | openssl sha256

-show_chain prints the chain it built, which tells you whether the intermediate you supplied was used or whether it found something else in the trust store. -untrusted is the flag that lets you test a bundle you have not installed anywhere yet.

The ordered diagnostic

Knowledge check