← All runbooks in Secrets, PKI & Certificates

medium riskservice affecting~60 min

Runbook: Investigate a Mutual TLS Failure

1 · Prerequisites

Confirm every item is in place before any state change.

secrets-pki-rb-02-troubleshoot-tls-validation-failure
secrets-pki-lab-12-configure-and-verify-mutual-tls
secrets-pki-cl-03-tls-service-readiness
Read access to the server error log, because the verdict on a client certificate is recorded there and nowhere else.
A copy of the client certificate, its chain and the trust anchor each side is configured with, gathered onto one host where they can be compared offline.
The ability to run a handshake from the same network position as the failing client, since a proxy or service mesh between them changes what is actually presented.
Agreement with the service owner that this is an investigation and that no configuration will be changed without a separate change record.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the connection is genuinely mutual. A one-way TLS failure investigated as a mutual one wastes the first hour. Establish whether the server asks for a client certificate at all: on nginx the request is controlled by ssl_verify_client, whose documented default is off, so a server nobody configured never asks and never rejects.
· Check the clock on both endpoints before anything else. Skew produces validity failures from certificates that are entirely correct, on both sides at once, and it is the one cause that makes every other reading misleading. Compare the two clocks against the same reference, not against each other.
· Establish when it last worked and what changed. A certificate renewal, a trust bundle rebuild, a container image update, a mesh upgrade or a proxy insertion are the usual candidates. The change window is a stronger signal than any single command output.
· Confirm you are testing from the same network position as the failing client. A terminating proxy, an ingress controller or a service mesh sidecar presents its own certificate and verifies with its own bundle, so a test from the wrong hop measures a different connection entirely.
· Gather the five artefacts before diagnosing. The client certificate and its chain, the server certificate and its chain, the anchor the client is configured with, the anchor the server is configured with, and both private keys as files you can fingerprint without reading. Everything below is a comparison between two of these.
· Confirm the failure is reproducible. An intermittent mutual TLS failure across a pool usually means one member differs, so capture which backend served the failing attempt before treating the fault as universal.
· Agree that nothing will be turned off to make it work. Write it down before you start. Disabling verification converts an authentication failure into a silent security hole and destroys the evidence you are collecting.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Determine which direction failed. Open a handshake without offering a client certificate. If it fails before any client certificate could matter, the fault is server-to-client and the client certificate is irrelevant. If it reaches the point of asking for one, the fault is client-to-server. This single observation eliminates roughly half the search space.
2Read the server chain as the client sees it. openssl s_client -connect app.lab.example:443 -servername app.lab.example -showcerts prints every certificate the server sent and the verification outcome at each depth. An incomplete chain and an untrusted anchor produce different errors, and the difference is the diagnosis.
3Separate an incomplete chain from a missing anchor. A server that sends only the leaf produces verify error:num=20:unable to get local issuer certificate followed by verify error:num=21:unable to verify the first certificate. A complete chain that terminates at an anchor the client does not hold produces the same error 20 with no error 21. Fix the server in the first case and the client store in the second.
4Rule out a name problem, which is not a trust problem. If curl reports no alternative certificate subject name matches target hostname, the chain is fine and the name is wrong. Since RFC 9525 the common name must not be used to identify a service, so a certificate whose only correct name is in the subject rather than the SAN will fail on a current client and pass on an old one.
5Verify the client certificate offline against the anchor the server is configured with. openssl verify -CAfile server-clients-ca.crt -untrusted client-chain.pem client.crt answers the question the server is actually asking. Run it with the server side artefacts, not the client side ones; using the wrong anchor here is the commonest way to get a confident wrong answer.
6Assert the purpose, in both roles, on the certificate that is failing. openssl verify -CAfile root.crt -untrusted srv-ca.crt -purpose sslserver app.crt and the same command with -purpose sslclient. A certificate that passes as a server and fails as a client returns error 26 at 0 depth lookup: unsuitable certificate purpose, and that is a definitive answer rather than a hint.
7Read the extended key usage directly when the purpose check fails. openssl x509 -in client.crt -noout -ext extendedKeyUsage,keyUsage. A certificate carrying only TLS Web Server Authentication cannot be used as a client certificate, however valid it is, and no amount of trust store work will change that.
8Prove that the private key each side loaded matches the certificate it is presenting. Compare openssl pkey -in app.key -pubout | openssl sha256 with openssl x509 -in app.crt -noout -pubkey | openssl sha256. Identical digests confirm correspondence. A mismatch explains a handshake that fails after the certificate is accepted rather than during validation.
9Confirm the server trust store contains the issuer of the client certificate. List the certificates in the bundle the server verifies clients against with openssl storeutl -noout -certs clients-ca.pem and compare the subject DNs with the issuer of the client certificate. A bundle assembled from the wrong intermediate is invisible in every client side test.
10Check whether a client certificate was requested and whether one was sent. If the server never asked, the client will not send one and the failure is a server configuration matter. If the server asked and the client sent nothing, the client either has no certificate configured for that destination or declined to offer the one it has.
11Read the server log for the verification verdict. The client sees a generic handshake alert for almost every client certificate rejection, so the reason exists only on the server side. Correlate by timestamp and source address with the attempt you just made, not with an older entry.
12Consider revocation only after trust and purpose are eliminated. OpenSSL does not check revocation unless asked: the verification flags are documented as off by default and -crl_check is opt-in. A certificate that verifies cleanly on your workstation can still be refused by a server that has revocation checking enabled against a local CRL.
13Record the finding as one of the five causes, with the command output that proves it. Client certificate, server certificate, client trust store, server trust store, or purpose. Naming the cause precisely is what stops the same investigation being run again next month.

4 · Verification

Confirm the procedure actually fixed the problem.

✓A handshake from the same network position as the failing client now completes with a verify return code of 0 and the application response is served.
✓The client certificate passes openssl verify against the anchor the server is configured with, and the exit status is 0 rather than merely the absence of visible error text.
✓The certificate in the failing role passes the corresponding purpose check, and the previously failing role no longer returns error 26.
✓The server log records a successful client certificate verification for the new attempt, matched by timestamp and source address.
✓The public key digest of each deployed certificate matches the digest of the private key the serving process loaded.
✓The finding is reproducible in the opposite direction: reintroducing the identified cause on a test endpoint reproduces the original error exactly.
✓No verification setting was weakened anywhere during the investigation, confirmed by diffing the configuration against the version in source control.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶This runbook is read-only by design, so in the normal case there is nothing to roll back and that is the point of running it this way.
↶If a certificate or bundle was copied to a host for offline comparison, remove the copies afterwards. Private key material gathered for an investigation must not be left on a shared host.
↶If a test endpoint was created to reproduce the fault, remove it and confirm it is no longer listening, so a permissive test configuration does not outlive the investigation.
↶If verification was weakened at any point despite the pre-check, restore it immediately from source control and treat the interval as a security event to be reported, not as a detail of the investigation.
↶If a trust bundle was edited to test a hypothesis, restore the original file and reload the process, then re-verify from off-host that the served behaviour is unchanged.
↶If the investigation moved traffic away from a backend, return it to the pool only after that backend passes the same checks as its peers.

6 · Escalation

When the runbook isn't enough, contact:

· The client certificate is valid, correctly purposed and correctly anchored, and the server still rejects it: escalate to the owner of the server trust configuration. The remaining explanations are on that side and are not visible from the client.
· The failure occurs only through a proxy, ingress controller or service mesh: escalate to the platform team that owns that hop. It terminates and re-originates TLS, so both certificates and both trust stores in the second segment are theirs.
· A certificate must be reissued to correct its extended key usage: escalate to the CA operator, because this is an issuance policy question rather than a deployment one, and the same defect is likely to be present in the whole batch.
· Evidence appears that a private key has been copied off its host during the investigation: stop and escalate to the security owner. That is a key compromise, and it outranks the connectivity problem.
· The failure is intermittent and correlates with a subset of backends that cannot be identified from the client: escalate to the load balancer owner for per-connection backend attribution.

Mutual TLS is two independent authentications carried inside one handshake, and they fail for entirely separate reasons. The server proves possession of the key in its certificate; the client, if asked, proves possession of the key in its own. Either proof can fail, and the failure can be about the certificate, about the store that was supposed to trust it, or about whether the certificate was ever permitted to play that role at all.

That gives five distinct causes: the client certificate, the server certificate, the client trust store, the server trust store, and a purpose problem where a certificate is technically valid but not authorised for the role it is being used in. Under pressure they get conflated, because the client-visible symptom is nearly the same for several of them and completely uninformative for the rest.

The most important asymmetry is where the verdict is recorded. When the client rejects the server, the client tells you why in detail. When the server rejects the client, the client is told almost nothing and the reason exists only in the server log. An investigation that never reads the server log can eliminate at most three of the five causes.

When this runbook applies, and when it does not

It applies when a connection that requires a client certificate is failing, both endpoints are reachable at the network layer, and you can observe the handshake from the same position as the failing client.

It does not apply when:

The server does not request a client certificate. Then this is not a mutual TLS failure at all. Establish that first, because everything below assumes a certificate was asked for.
Only the server certificate is in question. A plain TLS validation failure has its own, shorter procedure and does not need the client side artefacts.
The trust relationship is SSH. An SSH certificate carries one CA signature and no chain, has no extended key usage, and is revoked by a local key revocation list rather than by anything in this runbook.
The endpoints cannot reach each other. A connection refused or a timeout is a network fault wearing a TLS costume, and the certificates are innocent.

Blast radius

Action	Reversible?	What it costs if wrong
Run a handshake from a client host	Yes	Nothing; one extra connection in the log
Verify certificates offline with `openssl verify`	Yes	Nothing; no service is touched
Copy certificates and keys to one host for comparison	Only if you remove them	Key material left on a shared host is a compromise
Edit a trust bundle to test a hypothesis	Only if you kept a copy	The service may accept or reject the wrong peers until it is restored
Weaken or disable peer verification	No, in the sense that matters	Authentication is silently gone and the evidence is destroyed

The discrimination, as a decision tree

flowchart TD
    A["Handshake fails"] --> B{"Does it fail before a client certificate is requested?"}
    B -- "yes" --> C{"Did the server send its full chain?"}
    C -- "no" --> C1["Server certificate: incomplete chain"]
    C -- "yes" --> C2{"Does the client hold the anchor?"}
    C2 -- "no" --> C3["Client trust store: missing anchor"]
    C2 -- "yes" --> C4["Server certificate: name or validity"]
    B -- "no" --> D{"Does the client certificate verify against the server anchor?"}
    D -- "no" --> D1["Server trust store, or client chain incomplete"]
    D -- "yes" --> E{"Does it pass -purpose sslclient?"}
    E -- "no" --> E1["Purpose: wrong extended key usage"]
    E -- "yes" --> E2["Read the server log: key mismatch or revocation"]

Every branch above corresponds to a command in the steps that follow. The value of working it in this order is that each answer removes a whole side of the problem rather than narrowing it slightly, and the two most expensive mistakes (testing the client certificate against the client anchor, and concluding “trust problem” when the real answer is purpose) are both structurally excluded.

Step 1 - Read what the server presents, before any client certificate matters

Read-only / Safefrom the same host and network position as the failing client— Shows every certificate the server sent and the verification outcome at each depth.

$ HOST=app.lab.example
PORT=443

openssl s_client -connect "$HOST:$PORT" -servername "$HOST" -showcerts </dev/null

depth=0 CN=app.lab.example
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=app.lab.example
verify error:num=21:unable to verify the first certificate
verify return:1
Certificate chain
0 s:CN=app.lab.example
 i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA

Illustrative output

Read the chain block, not just the errors. One entry means the server sent the leaf alone, and error 21 confirms it: nothing could be verified because the issuer was never presented. That is a server-side defect and the client store is innocent. When the chain block shows leaf and intermediate but error 20 still appears without error 21, the chain is complete and the client simply does not hold the anchor it terminates at.

Step 2 - Separate trust from naming

A name failure is not a trust failure and does not respond to any trust store change. From a client, the two read completely differently:

curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)
curl: (60) SSL: no alternative certificate subject name matches target hostname 'wrong.lab.example'

The first is a path problem. The second is an identity problem: the chain built correctly and the name presented did not match what was asked for. Since RFC 9525 a client must not fall back to the common name, so a certificate that names the service only in its subject will fail here even though older clients accepted it.

Step 3 - Verify the client certificate against the anchor the server uses

Read-only / Safeoffline, using the SERVER side anchor and the client certificate— Answers the question the server is actually asking, which is not the one the client asks.

$ openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt
openssl verify -CAfile srv-ca.crt app.crt

app.crt: OK
error 2 at 1 depth lookup: unable to get issuer certificate

Illustrative output

The two calls differ only in what is treated as the anchor. Supplying the intermediate as the anchor produces error 2 at 1 depth lookup: unable to get issuer certificate, which is what a server trust bundle built from the wrong certificate will do to every client at once. Check the exit status as well as the text: openssl verify exits 0 on success and 2 on failure, and a wrapper script that ignores the status will report a broken chain as fine.

Step 4 - Assert the purpose in both roles

Read-only / Safethe check that separates a trust problem from an authorisation problem— A certificate can be perfectly valid and still not be permitted to act as a client.

$ openssl verify -CAfile root.crt -untrusted srv-ca.crt -purpose sslserver app.crt
openssl verify -CAfile root.crt -untrusted srv-ca.crt -purpose sslclient app.crt

app.crt: OK
error 26 at 0 depth lookup: unsuitable certificate purpose

Illustrative output

This is the decisive check in most mutual TLS investigations. The certificate above verifies cleanly, chains correctly and is in date. It is simply not a client certificate: its extended key usage carries TLS Web Server Authentication and nothing else, so every server that verifies purpose will refuse it while every trust store check you run will pass. Confirm by reading the extension directly with openssl x509 -in app.crt -noout -ext extendedKeyUsage,keyUsage.

Step 5 - Confirm the key each side loaded matches the certificate it presents

Read-only / Safeon each endpoint, against its own key and certificate— Identical digests prove correspondence without exposing the private key.

$ openssl pkey -in app.key -pubout | openssl sha256
openssl x509 -in app.crt -noout -pubkey | openssl sha256

SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b
SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b

Illustrative output

A mismatch here explains a handshake that fails after the certificate has been accepted, because the proof of possession cannot be produced. It is the usual result of a renewal that replaced the certificate and left the previous key, or of two files that were copied from different generations of the same service.

Step 6 - Read the server trust store and the server log

The last two causes are both invisible from the client. List the certificates in the bundle the server verifies clients against with openssl storeutl -noout -certs clients-ca.pem and compare the subject DNs against the issuer of the client certificate. Then read the server log for the verdict on the attempt you just made, correlated by timestamp and source address.

Note one default that changes the meaning of a clean local verify: OpenSSL does not check revocation unless it is asked to, and -crl_check is opt-in. A certificate that verifies on your workstation may still be refused by a server that enforces a local CRL.

Common pitfalls

Symptom	Cause	Action
Client certificate verifies locally but the server rejects it	It was verified against the client anchor, not the server anchor	Repeat the verify with the server side bundle
Everything is in date and trusted, and it still fails	Extended key usage does not permit the role	Run both purpose checks; reissue if error 26 appears
Works from a workstation, fails from the application	The application uses a different trust store from the operating system bundle	Find the store that runtime actually reads
Fails only through the ingress or mesh	TLS is terminated and re-originated at that hop	Investigate the second segment separately; both sides of it are different certificates
Intermittent across a pool	One backend has a different certificate or bundle	Capture backend attribution before treating the fault as universal
Server log shows nothing at all	The server never requested a certificate	Check the verification directive; the default on nginx is off

Verification

The investigation is complete when a handshake run from the failing client’s own position completes with a verify return code of 0 and the application response is returned, and when the identified cause has been demonstrated in both directions: the client certificate passes openssl verify against the server side anchor with exit status 0, the previously failing purpose check now returns OK, and the server log records a successful client certificate verification correlated by timestamp and source address with that attempt. Each deployed certificate must still match the private key its process loaded, proved by public key digest. The strongest confirmation is reproducing the fault deliberately on a test endpoint and seeing the original error text appear again. Finally, diff the configuration of both endpoints against source control to prove that no verification setting was weakened while the investigation ran.

Rollback

There is nothing to roll back from a correctly run investigation, because every step above is a read. The rollback obligations are about what the investigation leaves behind. Remove any certificate or key copied to a host for offline comparison, because key material gathered for an investigation must not outlive it on a shared machine. Remove any test endpoint created to reproduce the fault and confirm it is no longer listening. If a trust bundle was edited to test a hypothesis, restore the original and reload, then re-verify from off-host that the served behaviour is unchanged. If verification was weakened at any point, restore it immediately from source control and report the interval as a security event rather than recording it as an investigation detail.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

When this runbook applies, and when it does not

Blast radius

The discrimination, as a decision tree

Step 1 - Read what the server presents, before any client certificate matters

Step 2 - Separate trust from naming

Step 3 - Verify the client certificate against the anchor the server uses

Step 4 - Assert the purpose in both roles

Step 5 - Confirm the key each side loaded matches the certificate it presents

Step 6 - Read the server trust store and the server log

Common pitfalls

Verification

Rollback

References