Diagnosing works on my machine but fails in the cluster

Somebody says the endpoint is fine because it opens on their laptop, and somebody else says the deployment is broken because the pod cannot reach it. Both are correct observations of different systems. Treating this as a disagreement wastes the first hour; treating it as a controlled comparison with a small number of variables usually resolves it in ten minutes.

Move the probe to the failure

The reflex is to reproduce the failure locally. That is the wrong direction. The local machine is the one environment already proven to work, so any experiment run there starts by removing the only interesting variable.

Run the diagnostic where the failure lives, inside the same image, the same namespace and the same network path as the failing process. If the image carries a shell, start a container from that exact tag and run the probe. If it does not, because it is distroless or built from scratch, attach an ephemeral debug container that shares the failing pod’s network namespace, so name resolution and egress remain identical rather than merely similar.

IMAGE=registry.example.com/app:2026-08-26
HOST=api.internal.example.com

# The probe belongs in the failing environment, not on the laptop.
docker run --rm -i "$IMAGE" \
  openssl s_client -connect "$HOST:443" -servername "$HOST" -showcerts </dev/null

Note the explicit -servername. Modern s_client derives the server name indication from the argument to -connect, which means a probe aimed at an IP address sends no name indication at all and may be answered by a completely different virtual host from the one your application reaches. Any comparison that skips this is comparing two different requests.

The question that splits the problem in half

There is exactly one measurement that eliminates half the hypotheses at once, and it should always be the first: what did the server transmit?

The certificate list a server sends is a property of the connection and of whatever terminated it. It is not affected by the client’s trust store, its runtime, or its environment variables. It is also printed even when verification fails, which is what makes it usable during an outage rather than after one.

• The two clients see the same transmitted chain. The server is behaving consistently, so the difference lives in the client: its anchor set, its runtime’s store, an environment override or its clock.
• The two clients see different transmitted chains. The server is not one server. Something between them differs: a load balancer member, a terminating proxy, split-horizon name resolution, or a service mesh sidecar that only one side traverses.

Comparing by common name is not enough, because the interesting cases involve two certificates carrying the identical name. Compare identity: the serial number and the SHA-256 fingerprint.

HOST=api.internal.example.com

openssl s_client -connect "$HOST:443" -servername "$HOST" </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -serial -fingerprint -sha256

A real capture of that shape looks like the block below, and a serial of 21173B360D80F4A69A91164F1067F4F81A1B1B6E on one side and anything else on the other ends the discussion immediately.

subject=CN=app.lab.example
issuer=O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
serial=21173B360D80F4A69A91164F1067F4F81A1B1B6E

flowchart TD
    A["One client works, one fails"] --> B["Run the same probe inside the failing context"]
    B --> C{"Is the transmitted chain identical?"}
    C -- "no" --> D["Server or path: member, proxy, DNS or mesh"]
    C -- "yes" --> E{"Does an explicit CAfile make it pass?"}
    E -- "yes" --> F["Trust store or runtime on the failing client"]
    E -- "no" --> G["Clock, name, or local protocol policy"]

The second decision in the diagram is the offline verification from earlier in this part. Capture the transmitted certificates, hand them to openssl verify with the candidate anchor named explicitly, and see whether the material validates when the store is taken out of the question. A pass there localises the fault to the store the failing runtime consults, which the previous lesson tells you how to find.

The variables that actually differ

Once the split is made, work a fixed list rather than improvising. These are the differences that produce this symptom, in the order they are worth checking.

Environment overrides deserve their place high on that list because they are invisible to every filesystem inspection. A value inherited from a base image, a systemd drop-in or a pod specification silently outranks the store you have been examining, and the first sign of it is usually a host that trusts something nobody installed.

Errors that describe your client, not the server

Some failures name a remote condition and describe a local one. This capture came from an attempt to negotiate an obsolete protocol version, and it is not evidence about the server at all:

error:0A0000BF:SSL routines:tls_setup_handshake:no protocols available

The connection never reached the server. A modern OpenSSL build declined to offer that version at its configured security level, so the refusal happened inside the client process. Quoting it in an incident channel as proof that the server rejects old protocols sends the whole investigation to the wrong machine.

The same care applies to matching on error text. Two builds of the same tool phrase the same verdict differently:

curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)
curl: (60) SSL certificate problem: unable to get local issuer certificate

Those two lines were produced by the same underlying condition, one on a host and one inside a container image with an older build. Anything that alerts on an exact string will catch one and miss the other. Match on the numeric verify result where it is available, and on the concept otherwise.

Production discipline

1. Never accept a laptop as the reference implementation. Managed endpoints carry extra anchors and often sit behind interception, so they succeed for reasons your servers do not share.
2. Capture both sides before changing either. The transmitted chain, the resolved address and the clock, recorded from each vantage point, are what make the conclusion defensible afterwards.
3. Record the comparison as the incident artefact. A filled-in table of the seven variables is a better handover than a paragraph of narrative, and the next occurrence is diagnosed from it in minutes.
4. Fix the cause you proved, not the one you suspect. Adding an anchor because the message mentioned an issuer is how fleets acquire trust they cannot account for.

Cross-course references

• Kubernetes for Production Sysadmins - Part CXXII (DNSTroubleshoot) covers resolution differing between a workload and the outside world, which is one of the variables that produces two different endpoints behind one name.
• Linux for Production Sysadmins - Part XXIV (Time) covers clock synchronisation, the variable that turns a perfectly valid certificate into an expired or not-yet-valid one on a single host.
• Observability for Production Sysadmins - Part CIX (InvestigationWorkflows) covers structuring a comparison so that the evidence, not the loudest hypothesis, chooses the next step.

Quiz