Reading openssl s_client: the five flags that decide what you learn

openssl s_client is a complete TLS client that prints its own verification decisions as it makes them. That narration is the reason it is the right instrument for an incident: unlike a browser or an application, it shows the intermediate steps and not merely the verdict. It is also why it is so easy to misread, because by default it reports problems and then carries on.

A client that narrates, not a certificate viewer

Two habits get conflated. Inspecting a certificate file on disk is openssl x509 work. Asking what a running service presents to a network peer, and what a client makes of it, is s_client work. The second question is the one that matters during an outage, because the file on disk and the bytes on the wire disagree more often than anyone expects.

The tool opens a real connection, completes a real handshake, and then hands you an interactive session on the socket. That last part is a trap in scripts and pipelines: without redirected input it waits for you to type. Terminate stdin explicitly.

HOST=api.example.com
PORT=443

openssl s_client -connect "$HOST:$PORT" -servername "$HOST" \
  -showcerts </dev/null

The five flags, and the question each one answers

• -connect address:port decides which socket is opened. It accepts an address as readily as a name, which is how you interrogate one member of a load-balanced pool instead of whichever member the resolver happens to return.
• -servername name sets the Server Name Indication extension. It travels in the ClientHello, before any encryption exists, and is how a server with many virtual hosts chooses which certificate to send. Connect by address without it and the server has nothing to select on, so it falls back to a default and you inspect a certificate nobody was complaining about.
• -showcerts prints the full PEM of every certificate the server transmitted, in the order it transmitted them. This is the flag that turns a vague chain complaint into a countable fact.
• -CAfile path replaces the host default trust anchors with exactly the anchors in that file. It converts the question from “does this host trust the service” into “does this specific anchor set trust the service”, which is the question an application with its own bundle actually asks.
• -verify_return_error makes a verification failure end the handshake instead of being noted in passing. Use it when you want the exit status to mean something.

flowchart TD
    A["-connect address:port"] --> B["TCP socket opens"]
    B --> C["-servername selects the virtual host"]
    C --> D["server transmits its certificate list"]
    D --> E["-showcerts prints every PEM received"]
    D --> F["path building uses -CAfile or the host default"]
    F --> G["verify error lines, one per problem found"]
    G --> H["-verify_return_error ends the handshake here"]
    G --> I["otherwise: Verify return code on the summary"]

Each flag intervenes at exactly one point in that sequence, and that is how to choose them. If the question is about which certificate is served, -servername is the flag that matters. If the question is about what the server failed to send, -showcerts is. If the question is about whose trust store is wrong, -CAfile is.

Reading the trace line by line

Verification produces one group of lines per certificate the client examines. Here is real output from a server that sent only its leaf:

depth=0 CN=app.lab.example
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=app.lab.example
verify error:num=21:unable to verify the first certificate
verify return:1

Three distinct pieces of information are packed into that. depth=0 names the position in the chain being examined, where zero is the leaf and the number increases towards the trust anchor; the depth tells you where the chain broke, which is often more useful than the message. num=20 is the numeric verification error, and unlike the human text it is stable across builds and locales. verify return:1 is the verification callback electing to continue in spite of the problem, which is why the transcript does not stop there.

Error 21 appearing after error 20 is not a second independent fault. It is the summary consequence of the first: no issuer could be located for the leaf, therefore the first certificate could not be verified at all.

A successful connection ends with a summary block instead:

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
Verify return code: 0 (ok)

Verify return code: 0 (ok) is the authoritative verdict, and it is the line to quote in a ticket. Note what the cipher suite name does and does not say. In TLS 1.3 a suite name specifies only the AEAD algorithm and the hash, here AES-256-GCM with SHA-384. Key exchange and authentication are negotiated separately, so you cannot read the certificate’s key type out of that line the way TLS 1.2 suite names allowed.

What the chain block proves, and what it does not

With -showcerts the transcript contains a chain summary in which s: is a subject and i: is that certificate’s issuer. This is the server’s transmission, not the client’s conclusion.

Certificate chain
 0 s:CN=app.lab.example
   i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
 1 s:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
   i:O=RunBook Academy Lab, CN=RunBook Lab Root CA

Read it as a linked list. Entry 0 is issued by the subject of entry 1, and entry 1 is issued by a root that the server did not send, which is correct: an anchor must come from the client’s own store, not from the party being authenticated. If entry 1 is absent while entry 0 names an issuer that is not itself an anchor, the server is under-sending and that is a server-side defect regardless of how many clients happen to work around it.

The reverse mistake is also common. A server that sends the root as well is merely wasteful, not broken; the client ignores an anchor it does not already trust.

Order matters too, though less than people fear. The leaf must come first, because the peer’s own certificate is defined to be the first entry; the certificates after it are supporting material for path building. A list that arrives in a scrambled order is a configuration defect worth fixing even where a tolerant client copes, because the next client along may not.

The PEM blocks that -showcerts prints are ordinary certificates, and that is more useful than it sounds. Save one to a file and every offline tool you already know applies to it: you can read its subject alternative names, its extended key usage, its validity window and its fingerprint, all against exactly the bytes the service is serving rather than against the file somebody believes is deployed. Comparing the fingerprint of the served leaf with the fingerprint of the file on the server is a two-command way to catch a process that was never reloaded after a renewal.

Asking about protocol versions honestly

s_client can be pinned to a version, and the result is easy to misattribute. Requesting an obsolete version from a modern OpenSSL build produces this:

error:0A0000BF:SSL routines:tls_setup_handshake:no protocols available

Nothing was sent. The local library refused to offer that version at its configured security level, so this string is evidence about your client, not about the server’s policy. To learn what a server accepts you need a client that is willing to offer the version in question, and the honest answer in most estates is that you should be measuring which versions the server offers rather than hunting for one it still tolerates.

Production discipline

1. Always redirect stdin. </dev/null is the difference between a command and a hung pipeline step.
2. Always pass -servername explicitly. Especially when -connect names an address, where there is no name for the tool to infer.
3. Quote the numeric error, not the sentence. num=20 and Verify return code: 0 (ok) survive translation between tool versions; paraphrases do not.
4. Verify against the failing application’s anchor file. -CAfile reproduces the process that is actually broken rather than the shell you happen to be typing in.
5. Add -verify_return_error in automation. A check that reports a certificate problem and then exits zero has taught your monitoring to ignore the fault.

Cross-course references

• Linux for Production Sysadmins - Part LXXI (TLS) covers where the host default trust anchors live on each distribution family, which is the store s_client consults when no anchor file is nominated.
• Kubernetes for Production Sysadmins - Part CXIV (TLS) covers running the same interrogation from inside a Pod, where the anchor set belongs to the image rather than to the node.
• Observability for Production Sysadmins - Part LXIV (TLSMonitoring) covers scheduling this handshake as a probe so that the verify return code becomes a time series instead of a one-off command.

Quiz