Secrets, PKI & CertificatesVII · TLS for OperatorsTLS
The TLS 1.3 handshake, message by message
What you'll learn
- Order the TLS 1.3 handshake messages into their key exchange, server parameters and authentication phases
- State precisely which handshake messages are cleartext and which are protected by handshake traffic keys
- Explain what EncryptedExtensions, Certificate, CertificateVerify and Finished each contribute
- Locate a handshake failure by identifying the last message that was successfully exchanged
Prerequisites
Practice
Verified against OpenSSL 3.5.x teaching target; 3.0+ minimum · OpenSSH 10.x teaching target; 8.2+ minimum for certificate workflows · OpenBao 2.6.x · Smallstep step-ca 0.30.x · Certbot / Pebble Certbot current release; Pebble 2.10.x ACME test server · Kubernetes (cross-course target) 1.36.x · PostgreSQL 17.x · 2026-08-26
A TLS 1.3 handshake is a small, strictly ordered conversation that finishes in one round trip. Knowing the order is not academic. When a connection dies, the message you last saw on the wire narrows the cause from “TLS is broken” to one of four specific problems, and it does so before you have read a single log line from the application.
Three phases inside one round trip
RFC 9846 groups the messages into three phases. The key exchange phase establishes the shared secret and settles the cryptographic parameters. The server parameters phase carries the negotiated extensions that are not needed to derive keys. The authentication phase carries the certificate, the proof of key possession, and the key confirmation.
sequenceDiagram
participant C as Client
participant S as Server
C->>S: ClientHello, key_share, signature_algorithms (cleartext)
S->>C: ServerHello, key_share (cleartext)
Note over C,S: both sides derive handshake traffic keys
S->>C: EncryptedExtensions
S->>C: CertificateRequest (only for mutual TLS)
S->>C: Certificate
S->>C: CertificateVerify
S->>C: Finished
Note over C,S: client validates the chain and the signature
C->>S: Finished
Note over C,S: application traffic keys now in use
The diagram is the whole protocol for a normal certificate-based connection. Two cleartext hellos settle the parameters and exchange key shares. The moment both sides have a shared secret they switch to handshake traffic keys, and the entire remainder of the server’s first flight travels encrypted. The client answers with a Finished message of its own, and application data begins. There is no separate confirmation round trip, which is precisely why TLS 1.3 connects in one round trip where TLS 1.2 needed two.
What is cleartext, and exactly when that stops
This is the single most misreported fact about TLS 1.3. ClientHello and ServerHello are not encrypted. RFC 9846 section 1.3 is explicit that all handshake messages after the ServerHello are now encrypted, and the protocol overview marks the end of the key exchange phase with the note that everything after it is encrypted. Encryption begins at EncryptedExtensions, not at ClientHello.
What a passive observer therefore still sees on a TLS 1.3 connection:
- The server name indication. The
server_nameextension in ClientHello carries the hostname the client is asking for, in plain text, unless Encrypted Client Hello is in use. - The application protocol offer. The ALPN list, so an observer learns whether this is HTTP/2, HTTP/1.1 or something else.
- The client’s cryptographic preferences. The offered versions, the supported groups, the signature algorithms and the cipher suite list are all readable.
- The chosen parameters. ServerHello names the selected cipher suite and the server’s key share.
What TLS 1.3 newly hides, compared with TLS 1.2, is the server’s certificate and the extensions that moved into EncryptedExtensions. That is a real privacy gain, because the certificate names the service. It is not the same as hiding the connection.
The server first flight, message by message
Once the client has the ServerHello it can derive the handshake traffic keys and decrypt the rest of the flight. Each message in that flight has a distinct job.
EncryptedExtensions carries the negotiated extensions that were not required to compute keys, ALPN among them. It exists so those values stop travelling in ServerHello where anyone could read them.
Certificate carries the server’s end-entity certificate and, in practice, the intermediates needed to build a chain. Whether the server sends those intermediates is a configuration decision, and getting it wrong is the most common TLS misconfiguration in production. You can see what was actually sent:
HOST=app.lab.example
PORT=443
openssl s_client -connect "$HOST:$PORT" -servername "$HOST" -showcerts </dev/null
A correctly configured server produces a chain listing with both the leaf and its issuer:
Certificate chain
0 s:CN=app.lab.example
i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
1 s:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
i:O=RunBook Academy Lab, CN=RunBook Lab Root CA
Read it as a ladder. Entry 0 is the leaf, and its i: line names
the issuer it expects. Entry 1 is that issuer, and its own i: line
names the root. The chain the server sent is complete when the last
i: value is a certificate the client already trusts locally. The
root itself does not need to be sent.
CertificateVerify then proves that the peer holds the private key for the certificate it just sent, signed over the handshake transcript. Finished is not a signature at all: it is a MAC over the transcript including the Certificate and CertificateVerify messages, computed with a key derived from the handshake traffic secret. It provides key confirmation, proving both sides really did derive the same secret, and integrity over everything exchanged.
Reading a failure by how far the handshake got
The fixed order turns a vague failure into a short list. Work backwards from the last thing you observed.
If the connection dies with no ServerHello at all, the failure is in the key exchange phase. The two peers share no acceptable protocol version, no acceptable group, or no acceptable cipher suite. Nothing about certificates is implicated, because no certificate has been sent yet. This is also the shape of a failure caused by something in the path that mangles or blocks the ClientHello.
If you see a ServerHello and a certificate chain but the client refuses afterwards, the failure is in validation. The server did its job and the client rejected the identity: an incomplete chain, an untrusted issuer, an expired certificate, or a name that is not in the subjectAltName. Every one of those is diagnosed from the certificate and the client’s trust store, not from the protocol.
If the client asks for a certificate and gets nothing, look at the CertificateRequest. That message only appears when the server is configured for mutual TLS, and its absence is the fastest way to prove that a server you believed was requiring client certificates is not.
Finally, note what is not in the flow. TLS 1.3 removed ChangeCipherSpec from the state machine; it survives only as a dummy record sent for middlebox compatibility. If a colleague is looking for ChangeCipherSpec to tell them when encryption started, they are reading a TLS 1.2 diagram. Encryption is keyed off the ServerHello.
Production discipline
- Capture the handshake, not the application error. An HTTP 502 tells you a connection failed. A handshake capture tells you which message it failed after, which is the actual diagnosis.
- Always pass the server name explicitly when testing. Without it you exercise a different code path from your users and may be served a default certificate that nobody in production ever sees.
- Verify what the server sends, not what the file on disk contains. The chain in the Certificate message is what clients act on. A correct bundle that the daemon never reloaded is not serving anything.
- Do not design around handshake privacy you do not have. If concealing the destination hostname matters, that is an Encrypted Client Hello project with its own DNS dependencies, not a side effect of enabling TLS 1.3.
Cross-course references
- Linux for Production Sysadmins - Part XXII (NetTroubleshoot) covers packet capture and path analysis, which is how you prove that a ClientHello left the host and what came back.
- Kubernetes for Production Sysadmins - Part CXIV (TLS) covers Ingress termination, where the certificate presented in the server flight is chosen by the controller rather than by a file path.
- Observability for Production Sysadmins - Part LXIII (Synthetic) covers probes that complete a real handshake on a schedule, so a broken chain is discovered before a user finds it.
Quiz
Knowledge check · 4 questions
Q1. At which point in a TLS 1.3 handshake does encryption of handshake messages begin?
Q2. On a TLS 1.3 connection without Encrypted Client Hello, a passive observer can read the hostname the client requested.
Q3. A client sends a ClientHello and the connection closes with no ServerHello. Which class of problem is ruled out, and why?
Q4. Work out where in the handshake this failure lives and what to change.
A new partner integration to api.example.com fails from the partner network only. Your own probes from three regions succeed. The partner reports that their client library reports a connection reset with no certificate error text, and their security team has already opened a ticket asking you to reissue the certificate.
Passing score: 75%. Answers are checked in this browser.