How a certificate chain is assembled, and who supplies which certificate

A client does not accept a certificate because the certificate looks correct. It accepts it because it can build an unbroken run of signatures from that certificate up to a key it already holds and already believes. Two parties contribute material to that run, and they contribute different parts of it. Nearly every chain failure in production is a disagreement about which party owed which certificate.

What the server sends, and what it deliberately withholds

During the handshake the server sends a Certificate message, which carries an ordered list of certificates. The first entry is the server’s own end-entity certificate, normally called the leaf. Everything after it is supporting material placed there by whoever configured the service. In a two-tier private PKI that means one intermediate. Against a public CA it is usually one intermediate as well, occasionally two.

The root is normally not in that list, and its absence is a design decision rather than an oversight. A root is self-signed: the signature on it was produced with the private key that matches the public key inside it. That demonstrates only that whoever assembled the file held the matching key. It is no evidence of authority, because anyone can generate a key pair and sign a certificate naming any organisation they like. A client that already holds the root gains nothing from a second copy, and a client that does not hold it gains nothing either. Shipping it costs bytes on every handshake and buys no security.

The client supplies the anchor. On a general purpose Linux host the anchor set is a bundle of several hundred public CA certificates that the distribution ships and maintains. Inside a private PKI it may be a single file holding a single certificate. What matters is not where the file sits but what membership of it means: a certificate in the anchor store is believed without further proof, because there is no further proof to be had.

• The leaf. Supplied by the server. It names the service in its subjectAltName and carries the public key that matches the private key the server used to sign the handshake transcript.
• The intermediates. Supplied by the server. They are not trusted by the client. Their only job is to join the leaf to something the client does trust.
• The anchor. Supplied by the client, from its own store, never taken from the connection. This is the trust boundary.
• In mutual TLS the roles mirror exactly. The client sends its own leaf plus intermediates, and the server holds the anchor that terminates the client’s path.

flowchart LR
    L["Leaf: CN=app.lab.example"] -->|"issuer names"| I["Intermediate: Server Issuing CA"]
    I -->|"issuer names"| R["Root: Lab Root CA"]
    W["Certificate message on the wire"] -.-> L
    W -.-> I
    C["Client trust store"] -.-> R

The diagram shows three certificates in one path but only two of them travelling. The leaf points at its issuer by name, that issuer points at its own issuer by name, and the last hop lands on a certificate the connection never carried. The path is longer than the transmission.

Chain building is a search, not a walk down the list

The validator starts at the leaf and asks a narrow question: which certificate could have issued this one. A candidate is any certificate whose subject field equals the leaf’s issuer field. Candidates are drawn from two pools, the certificates the peer sent and the local anchor store. Having chosen a candidate, the validator checks the leaf’s signature using that candidate’s public key. If the signature verifies, it moves up one level and repeats. It stops when it reaches a certificate that is in the anchor store.

Calling this a search rather than a walk is not pedantry. More than one candidate can exist. A CA’s intermediate is frequently certified more than once: identical subject name and identical public key, wrapped in two different certificates signed by two different issuers. That is cross-certification, and it is how an operator migrates a fleet between roots without a flag day. Because it exists, an implementation that picked the first candidate and abandoned the attempt on failure would reject deployments that are entirely correct. Real validators backtrack and try the alternatives.

The practical consequence is that ordering on the wire is a strong hint rather than a contract. A modern client will usually cope with a misordered bundle and will ignore unrelated certificates that were included by accident. Do not lean on that tolerance. Minimal embedded stacks really do walk the list in order, and a bundle assembled the wrong way round is a latent fault waiting for the one client that reads it literally. Put the leaf first and place each issuer immediately after the certificate it issued.

Modelling both roles at the command line

The OpenSSL verify options name the two roles precisely, which makes them an accurate offline model of a handshake. The -CAfile option is the anchor store. The -untrusted option is the bag of certificates the peer sent: usable for building the path, never permitted to terminate it.

ROOT=root.crt
ISSUING=srv-ca.crt
LEAF=app.crt

# The intermediate the server sent is path material, not an anchor.
openssl verify -CAfile "$ROOT" -untrusted "$ISSUING" "$LEAF"

# The same leaf, with the intermediate withheld.
openssl verify -CAfile "$ROOT" "$LEAF"

# The intermediate offered as the anchor, with no root behind it.
openssl verify -CAfile "$ISSUING" "$LEAF"

app.crt: OK
error 20 at 0 depth lookup: unable to get local issuer certificate
error 2 at 1 depth lookup: unable to get issuer certificate

The first result proves that the leaf itself is sound: correct signature, correct issuer, inside its validity window. The second result changes only one thing, the availability of the intermediate, and the verification collapses at depth 0. That is the entire diagnostic value of -untrusted. It lets you separate a broken certificate from a correct certificate that was delivered incompletely, without touching the server.

The third result is the interesting one. The intermediate was offered as the anchor, and OpenSSL still failed, this time at depth 1 with a different error number. It walked from the leaf to the intermediate, accepted that hop, and then went looking for the intermediate’s own issuer, because by default it insists the path end at a self-signed certificate. The -partial_chain option relaxes exactly that requirement and lets any certificate in the store terminate the path.

Reading the chain as it was actually sent

Everything above becomes concrete the moment you look at a live endpoint. The -showcerts option prints the certificates exactly as the server transmitted them, before any local judgement is applied.

HOST=app.lab.example
PORT=443

openssl s_client -connect "$HOST:$PORT" -servername "$HOST" -showcerts </dev/null

Certificate chain
 0 s:CN=app.lab.example
   i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
 1 s:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
   i:O=RunBook Academy Lab, CN=RunBook Lab Root CA

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
Verify return code: 0 (ok)

Read the block as pairs. Line s: is a certificate’s subject and line i: is its issuer. Entry 0’s issuer line and entry 1’s subject line are identical character for character, and that identity is the join. Entry 1’s issuer names a certificate that is not present anywhere in the block, which is correct and expected: that is the root, and the client was required to hold it already. Two certificates crossed the network and three took part in the decision.

The last line reports the local verdict and nothing more. A Verify return code: 0 (ok) speaks only for the trust store on the machine that ran the command. It is not a property of the server, and it does not predict what a container, a Java service or a colleague’s laptop will conclude about the same bytes.

Production discipline

1. Configure the server with a bundle, not a single certificate. The leaf alone is a valid file that produces a valid looking configuration test and an invalid chain for every client that has no other source for the intermediate.
2. Treat the anchor set as inventory. Know which anchors each fleet holds, and know it well enough to answer the question without logging in to a host and guessing.
3. Verify from a client that is not the one you configured. A local success proves your own store, and your own store is the least representative one you have access to.
4. Never anchor on an intermediate to make a failure go away. It converts a chain delivery bug into a library-dependent outcome and sets an expiry date on your entire estate.

Cross-course references

• Linux for Production Sysadmins - Part LXXI (TLS) covers configuring a service to present its certificate file, which is the exact place where the intermediate is either included or forgotten.
• Kubernetes for Production Sysadmins - Part CXIV (TLS) covers ingress termination, where the chain a client sees is assembled by the ingress controller and not by the workload behind it.
• Observability for Production Sysadmins - Part LXIV (TLSMonitoring) covers probing an endpoint continuously, so that a change in the chain is noticed by a monitor rather than by a customer.

Quiz