Root, intermediate and leaf: why the middle layer exists

Almost every PKI you will meet has three layers: a root that is nothing but an anchor, one or more issuing CAs beneath it, and the leaf certificates that services actually present. The layers are not a convention or a habit. Each one exists because of a specific property the others cannot provide, and the middle one exists because of a property of key custody rather than cryptography.

What the encoding actually distinguishes

Nothing in the certificate format says root, intermediate or leaf. There are only extension values, and a verifier reads them mechanically. Two extensions carry almost all the meaning.

basicConstraints states whether this certificate’s public key may be used to verify further certificate signatures. keyUsage states which operations the key is permitted to perform, and the keyCertSign bit is the one that turns a certificate into a CA in practice. RFC 5280 is blunt about the interaction: if the cA boolean is not asserted, the keyCertSign bit must not be asserted, and if basicConstraints is absent or cA is false, the certified key must not be used to verify certificate signatures.

Here are the three profiles from a working two-tier lab PKI, written as OpenSSL extension files:

# root, self-signed
basicConstraints=critical,CA:TRUE,pathlen:1
keyUsage=critical,keyCertSign,cRLSign
subjectKeyIdentifier=hash

# issuing CA, signed by the root
basicConstraints=critical,CA:TRUE,pathlen:0
keyUsage=critical,keyCertSign,cRLSign
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always

# leaf, signed by the issuing CA
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:app.lab.example,DNS:www.app.lab.example
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer

The root and the issuing CA are structurally identical apart from pathlen and the fact that one is self-signed. The leaf differs in kind: it cannot sign certificates, it names a service rather than an authority, and it carries an extended key usage that pins what the key may be used for. Reading a certificate back confirms the profile landed:

LEAF=app.crt
openssl x509 -in "$LEAF" -noout -ext basicConstraints,keyUsage,extendedKeyUsage

X509v3 Basic Constraints: critical
    CA:FALSE
X509v3 Key Usage: critical
    Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
    TLS Web Server Authentication

Four reasons the middle layer earns its place

The obvious question is why not sign leaves directly from the anchor. Four independent arguments answer it, and they are worth separating because different estates are motivated by different ones.

• Key custody. The root key can be taken off every network permanently, because it signs perhaps a dozen objects in a decade. The key that must be available every time a service is issued a certificate is the issuing CA’s, and that one has to live somewhere reachable.
• Blast radius. If an issuing CA key is compromised, the response is to revoke one certificate, stand up a replacement issuing CA under the same root, and reissue. The anchor in every trust store on earth is untouched.
• Distribution asymmetry. Changing an anchor means editing every relying party. Changing an intermediate means editing the chain file on your own servers, because the server ships the intermediate in the handshake. One is a fleet-wide project; the other is a deployment.
• Operational separation. Different issuing CAs can carry different constraints, different policies and different operators, so a mistake or a breach in one does not automatically extend to the others.

flowchart TD
    R["Root CA\nCA:TRUE pathlen:1\noffline key"] --> I["Issuing CA\nCA:TRUE pathlen:0\nonline key"]
    I --> L1["app.lab.example\nCA:FALSE serverAuth"]
    I --> L2["www.app.lab.example\nCA:FALSE serverAuth"]

The hierarchy in the diagram is a chain of signatures, not a chain of files. The root signed the issuing CA certificate once. The issuing CA signs every leaf. A relying party holds only the root, receives the leaf and the issuing CA certificate from the server, and reconstructs the chain locally.

What the middle layer costs you

The intermediate has to reach the verifier somehow, and the verifier does not have it. That is the price, and it is paid in production far more often than any of the benefits are collected.

A server that presents only its leaf produces this from a client, even one whose trust store holds the correct root:

depth=0 CN=app.lab.example
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=app.lab.example
verify error:num=21:unable to verify the first certificate

The same endpoint, configured to present the leaf followed by the issuing CA certificate, gives the verifier everything it needs:

Certificate chain
 0 s:CN=app.lab.example
   i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
 1 s:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
   i:O=RunBook Academy Lab, CN=RunBook Lab Root CA

Notice what is not in that list: the root. The server must not send it and gains nothing by doing so, because a relying party that has the root already has it, and one that does not will not be persuaded by receiving a copy. The rule is that the chain runs from the leaf up to, but not including, the anchor.

The second cost arrives later, when the issuing CA itself has to be renewed, and it comes in two very different sizes depending on one decision. Renewing the intermediate on the same key pair leaves every existing leaf signature intact, because those signatures were made with that key. The only work is rebuilding and redeploying the chain files. Renewing on a fresh key pair invalidates nothing cryptographically, but no leaf signed by the old key verifies against the new certificate, so every certificate beneath it has to be reissued before the old intermediate expires.

Key continuity therefore buys a cheap renewal, and key rotation buys a fresh key at the cost of a full reissuance campaign. Both are legitimate; what is not legitimate is discovering which one you chose during the renewal. Decide it when the CA is created, write it down next to the expiry date, and size the renewal window accordingly.

Assembling the chain file, and checking it before deployment

The chain a server presents comes from a file you built, and almost no server software inspects that file for you. It reads the certificates and sends them. If the build step produced one certificate instead of two, the server starts cleanly, serves traffic to clients that happen to hold the intermediate already, and fails for everyone else. There is no log line and no health check failure.

The convention is leaf first, then each issuer in turn, ending below the anchor. Building and counting it takes one extra command:

cat app.crt srv-ca.crt > app-fullchain.crt
openssl storeutl -noout -certs app-fullchain.crt

0: Certificate
1: Certificate
Total found: 2

Total found: 2 is the assertion worth automating. A two-tier PKI should always produce a two-certificate chain file for a leaf, so a build that emits one has silently dropped the intermediate and a build that emits three has probably appended the root. Making that count an explicit check in the pipeline converts an invisible packaging mistake into a failed build, which is much cheaper than converting it into a client-visible outage.

Treat the chain as a versioned artefact rather than something assembled on the server with a text editor. When the issuing CA is eventually renewed, every chain file has to be rebuilt and redeployed, and that is far easier when the file has a build step and a known provenance than when each host has its own hand-edited copy.

Production discipline

1. Set pathlen deliberately on every CA certificate. Omitting it imposes no limit, which is rarely what a design intends.
2. Assemble and validate the chain file as an artefact. Build leaf plus intermediates in order, count the certificates in it, and treat that file as a deployable rather than something assembled by hand on the server.
3. Test with an empty local store. A verification that succeeds on your workstation because a stray intermediate is cached tells you nothing about a fresh client.
4. Never add an intermediate to a trust store to silence a chain error. It papers over a server misconfiguration on one host and promotes an issuing CA to anchor status everywhere it is applied.

Cross-course references

• Linux for Production Sysadmins - Part LXXI (TLS) covers how a web server is told which file holds the chain, which is where a leaf-only configuration is usually created.
• Kubernetes for Production Sysadmins - Part LXXVI (Certs) covers a cluster whose control-plane components use several distinct CAs at once, and shows the separation argument applied in practice.
• Observability for Production Sysadmins - Part XI (Blackbox) covers probing from outside the host, which is the only reliable way to detect that a server has stopped sending its intermediate.

Quiz