Rotating an intermediate CA

Of everything in a private PKI, the issuing intermediate is the component you can replace with the least disruption. That is not luck. It follows directly from where each part of a chain lives: anchors are configured out of band on every client, intermediates are sent by the server during the handshake. Anything that travels with the traffic can be changed by changing the traffic.

Why the intermediate is the tractable one

Path validation starts from a trust anchor the client already holds and works down to the leaf. The certificates in between do not need to be known in advance; the server supplies them, and a client that has never seen a particular intermediate accepts it the moment it can verify the signature on it using something it does trust.

The practical consequence is that a new intermediate signed by the same root is trusted by every existing client immediately, with no distribution step, no inventory of trust stores and no coordination with anyone who consumes the service. The root’s signature over the new intermediate is what carries the trust, and the root has not changed.

That property is worth naming precisely, because it is exactly what the next lesson does not have. Rotating an intermediate is an operation on servers. Rotating a root is an operation on every client that exists, and the difference between those two is the difference between a week of work and a year of it.

What has to be reissued, and what does not

Almost nothing, immediately. This surprises teams who expect a CA change to invalidate the certificates beneath it.

• Existing leaf certificates stay valid. They are signed by the outgoing intermediate, which is still a valid certificate under the root. Nothing about issuing a sibling intermediate affects them.
• New leaves come from the new intermediate. From the cutover date onwards, the issuing service signs with the new key.
• Chain files must contain both. Every server presenting a leaf signed by the outgoing intermediate must keep sending that intermediate. Every server presenting a new leaf must send the new one. During the overlap the fleet contains both, and the two populations do not interact.
• The outgoing private key must be retained. It is the only key that can sign a revocation list covering the certificates it issued. Destroying it at cutover leaves you unable to revoke anything issued before that date.

flowchart TD
    R["Root CA\nunchanged, the only trust anchor"]
    R --> I1["Issuing CA 2026A\nsigns no new leaves after cutover\nstill signs CRLs"]
    R --> I2["Issuing CA 2026B\nsigns every leaf from the cutover"]
    I1 --> L1["Leaves issued before cutover\nlast one expires 2026-11-29"]
    I2 --> L2["Leaves issued after cutover"]

The tree shows why this rotation is a drain rather than a migration. Nothing is moved from the left branch to the right one. The left branch simply stops growing, and time removes it as each leaf reaches its own expiry.

Creating the new intermediate

The new intermediate needs its own key, its own certificate signing request, and a signature from the root. If the root is held offline, this is the step that requires a ceremony, and it is the only step in this rotation that does.

# 1. New key and request for the incoming issuing CA.
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out srv-ca-2026b.key
openssl req -new -key srv-ca-2026b.key -sha256 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Server Issuing CA 2026B" \
  -out srv-ca-2026b.csr

# 2. The extensions the ROOT will apply. A signing request only
#    requests; the issuer decides.
cat > srv-ca-2026b.ext <<'EXT'
basicConstraints=critical,CA:TRUE,pathlen:0
keyUsage=critical,keyCertSign,cRLSign
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EXT

# 3. Sign with the root.
openssl x509 -req -in srv-ca-2026b.csr -CA root.crt -CAkey root.key \
  -CAcreateserial -sha256 -days 1825 -extfile srv-ca-2026b.ext \
  -out srv-ca-2026b.crt

pathlen:0 states that no further CA may appear below this one, which is correct for an issuing CA that signs only end entities. It is only meaningful because CA:TRUE is asserted and keyCertSign is present; a path length constraint without those is not a valid combination.

Publish before you cut over

The ordering rule for this rotation is short. Every server that will present a leaf from the new intermediate must already be sending the new intermediate in its chain before the first such leaf is issued. Inverting those two steps produces an outage in exactly the population you cut over first.

error 20 at 0 depth lookup: unable to get local issuer certificate

curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)

Those are the two faces of the same fault, one from a local verification and one from a client that trusts the root perfectly well. The client holds the anchor. It cannot build the path because the server did not send the certificate that connects the leaf to that anchor. Nothing is wrong with the new intermediate, the new leaf or the trust store, which is what makes this failure so consistently misdiagnosed.

The verification that clears the step is the one that supplies only the new intermediate as an untrusted input and only the root as the anchor:

# Prove the new path stands on its own before issuing anything
# that depends on it.
openssl verify -CAfile root.crt -untrusted srv-ca-2026b.crt app.crt

# Then confirm what each server is actually shipping.
openssl storeutl -noout -certs /etc/ssl/live/api/fullchain.pem

A result of app.crt: OK from the first command means the chain is complete and correctly signed. The second command lists what is really inside a deployed bundle, which is how you find the host whose chain file was assembled by hand in 2024 and has been copied forward ever since.

The overlap window, worked

The overlap is not a judgement call. It is determined by the longest-lived certificate the outgoing intermediate signed.

Take an estate issuing 90-day leaves, with issuance cutting over to the new intermediate on 2026-09-01. The last leaf the outgoing intermediate can sign is issued on 2026-08-31 and is therefore valid until 2026-11-29. Three obligations follow from that single date:

• The outgoing intermediate must remain in the chain file of every server still presenting one of its leaves, until 2026-11-29.
• The outgoing intermediate’s own notAfter must be later than 2026-11-29. A leaf cannot be validated through an expired issuer, so an intermediate that expires first invalidates every certificate beneath it on that date regardless of their own dates.
• The outgoing key must remain available to sign revocation lists until 2026-11-29.

Only on 2026-11-30 does the outgoing intermediate become removable from chain files, and the key becomes destroyable after a further soak during which nothing has needed it. The dates are arithmetic, which means they can be put in a calendar at the start of the rotation rather than debated at the end of it.

The name-collision trap

The most attractive shortcut in this rotation is to give the new intermediate exactly the same subject name as the old one, so that no configuration anywhere mentions a new string. It creates a genuinely awkward failure.

Path building is a search. A validator collects candidate issuers whose subject name matches the leaf’s issuer name, and tries to verify the signature with each. When two certificates share a subject name but hold different keys, the search has two candidates and only one of them works. Modern validators use the authority key identifier in the leaf to prefer the right one and will backtrack if the first attempt fails. Older and simpler implementations pick the first match, fail the signature check and stop.

The fix is free at design time and expensive afterwards: give the new intermediate a distinguishable common name, as the example above does with the 2026B suffix. The chain files change, which was the thing you were trying to avoid, but they change under your control rather than in a validator you do not own.

Production discipline

1. Publish the new chain first. No leaf may be issued from the new intermediate before every server that will present one is already shipping that intermediate.
2. Give the new issuing certificate a distinct name. A shared subject name turns path building into a coin flip on any validator that does not backtrack.
3. Keep the outgoing key until its last leaf expires. It is the only key that can revoke what it issued.
4. Derive the overlap from the longest leaf, and check the issuer’s own expiry. An intermediate that expires before its leaves invalidates all of them on that date.
5. Inspect deployed bundles, not the bundle you built. The chain file on a long-lived host is frequently not the one the automation believes it wrote.

Cross-course references

• Linux for Production Sysadmins - Part LXXI (TLS) covers the server-side chain file that this rotation edits on every host that terminates the service.
• Kubernetes for Production Sysadmins - Part LXXVI (Certs) covers a cluster PKI whose issuing certificates have exactly this structure, and which are renewed on a similar cadence.
• Observability for Production Sysadmins - Part XI (Blackbox) covers probing a service from outside, which is how the overlap is monitored rather than assumed.

Quiz