← All runbooks in Secrets, PKI & Certificates

high riskservice affecting~240 min

Runbook: Rotate an Intermediate Issuing CA

1 · Prerequisites

Confirm every item is in place before any state change.

secrets-pki-lab-04-build-two-tier-certificate-authority
secrets-pki-rb-05-renew-and-deploy-server-certificate
secrets-pki-cl-02-internal-pki-readiness
Authorised access to the root CA private key under whatever ceremony your organisation requires, because the root is what signs the replacement intermediate.
A current inventory of every certificate the outgoing intermediate issued, with the deployment path and the reload command for each service that holds one.
Written confirmation from the platform owner that no client pins the intermediate public key and no trust store carries the intermediate as an anchor.
A change window agreed with every team whose service will have its deployed chain file replaced during the cutover.
Commit access to the configuration repository that ships chain files, so the change is recorded rather than applied by hand on each host.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm that the root, not the intermediate, is the anchor. On one representative host of every client class, list the anchors the store actually holds with openssl storeutl -noout -certs /etc/ssl/certs/ca-certificates.crt and confirm the root is present. If any store carries the intermediate as an anchor, or any client pins the intermediate public key, stop here: that estate needs a trust transition, not an intermediate rotation.
· Read the outgoing intermediate before replacing it. Run openssl x509 -in srv-ca.crt -noout -subject -issuer -serial -dates and openssl x509 -in srv-ca.crt -noout -ext basicConstraints,keyUsage. Record the subject DN, the serial, the notAfter and the path length. The replacement must be able to issue everything the outgoing certificate could issue.
· Confirm the root has path length left. openssl x509 -in root.crt -noout -ext basicConstraints shows the constraint. RFC 5280 counts the non-self-issued intermediates that follow the certificate and excludes the leaf, so a root marked pathlen:1 permits exactly one intermediate beneath it. A root with pathlen:0 cannot sign an issuing CA at all.
· Confirm the root key is reachable and usable. An offline root that nobody has unlocked this year is a discovery you want before the change window, not during it. Prove the key signs by producing a throwaway signature on a scratch host, and confirm the quorum of key holders is available for the whole window.
· Size the validity of the replacement. The new intermediate must outlive every leaf it will issue, with margin for the reissue programme. An intermediate that expires before its own leaves turns a routine renewal into an outage that no leaf-level monitoring predicted.
· Record the current chain file for every service in scope. Copy each deployed fullchain to a timestamped path on the same host. Rollback for this runbook is a file copy plus a reload, and it is only that cheap if the copy exists before you start.
· Confirm the reload path for every service. For each host, record the command that makes the running process re-read its certificate material, and prove it works now. A chain replaced on disk that no process has re-read is not deployed, and the failure surfaces at the next restart instead of during the change.
· Confirm monitoring reads the served chain, not the file on disk. Expiry monitoring that parses a path on the filesystem cannot see a service that is still presenting the old chain from memory. Point at least one probe at the listener before you begin.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Freeze issuance from the outgoing intermediate. Stop the automation and revoke or suspend the issuance credentials that let it mint new certificates. Every certificate issued after the freeze is one more leaf you must chase during the reissue wave, and an automation that keeps issuing quietly extends the retirement date.
2Generate the replacement private key on the CA host under a restrictive umask. Run umask 077 first, then openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out srv-ca-2027.key. The key must never be copied to a workstation. If your CA key lives in an HSM or a PKCS#11 token, generate it there and export only the public half.
3Create the CSR with a subject that is visibly distinct from the outgoing one. openssl req -new -key srv-ca-2027.key -sha256 -subj "/O=RunBook Academy Lab/CN=RunBook Lab Server Issuing CA 2027" -out srv-ca-2027.csr. A distinguishable common name means every later chain dump tells you at a glance which intermediate is in play.
4Sign the CSR with the root, using an explicit extension section. openssl x509 -req -in srv-ca-2027.csr -CA root.crt -CAkey root.key -CAcreateserial -sha256 -days 1825 -extfile intermediate.ext -extensions v3_intermediate -out srv-ca-2027.crt. The section must assert basicConstraints = critical,CA:TRUE,pathlen:0 and keyUsage = critical,keyCertSign,cRLSign.
5Verify the new intermediate against the root before anything depends on it. openssl verify -CAfile root.crt srv-ca-2027.crt must print OK and exit 0. A non-zero exit here means the root did not sign what you think it signed, and nothing downstream can be trusted until it is resolved.
6Publish the new intermediate wherever chain material is fetched from. Config management repository, artefact store, the caIssuers URL in the AIA extension if you populate one. Publishing before cutover means a service that is deployed early still finds a complete chain.
7Point the issuing configuration at the new key and certificate. In an openssl ca deployment this is the certificate and private_key keys of the CA_default section, plus a fresh index.txt, serial and crlnumber for the new authority. Restart the issuance service and confirm it starts clean.
8Issue exactly one test leaf and verify it end to end. Verify with the new intermediate supplied as untrusted material and the root as the anchor, and assert the purpose: openssl verify -CAfile root.crt -untrusted srv-ca-2027.crt -purpose sslserver test.crt. It must print OK. Do not proceed to any real service until it does.
9Deploy the new chain to the lowest blast radius service first. Reissue its leaf, assemble the fullchain in leaf-then-intermediate order, validate the server configuration, then reload. Verify from a host that is not the server before moving on. One service proves the whole pipeline for a cost you can absorb.
10Reissue and deploy the remaining services in waves. Group by blast radius, smallest first, and leave a bake period between waves long enough for a slow client to reconnect. Each wave ends with an off-host verification, not with a successful reload.
11Track the residual population still chaining to the outgoing intermediate. Probe every listener in the inventory and record the issuer of the leaf it presents. The count must fall monotonically. A count that stops falling means a deployment path you have not found yet, and that path will still be there at retirement.
12Retire the outgoing intermediate from issuance while keeping it available for verification. Keep its certificate in verification bundles and keep its key available for CRL signing for as long as any leaf it issued is still live. Removing it early is what turns a rotation into an incident.
13Withdraw the outgoing intermediate once its last leaf has gone. When the residual count reaches zero and the last issued leaf has expired or been replaced, remove the certificate from published bundles, issue a final CRL, and archive the key material according to your retention policy.
14Record the change. Note the new subject DN, serial, notBefore and notAfter, the services reissued and when, the residual count at each wave, and the date the outgoing intermediate may be destroyed. The next operator to do this will read only this record.

4 · Verification

Confirm the procedure actually fixed the problem.

✓From a host that is not the server, openssl s_client -connect app.lab.example:443 -servername app.lab.example -showcerts returns a chain whose position 1 is the new intermediate and whose verify return code is 0.
✓A client that holds only the root as an anchor completes a request against every reissued service without a verification flag of any kind.
✓For each reissued leaf, openssl verify -CAfile root.crt -untrusted srv-ca-2027.crt -purpose sslserver prints OK and exits 0.
✓The public key in the deployed leaf matches the private key the service loaded, proved by comparing openssl pkey -in app.key -pubout | openssl sha256 with openssl x509 -in app.crt -noout -pubkey | openssl sha256.
✓The residual probe across the whole inventory reports zero listeners still presenting a leaf issued by the outgoing intermediate.
✓The new intermediate notAfter is later than the notAfter of every leaf it has issued, confirmed by comparing the recorded dates rather than by assumption.
✓Certificate expiry monitoring reports the new chain and has not silently kept a stale reading, confirmed by forcing a probe and reading the timestamp on the result.
✓Issuance from the outgoing intermediate is refused, proved by attempting one issuance with the retired configuration and observing it fail.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶For a single service that broke on cutover, restore the timestamped chain file recorded in the pre-checks and reload the process. The old leaf and old intermediate are both still valid, so this returns the service to a working state within one reload.
↶If several services broke in the same wave, restore all of them from their recorded copies before diagnosing. Diagnose on a host that is out of rotation, not on the one carrying traffic.
↶If the new intermediate fails verification against the root, discard it, destroy its key, and issue a fresh CSR. Nothing has been deployed at that point, so the cost is one signing ceremony.
↶If issuance has already moved and must move back, restore the previous issuing configuration and its database files from backup. Do not reuse a serial number that the new authority has already consumed.
↶If a leaf was issued by the new intermediate and deployed before the intermediate was published, publish the intermediate rather than reverting the leaf. The chain is incomplete, not wrong, and publishing is the smaller change.
↶If rollback restores a service but expiry monitoring still alerts, treat the alert as real until proved otherwise and read the served chain directly. A monitoring system pointed at a file cannot see what the process is presenting.

6 · Escalation

When the runbook isn't enough, contact:

· The root key cannot be unlocked, the quorum of key holders is unavailable, or the HSM refuses the operation: escalate to the CA owner and abandon the change window. There is no safe partial state to leave behind.
· A client class is discovered to pin the intermediate public key or to carry it as an anchor: escalate to the owning application team and the platform owner. That estate needs a planned trust transition, which is a different runbook and a different timescale.
· The residual count stops falling and the remaining listeners cannot be attributed to a known deployment path: escalate to the service owners. An unmapped deployment path is a bigger finding than the rotation itself.
· The outgoing intermediate will expire before the reissue programme can complete: escalate immediately to the platform owner. The remaining options all involve either an emergency reissue wave or accepting an outage, and both need a decision above the operator.
· Any evidence emerges that the outgoing intermediate key was exposed rather than merely ageing: stop the planned rotation and escalate to the security owner. A suspected CA key compromise is an incident, and its ordering constraints are the opposite of a planned rotation.
· A regulated or externally audited service is in scope and the change was not covered by the approved change record: escalate to the compliance owner before continuing.

An intermediate issuing CA is the certificate that sits between your root and everything you actually deploy. Replacing it is one of the few large PKI changes that can be done without touching a single client, and the reason is worth holding on to for the rest of this page: clients validate a path that terminates at an anchor they already hold. A new intermediate signed by the same root produces a path that ends at that same anchor, so nothing in any trust store has to change.

That is what makes this so much cheaper than rotating a root. Root rotation is slow because trust store distribution is slow, and trust store distribution is slow because it reaches clients you do not own: a partner integration, an appliance with a firmware release cycle, a container image someone rebuilds twice a year. Intermediate rotation touches only what your servers present, which is a file on a host you control and can put back in one reload.

The discipline that keeps it cheap is ordering. Create and prove the new intermediate first, publish it, move issuance, then replace deployed chains in waves, and retire the old certificate only after the last leaf it signed has gone. Do it in that order and every step is individually reversible.

When this runbook applies, and when it does not

It applies when the root remains valid and trusted, the intermediate beneath it needs replacing, and every client validates to the root rather than to the intermediate.

It does not apply when:

Any trust store holds the intermediate as an anchor. Some estates load an issuing CA directly into application trust stores because it was the file somebody had to hand. For those clients the intermediate is the anchor, and replacing it is a trust transition with the full distribute-first ordering. Running this runbook against them produces a verification failure at cutover on every one of them at once.
Any client pins the intermediate public key. Pinning survives a correctly signed replacement. Find the pins first and coordinate with the owning team.
The root itself needs replacing. Expiring root, changed key policy or a compromised root are all a different procedure with a different risk profile.
The intermediate key is suspected of being compromised. That is an incident. A planned rotation assumes you may run both intermediates in parallel while the old certificates drain, and a compromise removes exactly that assumption.

Blast radius

Action	Reversible?	What it costs if wrong
Generate the new key and CSR	Yes	Nothing, until it is signed
Sign the new intermediate under the root	Yes	Nothing, until it is published or used
Publish the new intermediate	Yes	Nothing; a published unused certificate is inert
Move issuance to the new intermediate	Yes	New leaves chain differently; existing leaves are untouched
Replace a deployed chain file	Only if you kept a copy	Every client of that service fails validation until it is put back
Remove the old intermediate from bundles	Only if you kept a copy	Any leaf it signed that is still live stops validating
Destroy the old intermediate private key	No	You can no longer sign a CRL covering anything it issued

Step 1 - Establish what the outgoing intermediate signed

Read-only / Safefrom a host that is not the server, for each listener in the inventory— Records which intermediate each service is presenting today.

$ HOST=app.lab.example
PORT=443

openssl s_client -connect "$HOST:$PORT" -servername "$HOST" -showcerts </dev/null

Certificate chain
0 s:CN=app.lab.example
 i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
1 s:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
 i:O=RunBook Academy Lab, CN=RunBook Lab Root CA

Illustrative output

Position 0 is the leaf and position 1 is the intermediate that signed it. The i: line under position 1 names the root. This is the reading you will repeat at every wave, and the count of listeners still showing the outgoing intermediate is the only honest measure of how far the rotation has progressed.

Step 2 - Create the replacement intermediate under the same root

Write the extension section first, so the constraints are reviewable rather than buried in a command line.

[ v3_intermediate ]
basicConstraints = critical,CA:TRUE,pathlen:0
keyUsage = critical,keyCertSign,cRLSign
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always

pathlen:0 says that no further non-self-issued intermediate may follow this certificate, which is exactly right for an issuing CA that mints leaves. RFC 5280 requires that pathLenConstraint appears only where cA is asserted and keyCertSign is set, and both are asserted above.

umask 077
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out srv-ca-2027.key
openssl req -new -key srv-ca-2027.key -sha256 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Server Issuing CA 2027" \
  -out srv-ca-2027.csr
openssl x509 -req -in srv-ca-2027.csr -CA root.crt -CAkey root.key -CAcreateserial \
  -sha256 -days 1825 -extfile intermediate.ext -extensions v3_intermediate \
  -out srv-ca-2027.crt

The year in the common name is not decoration. Two intermediates that share a subject DN and differ only by key make chain building ambiguous for clients that select by name, and they make every diagnostic dump you take during the cutover harder to read.

Step 3 - Prove the new chain before anything depends on it

Read-only / Safeon the CA host, before publishing anything— Two assertions: the root signed the intermediate, and the intermediate can issue a usable server leaf.

$ openssl verify -CAfile root.crt srv-ca-2027.crt
openssl verify -CAfile root.crt -untrusted srv-ca-2027.crt -purpose sslserver test.crt

Both commands must print OK and exit 0. A missing intermediate produces error 20 at 0 depth lookup: unable to get local issuer certificate. An unknown anchor produces error 2 at 1 depth lookup: unable to get issuer certificate. Read the exit status, not just the text: openssl verify exits 2 on failure, and a script that ignores the status will report success on a broken chain.

Step 4 - Move issuance, one test certificate first

Point the issuing configuration at the new key and certificate, and give the new authority its own database files rather than inheriting the old ones.

[ CA_default ]
dir             = /etc/pki/srv-ca-2027
certificate     = $dir/srv-ca-2027.crt
private_key     = $dir/private/srv-ca-2027.key
database        = $dir/index.txt
serial          = $dir/serial
crlnumber       = $dir/crlnumber
new_certs_dir   = $dir/newcerts
default_md      = sha256

Issue one leaf. Verify it with the purpose asserted, as in Step 3. Only when that single certificate validates should any real service be touched.

Step 5 - Deploy the new chain service by service

Service impact possibleon the service host, after the recorded chain copy exists— Assembles leaf-then-intermediate, validates the configuration, then reloads so the process re-reads the material.

$ LEAF=/etc/ssl/certs/app.lab.example.pem
NEWCA=/etc/pki/srv-ca-2027/srv-ca-2027.crt
OUT=/etc/ssl/certs/app.lab.example.fullchain.pem

cp -a "$OUT" "$OUT.pre-rotation"
cat "$LEAF" "$NEWCA" > "$OUT"
nginx -t
systemctl reload nginx

Order matters in a fullchain file: leaf first, then the issuing CA. The root is not included, because the client is expected to hold it already. After the reload, verify from off-host as in Step 1 and confirm position 1 now names the 2027 intermediate.

Step 6 - Retire the outgoing intermediate

Retirement has two distinct moments and conflating them is the classic way to break this. The first is removal from issuance, which is safe as soon as the new authority is minting certificates. The second is removal from verification bundles and destruction of the key, which is safe only when the last leaf it signed has expired or been replaced.

Common pitfalls

Symptom	Cause	Action
`unable to get local issuer certificate` after cutover	The deployed fullchain still carries the old intermediate, or the process was never reloaded	Reassemble the fullchain and reload; verify from off-host, not from the file
Most clients fine, one class fails	A second deployment path, typically a load balancer or a sidecar, holds its own copy of the chain	Find every place the chain is stored, not every place it is configured
Verification passes locally but fails from a client	The client trusts the intermediate directly rather than the root	Stop the rotation for that client class and plan a trust transition
The reissue wave stalls with a stable residual count	An unmapped deployment path is still serving old leaves	Treat the unmapped path as the finding; escalate to the service owner
Issuance succeeds but new leaves fail `-purpose sslserver`	The extension section used for leaves was not carried across to the new authority	Restore the leaf extension section; reissue the affected certificates
The old intermediate expires mid-programme	Its validity was shorter than the reissue timeline	Escalate; the remaining options are an emergency wave or an accepted outage

Verification

The rotation is complete when an independent observer, not the CA host and not the service host, agrees. From a client host, open a connection to every reissued listener and read the chain it presents: position 1 must be the new intermediate and the verify return code must be 0. Repeat the same check from a client whose trust store holds only the root, which proves the path terminates at the anchor rather than at something incidentally present. For each reissued leaf, openssl verify with the new intermediate as untrusted material and the purpose asserted must print OK and exit 0, and the deployed key must still match the deployed certificate by public key digest. The residual probe across the whole inventory must report zero listeners on the outgoing intermediate, the new intermediate must outlive every leaf beneath it, and one deliberate issuance attempt against the retired configuration must fail.

Rollback

Rollback for a single service is a file copy and a reload: restore the chain recorded in the pre-checks and put the process back to the material it was serving before the change. Both the old leaf and the old intermediate remain valid throughout, which is precisely why this rotation is cheap. If a whole wave broke, restore every host in it before diagnosing anything, and diagnose out of rotation. If the new intermediate failed verification against the root, discard it and its key and start again; nothing was deployed. If issuance has already moved and must move back, restore the previous issuing configuration and its database from backup, and never reuse a serial number the new authority has already consumed. If a leaf reached a host before its intermediate was published, publish the intermediate rather than reverting the leaf, because the chain is incomplete rather than wrong.