← All runbooks in Secrets, PKI & Certificates

critical risksecurity relevant~480 min

Runbook: Respond to a Compromised Certificate Authority Key

1 · Prerequisites

Confirm every item is in place before any state change.

An incident commander, because this procedure runs for days and will change hands across shifts
Administrative access to the certificate authority host and to whatever holds its key material, including any hardware security module or offline medium
A current inventory of every service that presents a certificate from this authority and every client that trusts its anchor, including containers and language runtimes with their own trust stores
Authority to take a planned outage on individual services, since some of them cannot be reissued without a restart
A build environment for the replacement hierarchy that is separate from the compromised one, with its own key storage
A written distribution mechanism for trust anchors across the estate, such as configuration management, and evidence that it currently converges
Read access to the issuance records, the authority logs and the host logs covering the whole period from suspected exposure onward

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Establish what is actually believed to be compromised, and on what evidence. A key exported from a hardware module, a backup restored to an untrusted host, a signing service reachable without authentication and a hardware module with an unexplained usage counter are four different findings with four different scopes.
· Determine whether this is a root key or an issuing key. A compromised issuing key can be distrusted from the root above it, if anything in the estate actually checks. A compromised root removes that option entirely and forces a full hierarchy replacement.
· Confirm the authority is private and not publicly trusted. A publicly trusted authority is governed by the root programmes and by the operator's own incident obligations, and this procedure does not apply to it.
· Count what the key signed, from the issuance record rather than from memory. The number of live certificates determines whether reissuance is an afternoon or a fortnight, and it is the single figure the incident commander needs first.
· Establish which trust stores hold the anchor. The operating system store is the one people remember. Java keystores, language runtime bundles, container images, appliances and hardware devices each hold their own copy and each needs its own distribution path.
· Confirm the replacement hierarchy can be built somewhere the compromise did not reach. Building the new root on the host you are treating as compromised repeats the incident with fresh keys.
· Confirm you can take a service outage, and which services need one. Anything that reads its certificate once at start-up requires a restart, and a restart during an incident needs an owner who has agreed to it.
· Agree the honest timescale before starting. Stopping issuance takes minutes, scoping takes hours, and reissuing plus redistributing trust across a real estate takes days. Communicating that at the start prevents the procedure being abandoned halfway, which is the worst possible end state.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Stop issuance before anything else. The holder of the key does not need your service in order to sign, but your own automation will keep minting certificates from a compromised authority for as long as it is running, and every one of them becomes another thing to reissue.
2Preserve the authority state before you change it, and do not destroy the key yet. Copy the whole configuration, database and log set to evidence storage and checksum the copy. You still need the key to distinguish certificates issued before the incident from certificates issued after it, and destroying your copy has no effect whatsoever on the copy somebody else holds.
3Scope what the key signed. Read the issuance record, then confirm it against reality by connecting to every endpoint in the inventory and reading the chain it presents. The record and reality disagree more often than teams expect.
4Scope what trusts the anchor. Enumerate every operating system store, every application store, every container image and every appliance. Each one is a separate distribution path and a separate opportunity to be forgotten.
5Choose the distrust mechanism honestly, and name what enforces it. A revocation list signed by the compromised key is not a trustworthy statement, because whoever holds the key can sign a different one. In a private estate the reliable mechanism is removing the anchor from the trust stores you control, and revocation is a record rather than a control unless you can name the component that fetches and enforces it.
6Build the replacement hierarchy on separate infrastructure, with new key material throughout. Reusing any key from the old hierarchy carries the old exposure into the new one. Constrain the new issuing authority with a path length of zero and a key usage limited to certificate and revocation list signing.
7Distribute the new anchor before you remove the old one. During the overlap both anchors are trusted, which is what allows services to be cut over one at a time rather than all at once. Skipping the overlap converts a controlled migration into an estate wide outage.
8Reissue service by service, in dependency order, with a new key for every certificate. Start with the services other services authenticate to, and prove key and certificate correspondence before deploying each pair.
9Restart or reload each service and confirm it is presenting the new chain. A renewed certificate on disk that the running process has not read is the most common false completion in this whole procedure.
10Remove the old anchor only when every service has been reissued and every client has the new one. Removing it early breaks every service that has not been cut over yet, at the same moment, across the estate.
11Prove the old chain now fails. Verifying an old certificate against the updated store must produce an issuer error. If it still verifies, the old anchor is present somewhere you have not looked.
12Handle the platforms that cannot be reissued in place separately. A Kubernetes cluster built with kubeadm is the standard example: kubeadm does not support rotation or replacement of certificate authority certificates out of the box, so it needs its own planned procedure rather than an improvisation inside this incident.
13Shorten the lifetimes you reissue with. The durable lesson of an authority compromise is that reissuance speed is the control. Certificates measured in weeks with automated renewal make the next incident an afternoon rather than a fortnight.
14Close with a written record of scope, timeline and residual risk. Name every certificate reissued, every trust store updated, every store you could not reach, and the date each remaining gap is owned to.

4 · Verification

Confirm the procedure actually fixed the problem.

✓Every endpoint in the inventory presents a chain whose issuer is the new issuing authority, read from a client connection rather than from the file on the server.
✓A client on a freshly built host, carrying only the new anchor, completes a connection to each endpoint and reports a successful verification result.
✓An old certificate verified against the updated trust store fails with an issuer error, and the error names the level at which the chain broke.
✓The public key in each deployed certificate matches the private key the service is using, compared by digest rather than by file name.
✓The old anchor is absent from every enumerated trust store, including application keystores and container images, checked by an owner other than the person who removed it.
✓The old certificate authority service remains stopped and disabled, confirmed from the service manager rather than from the absence of new certificates.
✓The issuance record for the new authority accounts for every certificate now in production, with no certificate in production that the record does not know about.
✓The evidence archive verifies against its recorded checksum and is stored where the incident record can cite it.
✓Each remaining unreachable trust store is listed in the incident record with a named owner and a date.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶The compromise itself does not roll back, and the old authority is never returned to service. Any rollback here is a rollback of an individual migration step.
↶If a service fails on its new certificate, roll that service back to its previous certificate only while the old anchor is still present in the trust stores, and record it as an open exposure until it is cut over.
↶If the old anchor has already been removed and a service is broken, move forward rather than back: reissue that service again from the new hierarchy, because reinstating the old anchor restores trust in the compromised key across the whole estate.
↶If the new issuing authority certificate is wrong, for example carrying the wrong constraints, reissue it from the new root and redeploy the chain. The root itself does not need rebuilding for a mistake one level down.
↶If the replacement root itself is wrong, build a second replacement root and repeat the overlap. Do not attempt to correct a root in place.
↶If distribution of the new anchor stalls partway, hold the overlap open rather than removing the old anchor on schedule. An extended overlap is a manageable risk; a partial removal is an outage.
↶If the evidence archive is found to be incomplete after the authority host has been rebuilt, record the gap explicitly rather than reconstructing it from recollection.

6 · Escalation

When the runbook isn't enough, contact:

· Evidence that the key was used to issue a certificate you did not request: this is an active intrusion with a forged identity in play, and it belongs to the security incident responder immediately.
· The compromised authority is a root rather than an issuing authority: every certificate beneath it is in scope, the overlap becomes an estate wide trust migration, and the change advisory owner has to be engaged before the schedule is set.
· The authority signs certificates used for authentication rather than only for server identity: every mutual authentication relationship it underpins has to be re-established, and the owners of those relationships need to be in the response.
· A Kubernetes cluster, an appliance or a supplier integration depends on this authority: each needs its own procedure and its own window, and improvising one inside this incident is how outages are created.
· A trust store cannot be reached, for example on an unmanaged device or an appliance without a distribution path: that store is a permanent residual risk and belongs to a named risk owner with a date.
· The reissuance work cannot complete inside the agreed window: this is a decision about accepting continued exposure, and it belongs to the incident commander and the risk owner rather than to the responder on shift.

A certificate authority key is not a credential that grants access to one thing. It is the ability to create identities, and every system that trusts the authority will accept anything the holder chooses to sign. That is why this is the largest incident in the course: containment is not a revocation, it is a rebuild of trust across everything that ever trusted the anchor.

Be honest about the timescale from the first message you send. Stopping issuance takes minutes. Scoping takes hours. Building the replacement hierarchy takes an afternoon. Reissuing every certificate and getting a new anchor into every trust store, including the ones in container images and application keystores, takes days in a real estate. The figure in the header covers the first day only. Teams that promise same day resolution abandon the migration halfway, which leaves both anchors trusted and is worse than either end state.

Hold on to one distinction throughout. Revocation is a statement, and a statement signed by a key someone else also holds is worth nothing. Distrust is the removal of the anchor from the stores you control, and it is the only mechanism here that actually stops the holder of the key from minting an identity your systems will accept.

When this runbook applies, and when it does not

It applies when the private key of an internal certificate authority, root or issuing, is believed to be held by anyone other than the authority itself: a key exported from its hardware module, a backup opened on an untrusted host, a signing endpoint reachable without authentication, or a hardware module with usage that nobody can account for.

It does not apply when:

The authority is publicly trusted. Distrust of a public authority is decided by the root programmes and executed by the operator under their own incident obligations. Nothing you do to your own trust stores changes what the rest of the world accepts.
This is a planned rotation with no compromise. A scheduled intermediate replacement or a root trust transition is a controlled change with an overlap and a rollback, and treating it as an incident costs the estate an unnecessary emergency.
A leaf key was compromised rather than the authority key. Replace that certificate and key pair and be done. The authority is unaffected and rebuilding it would be a self-inflicted outage.
The compromised key belongs to an SSH certificate authority. SSH trust has no chain and no revocation list, and the mechanisms there are the revocation list and the authority key configuration on each host, not trust store surgery.

Blast radius

Action	Reversible?	What it costs if wrong
Stopping issuance	Yes	Every automated renewal fails until the replacement hierarchy is issuing. Certificates near expiry become a second incident inside the first.
Preserving the authority state	Yes	Nothing, and skipping it destroys the only record of what the authority held.
Publishing a revocation statement	No	Clients that do enforce it stop trusting the named certificates immediately, which is an outage if reissuance has not happened yet.
Adding the new anchor alongside the old	Yes	Very little. This is the step that makes the whole migration controllable.
Reissuing a service certificate	Yes, while the old anchor is still trusted	A mismatched key and certificate pair takes the service down at reload.
Removing the old anchor	Only if you kept a copy	Every service not yet reissued fails at once, across every client, with no staged failure to warn you first.
Destroying the old key material	No	You lose the ability to distinguish certificates issued before the incident from those issued after it.

Step 1 - Stop issuance, then preserve the evidence

Service impact possibleon the certificate authority host— Stop the issuing service, then archive its whole state and checksum the archive.

$ INCIDENT=/run/incident
EVIDENCE="$INCIDENT/evidence"
install -d -m 0700 "$INCIDENT" "$EVIDENCE"

# Stop issuance first. The holder of the key does not need your
# service to sign, but your automation will keep minting from it.
systemctl stop step-ca
systemctl is-active step-ca

# Preserve, do not delete. Archive the whole state before anything
# else touches it, then checksum the archive.
tar --create --file "$EVIDENCE/ca-state.tar" --directory /etc/step-ca .
sha256sum "$EVIDENCE/ca-state.tar" | tee "$EVIDENCE/ca-state.sha256"

Stopping the service is containment of your own automation, not of the attacker. It matters because every certificate issued after the incident began is another certificate to reissue, and because the issuance record stops growing at a timestamp you can point to. Do not destroy key material at this stage: you still need it to tell certificates issued before the incident from certificates issued after it, and destroying your copy has no effect on anyone else’s.

Step 2 - Scope what the key signed and what trusts it

Read-only / Safefrom a host with network reach to the estate— Read the authority constraints, then confirm the record against live endpoints.

$ INCIDENT=/run/incident

# What the authority is, and what it was permitted to do.
openssl x509 -in /etc/step-ca/certs/intermediate_ca.crt -noout \
-subject -issuer -serial -dates
openssl x509 -in /etc/step-ca/certs/intermediate_ca.crt -noout \
-ext basicConstraints,keyUsage,extendedKeyUsage

# What is presenting a certificate from it right now. The inventory
# file holds one host and port per line.
while read -r host port; do
echo "== $host:$port"
openssl s_client -connect "$host:$port" -servername "$host" \
  -showcerts </dev/null 2>/dev/null | sed -n '/ s:/p;/ i:/p'
done < "$INCIDENT/tls-endpoints.txt"

The chain lines printed by that loop give you the subject and issuer of each certificate the server actually sends, which is the ground truth the issuance record should agree with. Where they disagree, the live endpoint wins. Then enumerate the trust stores separately: the operating system bundle is only the first of them, and Java keystores, language runtime bundles, container images and appliances each hold an independent copy that no host sweep will find.

Step 3 - Choose the distrust mechanism honestly

Read-only / Safeagainst a representative leaf certificate— Establish whether anything in the estate actually fetches and enforces revocation.

$ INCIDENT=/run/incident

# Does a certificate even name a distribution point?
openssl x509 -in "$INCIDENT/sample-leaf.crt" -noout \
-ext crlDistributionPoints

# Does anything fetch and enforce it? Name the component. A comment in
# a configuration file mentioning revocation is not enforcement.
grep -rn 'crl' /etc/nginx/conf.d 2>/dev/null
grep -rn 'crl' /etc/haproxy 2>/dev/null

If nothing in the estate fetches and enforces the list, publishing one changes nothing at all, and recording it as containment in the incident notes is a false statement that will be read as true later. There is a sharper problem specific to this incident: a revocation list is signed by the authority key, so whoever holds the compromised key can sign their own list that omits whatever they choose. Publish the list as a record if you have one, and treat anchor removal as the control.

Step 4 - Build the replacement hierarchy

Configuration changeon infrastructure the compromise did not reach— Create a new root and a constrained new issuing authority, with all new key material.

$ NEWCA=/run/incident/new-ca
install -d -m 0700 "$NEWCA"
cd "$NEWCA"

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out root.key
openssl req -x509 -new -key root.key -sha256 -days 3650 \
-subj "/O=Example Org/CN=Example Org Root CA 2" \
-addext "basicConstraints=critical,CA:TRUE,pathlen:1" \
-addext "keyUsage=critical,keyCertSign,cRLSign" \
-out root.crt

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out issuing.key
openssl req -new -key issuing.key -sha256 \
-subj "/O=Example Org/CN=Example Org Issuing CA 2" -out issuing.csr

cat > issuing.ext <<'EOF'
basicConstraints=critical,CA:TRUE,pathlen:0
keyUsage=critical,keyCertSign,cRLSign
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF

openssl x509 -req -in issuing.csr -CA root.crt -CAkey root.key \
-CAcreateserial -sha256 -days 1825 -extfile issuing.ext -out issuing.crt
openssl verify -CAfile root.crt issuing.crt

Every key in the new hierarchy is new. Reusing the old root key, or the old issuing key, carries the exposure straight into the replacement and makes the entire exercise theatre. The path length of zero on the new issuing authority means it cannot itself create further authorities, which limits what a future compromise at that level can do. Verify the new issuing certificate against the new root before going any further: a hierarchy that does not verify on the bench will not verify in production either.

flowchart TD
    A["Compromised root CA"] --> B["Old issuing CA"]
    B --> C["Every certificate issued before the incident"]
    D["New root CA"] --> E["New issuing CA, pathlen 0"]
    E --> F["Reissued certificate, one service at a time"]
    G["Trust store during the overlap"] --> A
    G --> D
    H["Trust store after cutover"] --> D

During the overlap the trust stores hold both anchors, so a service still presenting an old certificate and a service already presenting a new one both validate. That is what lets the migration proceed one service at a time instead of as a single estate wide switch. After cutover the stores hold only the new anchor, and at that moment every certificate from the old hierarchy stops validating everywhere at once.

Step 5 - Reissue, service by service, with new keys

Configuration changeper service, in dependency order— Generate a fresh key, issue against the new authority, and prove correspondence.

$ NEWCA=/run/incident/new-ca
SVC=api.example.com
cd "$NEWCA"

# A new key per service. Reusing the old key would carry the old
# exposure into the new hierarchy.
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 \
-out "$SVC.key"
openssl req -new -key "$SVC.key" -sha256 -subj "/CN=$SVC" -out "$SVC.csr"

cat > "$SVC.ext" <<'EOF'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:api.example.com
EOF

openssl x509 -req -in "$SVC.csr" -CA issuing.crt -CAkey issuing.key \
-CAcreateserial -sha256 -days 90 -extfile "$SVC.ext" -out "$SVC.crt"

# Prove the pair belongs together before deploying it.
openssl pkey -in "$SVC.key" -pubout | openssl sha256
openssl x509 -in "$SVC.crt" -noout -pubkey | openssl sha256

The two digests must be identical. That comparison catches the single most common deployment failure in this whole procedure, which is a certificate deployed next to the wrong key, and it catches it before the reload rather than during it. Identity lives in the subject alternative name and not in the common name, so the extension file is where the names that matter are set. Issue in dependency order, starting with the services that other services authenticate to.

Step 6 - Distribute the new anchor before removing the old one

Configuration changeon every client host, through configuration management— Install the new anchor alongside the old and confirm the bundle carries both.

$ NEWCA=/run/incident/new-ca

# Both anchors are trusted during the overlap, which is what allows
# services to be cut over one at a time.
cp "$NEWCA/root.crt" \
 /usr/local/share/ca-certificates/example-org-root-ca-2.crt
update-ca-certificates

# Confirm the generated bundle now carries the new anchor.
openssl storeutl -noout -certs /etc/ssl/certs/ca-certificates.crt \
| grep -F 'Example Org Root CA 2'

The path and the update command above are the Debian family form, which is the pair executed for this course. Other distributions use their own anchor directory and their own trust update command, and you must confirm those from your own distribution documentation before running the sweep. Beyond the operating system, work the list from Step 2: Java keystores, language runtime bundles, appliances and every container image that baked the old anchor in need their own updates and their own rebuilds.

Step 7 - Remove the old anchor and prove the old chain fails

Cluster-wide riskonly after every service is reissued and every client updated— Remove the compromised anchor, then require an old certificate to fail verification.

$ INCIDENT=/run/incident

rm -f /usr/local/share/ca-certificates/example-org-root-ca.crt
update-ca-certificates --fresh

# With the old intermediate supplied, the failure names the unknown
# issuer one level up.
openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt \
-untrusted "$INCIDENT/old-issuing.crt" "$INCIDENT/old-leaf.crt"

# Without the intermediate, the same certificate fails one level down.
openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt \
"$INCIDENT/old-leaf.crt"

The first command must report error 2 at 1 depth lookup: unable to get issuer certificate, because the old intermediate is present but its issuer is no longer trusted. The second must report error 20 at 0 depth lookup: unable to get local issuer certificate, because the chain breaks one level lower when the intermediate is not supplied at all. Either failure is the proof you want. A result of OK from either command means the old anchor is still trusted somewhere in that store, and the removal has not actually happened.

Step 8 - Validate the estate end to end

Read-only / Safefrom a freshly built host carrying only the new anchor— Confirm each endpoint presents the new chain and verifies cleanly.

$ INCIDENT=/run/incident

while read -r host port; do
echo "== $host:$port"
openssl s_client -connect "$host:$port" -servername "$host" \
  -showcerts </dev/null 2>/dev/null \
  | grep -E 'Verify return code|Protocol|Cipher is'
curl --fail --silent --show-error --output /dev/null \
  --write-out '%{http_code} %{ssl_verify_result}\n' \
  "https://$host:$port/"
done < "$INCIDENT/tls-endpoints.txt"

Run this from a host that was built after the migration and carries only the new anchor, because a machine that still holds the old one cannot tell you whether the cutover worked. A clean run reports Verify return code: 0 (ok) for every endpoint. Where a chain is incomplete the client instead reports that it is unable to get the local issuer certificate, which means the server is sending its leaf without the new intermediate rather than that the anchor is missing.

Common pitfalls

Symptom	Cause	Action
The new certificate verifies on the bench but clients still fail	The server is sending the leaf alone; the new chain has a new intermediate that must be sent with it	Deploy the full chain and confirm entry one in the presented chain is the new issuing authority
Some clients trust nothing after the anchor removal	They read their own trust store rather than the operating system one	Update the application keystore, the runtime bundle and every container image, then re-test from inside each
Certificates keep appearing from the old authority	An automation still holds valid issuing credentials, or the service was restarted by configuration management	Confirm the service is stopped and disabled, and that its automation is paused
The published revocation list changed nothing	Nothing in the estate fetches or enforces it	Treat anchor removal as the control and keep the list as a record
A service fails at reload with a key mismatch	The certificate was deployed next to a different key than the one it was issued for	Compare the two public key digests before every deployment
The cluster cannot be reissued in place	kubeadm does not support rotation or replacement of certificate authority certificates out of the box	Plan a cluster level procedure with its own window rather than improvising it here
The migration stalls with both anchors trusted	The overlap was opened with no completion date and no owner	Set the cutover date at the moment the overlap opens, and track remaining services against it daily

Verification

Connect to every endpoint in the inventory and confirm the chain it presents is issued by the new authority, reading it from the connection rather than from the file on the server. Repeat that from a freshly built host that carries only the new anchor, and require a successful verification result from each. Verify an old certificate against the updated store and require an issuer error, with and without the old intermediate supplied, so that you know which level the chain breaks at. Compare the public key digest of every deployed certificate against the digest of the key the service is actually using. Have someone other than the person who removed it confirm the old anchor is absent from every enumerated trust store, including application keystores and images. Confirm from the service manager that the old authority is stopped and disabled rather than merely idle. Reconcile the new issuance record against production, in both directions. Confirm the evidence archive still verifies against its checksum, and confirm every trust store you could not reach is listed with an owner and a date.

Rollback

The compromise does not roll back and the old authority never returns to service, so everything here is the rollback of an individual migration step. While the old anchor is still present, a service that fails on its new certificate can be returned to its previous certificate and recorded as an open exposure. Once the old anchor has been removed, the only safe direction is forward: reissue that service again from the new hierarchy, because reinstating the old anchor would restore estate wide trust in the compromised key. A wrong new issuing certificate is corrected by reissuing it from the new root and redeploying the chain, without rebuilding the root. A wrong new root means building a second replacement root and repeating the overlap, never correcting a root in place. If anchor distribution stalls, hold the overlap open rather than removing the old anchor on schedule, because an extended overlap is a managed risk and a partial removal is an outage. If the evidence archive turns out to be incomplete after the authority host has been rebuilt, record that gap rather than reconstructing it from memory.