CA compromise: the incident that is different in kind

When a server key is stolen, the attacker can be one service. When a certificate authority key is stolen, the attacker can be any service that CA is trusted for, including services that do not exist. This is not a bigger version of the previous incident; it is a different one, and the thing that makes it different is that you lose the ability to enumerate what will validate.

Why the issuance index stops being an inventory

A relying party accepts a certificate because the signature on it verifies against a public key it already trusts, the chain reaches a trust anchor, and the extensions permit the use. Nowhere in that process does the client consult a record of what the CA meant to issue. There is no registry lookup and no callback. The signature is the authority.

So an attacker holding the CA private key can sign a certificate for any name inside the CA scope, choose its serial, choose its validity window, and hand it to a client that will accept it exactly as it accepts yours. Your index.txt, your issuance API logs and your ticketing system record what you issued. They say nothing about what exists. That is the whole difficulty: every certificate the CA ever issued becomes suspect, because you can no longer distinguish a certificate you issued from one somebody else minted, and no certificate is trustworthy merely because it appears in your records.

Two things bound the damage, if you put them in place before the incident. A CA certificate carrying nameConstraints restricts the names a client will accept beneath it, for every client that enforces the extension, which turns an unbounded compromise into a bounded one. And pathLenConstraint on the issuing CA prevents the attacker from minting a further CA beneath it and delegating the problem onwards. Neither helps if you learn about them during the incident.

The response, in order

flowchart TD
    A["Compromise confirmed"] --> B["1. Stop issuance"]
    B --> C["2. Preserve evidence"]
    C --> D["3. Assess scope"]
    D --> E["4. Revoke or distrust"]
    E --> F["5. Build replacement hierarchy"]
    F --> G["6. Reissue leaf certificates"]
    G --> H["7. Redistribute trust"]
    H --> I["8. Validate ecosystem recovery"]
    I --> J["9. Communicate"]

Stopping issuance comes first because it is this incident’s version of revoke or rotate: it strips the compromised authority of its ability to keep producing valid statements through your own pipeline. Do it by isolating the CA, disabling the issuing endpoint and pulling the credentials the automation uses, not by wiping the host.

Preserving evidence sits second precisely because the first step is so tempting to perform destructively. Image the host, copy the audit log, the serial file, the issuance index and the issued-certificate directory to write-once storage before anything else happens. If somebody rebuilds the CA host to get issuance working again, you lose the only record that could tell you when the key was taken.

Assessing scope means answering four questions: which trust stores hold this CA, which services validate against it, how many leaf certificates chain to it, and what constraints the CA certificate carries. Revocation or distrust follows. If the compromised CA is an intermediate with a parent you control, revoke it at the parent and publish the CRL. If it is a root, there is nothing above it to revoke at, and distrust means removal from every trust store, which is step seven wearing different clothes.

The replacement hierarchy uses new keys and a distinguishable subject name. Reusing the old subject name with a new key creates a situation where two different keys claim the same issuer identity, which makes chain building and CRL scoping ambiguous at the worst possible time. Then reissue every leaf, redistribute the new anchor, validate that the estate is actually working against it, and communicate to the people who depended on the old one.

Trust redistribution is the long pole

Building a new two-tier hierarchy takes an afternoon. Getting the new anchor into everything that must trust it is what makes this a multi-day operation, and occasionally a multi-week one. The estate usually contains operating system trust stores managed by configuration management, JVM trust stores managed by whoever packaged the application, container images that bake the anchor at build time, mobile devices, network appliances with a web form and no automation, partner systems on the other side of a contract, and embedded devices with no update path at all.

The transition is two-phase and the order is not negotiable. Add the new root everywhere first, verify it is present, reissue leaves onto the new hierarchy, verify again, and only then remove the old root. Removing first produces a fleet-wide validation failure that looks exactly like this, from a client that does not hold the anchor for the chain it was offered:

error 2 at 1 depth lookup: unable to get issuer certificate
curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)

Those two lines are the signature of an incomplete trust distribution, and during a CA transition they will arrive from the clients you forgot rather than from the ones you tested. Any client you cannot reach is a client you cannot cut over, so enumerate the unreachable population early and decide explicitly whether they get an exception, a manual visit, or a service outage.

If the compromised CA is publicly trusted, none of this timeline is yours. Root programme operators decide when distrust lands in their products, and the practical consequence is that you find out your certificates stopped working from your users.

Establishing what was issued, and what merely exists

You cannot enumerate what the attacker minted. You can enumerate what is being served, and compare it against what you meant to issue. Anything present in the estate but absent from your index is either a certificate somebody issued outside process, which is worth knowing anyway, or a certificate you did not issue at all.

# For each endpoint in the inventory, record the certificate that
# host is actually serving, keyed by issuer and serial.
INV=/var/incidents/ca-compromise/endpoints.txt
OUT=/var/incidents/ca-compromise/served-serials.txt
: > "$OUT"

while read -r host port; do
  printf '%s:%s ' "$host" "$port" >> "$OUT"
  openssl s_client -connect "$host:$port" -servername "$host" </dev/null 2>/dev/null |
    openssl x509 -noout -issuer -serial >> "$OUT"
done < "$INV"

Read the result against the issuance index. Certificates in the index but not in the estate are stale entries. Certificates in the estate but not in the index are the interesting ones. Neither list is complete, because a certificate minted by the attacker for a name they host on their own infrastructure will never appear in your scan, which is exactly why distrust of the whole CA, rather than revocation of a list of certificates, is the only response that closes the incident.

Production discipline

1. Constrain issuing CAs at creation time. Name constraints and a path length limit cost nothing on the day you build the hierarchy and convert an unbounded incident into a bounded one.
2. Keep the root offline and rehearse using it. A root you cannot reach in an emergency is not offline, it is lost, and a root you use weekly is not offline either.
3. Maintain a trust distribution inventory, not a trust store list. For every population that holds the anchor, record how a new anchor gets there and how long it takes. That table is your recovery time estimate.
4. Test the transition annually with a throwaway anchor. Push a dummy root through every distribution path and measure. The number you get is the honest recovery objective for this incident.

Cross-course references

• Kubernetes for Production Sysadmins - Part LXXVI (Cluster Certificates) covers a cluster CA that kubeadm does not support rotating out of the box, which makes this incident an architectural problem there rather than a procedural one.
• Linux for Production Sysadmins - Part XII (Repository Security and Supply Chain) covers the analogous failure where a signing key rather than a certificate authority key is the thing that grants unearned trust.
• Observability for Production Sysadmins - Part XX (Alert Quality) covers the difference between paging on a symptom and paging on a cause, which decides whether the fleet-wide validation failure during a botched transition reaches one engineer or fifty.

Quiz