← All runbooks in Secrets, PKI & Certificates

high risksecurity relevant~90 min

Runbook: Replace a Compromised Certificate and Key

1 · Prerequisites

Confirm every item is in place before any state change.

Authority to declare a security incident and to act outside the normal change window
A clean host on which to generate replacement key material, one that is not part of the suspected exposure
An issuing path that will sign out of hours, and the credentials to drive it that are not themselves protected by the exposed key
An out-of-band communication channel that does not depend on the service being replaced
Read access to the logs, backups, images and pipelines that may hold a copy of the key
A named incident scribe, because this procedure produces evidence that will be reviewed later
The ability to preserve system and access logs before any cleanup rotates them away

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Treat suspicion as confirmation. If the question is whether the key left the host, the answer for operational purposes is yes. A key cannot be un-copied, and waiting for proof costs exactly the time an attacker needs. Record the basis for the suspicion and proceed.
· Freeze the evidence before you change anything. Copy the relevant access logs, shell history, pipeline logs and file timestamps to a location outside the affected hosts. Rotation, redeployment and cleanup all destroy the record of how the key escaped.
· Establish where else this key is used. One key may terminate several hostnames, may be shared by a pair of load balancers, and may also be configured for client authentication to a partner. Every one of those uses is compromised, not just the one that raised the alarm.
· Confirm the replacement path works without the exposed key. If the CA credentials, the deployment pipeline or the bastion all authenticate with material that is part of this incident, you need a break-glass route before you start.
· Check whether any recorded traffic is at risk. Under TLS 1.3 the session keys come from an ephemeral exchange and the server key only signs the handshake transcript, so a stolen key does not decrypt captured sessions. If the estate still permits a non-ephemeral RSA key exchange anywhere, recorded traffic from those sessions is a separate exposure.
· Identify who has to be told and by when. A key compromise reason code is a public statement, and some contracts and regulations start a notification clock at the moment of discovery rather than at the moment of remediation.
· Confirm the certificate authority will accept a key compromise revocation from you. For some issuers this permanently blocklists the public key, which is desirable here and needs to be understood before it is irreversible.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Declare the incident and start a timeline. Note the discovery time, the reporter, and the first known point at which the key could have been read. The timeline is what turns a rotation into an investigation, and it cannot be reconstructed after the hosts are rebuilt.
2Preserve evidence before remediation touches it. Snapshot the affected hosts if you can, and in any case copy authentication logs, web server logs, deployment logs and file metadata off the host. Do this even though it delays the fix by minutes, because the alternative is never knowing the exposure window.
3Contain what can be contained immediately. Restrict inbound access to the affected service, tighten the network path, and disable any automation that would copy the key further. Containment is not the fix and does not replace the key, but it narrows what an attacker can do while you work.
4Generate the replacement key on a clean host. openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out app-incident.key run under umask 077. Do not generate on the host you suspect, and do not reuse a key from a staging environment on the theory that it was never public.
5Build a fresh signing request over the new key. Never resubmit the stored request from the previous issuance: it carries the exposed public key, and the certificate it produces would be a new wrapper around the compromised material.
6Have the certificate issued and prove it matches the new key. Compare the digest of the public key extracted from the new private key with the digest of the public key inside the issued certificate. Equal digests are the gate. Under incident pressure this is the check people skip and then spend twenty minutes debugging a listener that will not start.
7Verify the chain offline, then deploy and reload. Confirm the leaf verifies against the anchor the clients use, with the hostname and the server purpose checked, before the configuration is repointed. Deploy the full chain, not the leaf alone.
8Confirm from outside the host that the served key has changed. Read the public key digest off the wire and compare it against the value the exposed key produced. Until that digest changes, the compromised key is still the service identity regardless of what has been generated.
9Revoke the old certificate with a key compromise reason. Do this after the replacement is in service, not before. Revoking first leaves the compromised certificate in production with a record saying it is gone, which is the worst of both.
10Hunt every copy of the exposed key. Search backups, machine images, container layers, configuration management, artefact stores, ticket attachments, chat history and developer workstations. Record each location found and each one cleared. A key that survives in an image is a key that will be redeployed by the next rebuild.
11Look for the key in version control history, not just the working tree. A file deleted in a later commit is still present in the history and still readable by anyone with a clone. Removing it from the tip changes nothing about who already has it.
12Rotate anything the exposed key protected in turn. If the key authenticated to a partner, or unlocked another secret, or signed anything that grants access, those artefacts are inside the blast radius and need their own rotation with their own verification.
13Set up detection for continued use of the old identity. Watch certificate transparency for new certificates over the affected names, watch for the old serial being presented anywhere in the estate, and keep those watches running past the old certificate's expiry date.
14Close the incident with the residual risk written down. State plainly which client populations will keep accepting the old certificate, until when, and what compensating controls are in place until that date.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The public key digest observed from outside the host differs from the digest of the exposed key and equals the digest of the key generated on the clean host.
✓The full chain verifies from a client host with the hostname and the server purpose checked, using the trust anchor the real clients use rather than a locally assembled bundle.
✓The certificate authority confirms the revocation with the key compromise reason, and the serial appears in the published revocation data.
✓Every location identified in the key hunt is recorded with an outcome, and the ones still holding a copy have an owner and a date.
✓A search of version control history for the key material returns nothing, executed against a fresh clone rather than a working copy.
✓The evidence bundle collected before remediation is stored outside the affected hosts and is readable by the incident reviewer.
✓Detection for reuse of the old certificate and for new issuance over the affected names is active and has been tested with a known event.
✓The exposed private key is absent from every host, image, backup and pipeline that the inventory named, with the exception of the sealed evidence copy, which is recorded as such.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the replacement certificate fails to load, restore service with a second freshly generated key and a new certificate. Never restore the exposed pair, even briefly: it is the one thing this procedure exists to remove.
↶If the reload succeeds but clients fail validation, the chain is wrong rather than the key. Deploy the leaf together with its issuing certificate and reload again, and keep the exposed key out of the recovery path.
↶If a consumer that pins the old public key breaks, accept the outage for that consumer and coordinate the pin update. A pin to a compromised key is not a reason to keep the key in service.
↶If the revocation caused a relying party to hard-fail, restore the previous published revocation data while you correct it, and keep the new certificate deployed. The availability problem is in the revocation distribution, not in the replacement.
↶If evidence was destroyed before it was preserved, say so in the incident record. An honest gap is recoverable; a reconstructed timeline that later proves wrong is not.
↶If the replacement key is generated on a host that turns out to be part of the exposure, discard it and start again on ground you can defend. The cost is one more issuance.

6 · Escalation

When the runbook isn't enough, contact:

· The exposed key belongs to a certificate authority rather than a leaf: stop this procedure and escalate immediately, because every certificate that CA issued is now in question.
· The key was also used for client authentication to a third party, or to sign artefacts: escalate to the relationship owner and to the supply chain owner, since the blast radius extends outside your estate.
· The exposure window cannot be bounded, or the logs that would bound it are gone: escalate to the security lead, because the assumption then has to be the full lifetime of the key.
· A regulator, a customer contract or a certification scheme requires notification: escalate to legal and to the accountable executive within the clock those obligations define, not after remediation completes.
· Evidence appears that the attacker used the key rather than merely holding it: this is no longer a key replacement, it is an intrusion. Escalate to full incident response and preserve everything.

A private key that has been read by somebody else is compromised from the moment it was read, not from the moment it is used. Nothing you do afterwards makes the copy go away. The whole of this procedure is therefore about two things: making the copy useless as fast as possible, and being able to say afterwards how long it was useful for.

Understand precisely what an attacker gains. Holding your server private key lets them authenticate as your service to anyone who trusts the certificate. In an active network position that is impersonation, and it is complete. What it does not give them, provided the estate negotiates modern TLS, is the contents of sessions they recorded earlier: the session keys come from an ephemeral exchange and the long-term key only signs the handshake transcript. Recorded traffic is at risk only where a non-ephemeral RSA key exchange was permitted, which is worth establishing early because it changes the disclosure conversation entirely.

The order below is deliberate and it is not the order under pressure feels right. Evidence first, containment second, replacement third, revocation fourth, and the search for remaining copies fifth. Revoking early feels decisive and achieves nothing, because revocation does not remove a certificate from a running server and most clients will not notice it anyway.

When this runbook applies, and when it does not

It applies when a server private key has been exposed or may have been: it was committed to a repository, attached to a ticket, copied into a support bundle, present on a host that was breached, included in a backup that left the estate, or handled by somebody who should not have had it.

It does not apply when:

The rotation is planned and there is no exposure. Use the planned key rotation procedure. It is calmer, it is reversible for longer, and it does not spend an hour of the team collecting evidence that nobody will need.
The exposed key is a certificate authority key. Stop. Every certificate that CA has issued is in question and the response is a CA compromise procedure with executive involvement, not a leaf replacement.
The key is held in an HSM or KMS and never had an exportable form. Establish first whether an exportable copy ever existed. If it genuinely did not, what leaked is an access path rather than a key, and the response is to revoke that access.
Only the certificate leaked, not the key. A certificate is public. There is nothing to contain, and no reason to declare an incident over a file that any client can download by connecting.

Blast radius

Action	Reversible?	What it costs if wrong
Preserving logs and metadata	Yes	Minutes, and it is the one delay that repays itself
Restricting access to the service	Yes	Reduced availability while containment is in place
Generating a replacement key on a clean host	Yes	Nothing, and a second attempt is cheap
Deploying the replacement	Yes	A reload; a bad pair fails every new connection
Revoking with a key compromise reason	No	Some issuers permanently refuse that public key, for everyone
Rebuilding or reimaging the affected host	Only if you snapshotted first	The evidence of how the key escaped is gone permanently
Declaring the incident closed early	No	The residual exposure continues with nobody watching it

Step 1 - Freeze the evidence, then contain

INC="/var/incident/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$INC"

# Copy the record of how the key could have been read, before anything rotates it.
cp -a /var/log/auth.log* "$INC/" 2>/dev/null || true
cp -a /var/log/nginx/access.log* "$INC/" 2>/dev/null || true
stat /etc/ssl/private/app.key > "$INC/key-stat.txt"
sha256sum /etc/ssl/private/app.key > "$INC/key-digest.txt"
getfacl -p /etc/ssl/private > "$INC/keydir-acl.txt" 2>/dev/null || true

Record the digest of the key file rather than the key itself, so the incident bundle can prove which key was in service without carrying the secret. Timestamps and ownership on the key file often bound the exposure window better than any log does, and both are destroyed by the redeployment you are about to perform.

Step 2 - Generate the replacement on ground you can defend

umask 077
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 \
  -out /root/incident/app-incident.key

openssl req -new -key /root/incident/app-incident.key -sha256 \
  -subj "/CN=app.lab.example" \
  -addext "subjectAltName=DNS:app.lab.example,DNS:www.app.lab.example" \
  -out /root/incident/app-incident.csr

Generate on a host that is not part of the suspected exposure. A replacement key created on the compromised machine inherits the question you are trying to answer. Do not reach for the staging key either, however convenient: a key that has been sitting in a pipeline artefact for a year is not a clean key, it is an unexamined one.

Step 3 - Prove the certificate belongs to the new key

Read-only / SafeCaptured in the course lab: matching digests over the SubjectPublicKeyInfo— This is the gate before deployment. Under pressure it is the check people skip.

$ openssl pkey -in app.key -pubout | openssl sha256
openssl x509 -in app.crt -noout -pubkey | openssl sha256

SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b
SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b

Run the same two commands against the incident pair. The digest must also differ from the one recorded in step 1 for the exposed key. Two comparisons, both mandatory: the certificate matches the new key, and the new key is not the old key.

Step 4 - Deploy, reload, and read the wire

Service impact possibleDeploy the incident pair and confirm the served identity changed— Every new connection uses the replacement from the reload onward.

$ install -o root -g root -m 600 /root/incident/app-incident.key /etc/ssl/private/app.key
install -o root -g root -m 644 /root/incident/app-incident-fullchain.crt /etc/ssl/certs/app.crt
nginx -t
systemctl reload nginx
openssl s_client -connect app.lab.example:443 -servername app.lab.example </dev/null 2>/dev/null | openssl x509 -noout -pubkey | openssl sha256

Until the digest read from the network changes, the compromised key is still the identity of the service, whatever has been generated locally. This is the moment the incident clock stops for impersonation risk on this endpoint, and it belongs in the timeline to the second.

Step 5 - Revoke with a key compromise reason

openssl ca -config /etc/ssl/ca/ca.cnf -revoke /etc/ssl/certs/app-old.crt \
  -crl_reason keyCompromise
openssl ca -config /etc/ssl/ca/ca.cnf -gencrl -out /srv/pki/crl.pem
openssl crl -in /srv/pki/crl.pem -noout -lastupdate -nextupdate

Revocation comes after the replacement is serving, never before. Be aware that a key compromise revocation is a strong and durable statement: some issuers will refuse to sign that public key again for anybody, which is exactly the behaviour you want and also the reason to be certain you are naming the right certificate.

Step 6 - Hunt every copy of the exposed key

# The working tree is not the question. History is.
git -C /srv/repos/app.git log --all --full-history --name-only -- '*.key' '*.pem'
git -C /srv/repos/app.git rev-list --all | head -50

# Images, artefacts and backups hold copies that a rebuild will redeploy.
find /var/lib/backups -name '*.key' -newermt '2026-01-01' -print
grep -rl 'BEGIN PRIVATE KEY' /srv/artifacts 2>/dev/null | head -20

Each location found gets an owner and a disposition. A key deleted from the tip of a branch is still in the history and still readable by anyone with a clone, so removing the file is not the remediation for a repository, rewriting or rotating is. A key baked into a machine image will be redeployed by the next scale-out event, which is how a closed incident reopens six weeks later.

Step 7 - Watch, and write down what is still true

Keep two detections running past the old certificate’s expiry: one for the old serial being presented anywhere in the estate, and one on certificate transparency for new issuance over the affected names. Then record the residual risk plainly. Most clients will continue to accept the revoked certificate until its notAfter passes, so that date, and not the revocation timestamp, is when the exposure ends for them. An incident closed without that sentence will be reopened by the first person who reads the timeline carefully.

Common pitfalls

Symptom	Cause	Action
The listener will not start after deployment	The certificate does not match the deployed key	Repeat the digest comparison; deploy only a matching pair
The served digest is unchanged after the reload	The configuration still names the old file	Re-read the effective configuration and reload again
The incident report cannot bound the exposure window	The host was rebuilt before evidence was copied	Record the gap honestly and assume the full key lifetime
The key reappears after a rebuild	A copy survives in a machine image or artefact	Rebuild the image from a clean base and rescan the artefact store
The CA refuses the key compromise revocation	The authenticating account did not order the certificate	Revoke using the certificate private key, or escalate to the PKI owner
A partner integration fails immediately	The same key authenticated to that partner	Treat that credential as compromised too and rotate it with its owner

Verification

The replacement is real when the public key digest observed from outside the host differs from the exposed key and matches the key generated on the clean host, and the full chain verifies from a client position with the hostname and the server purpose checked. The issuer confirms the revocation with a key compromise reason and the serial appears in the published revocation data. Every location the key hunt identified carries an outcome, a history search against a fresh clone returns nothing, and the evidence bundle sits outside the affected hosts. Detections for reuse of the old certificate and for new issuance over the affected names are active and tested, and the exposed key is absent everywhere the inventory named apart from the sealed evidence copy, which is recorded as such.

Rollback

There is no path back to the exposed pair. If the replacement certificate fails to load, generate a second key and issue again; restoring the compromised key even for five minutes undoes the only thing this procedure achieves. If the reload succeeds but clients fail validation, the fault is in the chain rather than the key, so deploy the leaf with its issuing certificate and reload. If a pinned consumer breaks, accept that outage and coordinate the pin update. If the revocation caused a relying party to hard-fail, restore the previous published revocation data while correcting it and leave the new certificate deployed. If evidence was lost before it was preserved, record the gap rather than reconstructing a timeline you cannot support. If the replacement key was generated inside the exposure, discard it and repeat step 2 on clean ground.