← All runbooks in Secrets, PKI & Certificates

high risksecurity relevant~90 min

Runbook: Respond to a Compromised SSH Key

1 · Prerequisites

Confirm every item is in place before any state change.

Root or equivalent administrative access on every host that trusts the key, whether directly or through a certificate authority
A second authenticated session, a console or an out-of-band management path on each host, because this procedure edits the configuration of the service you are connected through
The authoritative record of everything previously revoked, since a revocation list rebuilt without it silently reinstates old revocations
An inventory of hosts that lists which of them trust the user certificate authority and which still accept raw keys
Read access to the authentication logs on those hosts, covering the whole period from suspected exposure onward
Authority to disable the account the key authenticated as, if the audit shows it was used

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Fingerprint the key and work from that, never from the file name. ssh-keygen -l -f key.pub prints the fingerprint that both the daemon and the revocation list match on. Two files with different names are frequently the same key, and one file name is frequently reused for several keys.
· Determine whether a certificate was issued from this key. ssh-keygen -L -f cert.pub prints the serial, the key identity, the principals, the validity interval and the extensions. The serial is what a serial based revocation entry names.
· Establish which trust path each host uses. A host with TrustedUserCAKeys accepts certificates; a host with an ordinary authorised keys file accepts raw keys. Many estates have both, and containing only one of them leaves the door open.
· Establish the earliest moment the key could have left its host, and state the basis. A backup restored to the wrong place, a laptop reported stolen, a key copied into a build image and a key committed to a repository all give different starting timestamps.
· Check whether agent forwarding was in use. A forwarded agent lets an intermediate host use the key while the session is open, which widens the set of places the key could have been used from.
· Confirm you hold a second way in to every host before editing any configuration. This procedure changes the daemon that your session depends on, and a validation failure with no second path is a physical visit.
· Record the current revocation list and daemon configuration. Copy both aside before the first edit. They are the rollback, and they are also the evidence of what the estate looked like beforehand.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Revoke first, by adding the key or its certificate serial to the key revocation list. SSH certificates carry one signature and one level: there is no chain to break, no online status responder and no certificate revocation list. The revocation list, or a plain list of keys, is the only mechanism available, and nothing else you do contains the key.
2Rebuild the revocation list from the full historical record, not from this incident alone. The list is a file. Regenerating it with only today's entry makes every previously revoked key acceptable again, which is a silent and complete reversal of past incident work.
3Raise the revocation list version each time you rebuild it. The version number is how hosts and reviewers tell one generation of the list from another during distribution.
4Revoke the raw key as well as any certificate. Updating the list with the public key covers the case where the same key pair is also present in an authorised keys file somewhere in the estate.
5Distribute the list to every host, then make the daemon read it. RevokedKeys in the daemon configuration names the file. A list that exists on only some hosts is a partial containment that reads as a complete one.
6Validate the daemon configuration before reloading it. sshd -t parses the file without applying it. Reloading a configuration with a syntax error, from your only session, is how this procedure turns into a console visit.
7Prove the refusal from the server side, not from the client. ssh-keygen -Q -f answers directly for a given key, and the daemon log records the refusal naming the revocation file. A client seeing permission denied proves only that something refused it.
8Sweep every authorised keys file on every host. Match on the fingerprint rather than on the text of the key, and read the daemon configuration first: authorised keys can be read from outside home directories, and a command can supply them dynamically.
9Sweep the trust points that are not files on your hosts. Deployment keys on the code hosting platform, keys registered with cloud providers, keys held in continuous integration credential stores and keys embedded in container images are all separate places the same key is trusted.
10Answer what the key could actually reach. Read the accounts it authenticated as, the privilege those accounts held, and the hosts that trusted it. The blast radius of an SSH key is the union of everything reachable from every account it opened.
11Read the authentication logs across the exposure window. Since version 10.0 the daemon is split into sshd, sshd-session and sshd-auth, so a log search that matches only the first name will miss lines.
12Generate the replacement on the machine that will use it, and never copy a private key between hosts. A key that has travelled has been exposed to every hop it travelled through.
13Sign the replacement with an explicit validity interval. Without one the issued certificate is valid from the Unix epoch into the distant future, which turns a short lived credential into a permanent one.
14Give the replacement explicit principals. From version 10.3 an empty principal list matches nothing rather than everything, so a certificate that used to work everywhere will now work nowhere, and a certificate issued on older guidance may be doing the opposite of what its author intended.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The revocation list answers that the compromised key and its certificate are revoked, checked on a host other than the one where the list was built.
✓A live authentication attempt with the compromised key is refused, and the daemon log on the target host records the refusal naming the revocation file.
✓The same check performed on a randomly chosen host from the inventory gives the same answer, which is what proves distribution rather than local success.
✓A fingerprint sweep of every authorised keys file across the inventory returns no match, including files named by a non-default authorised keys setting.
✓The replacement key authenticates normally and its certificate shows an explicit validity interval and explicit principals when inspected.
✓The authentication logs for the exposure window have been read on every host that trusted the key, and every accepted session in that window is attributed to a known operator.
✓The historical revocation record contains every entry that was present before this incident plus the new one, confirmed by comparing entry counts.
✓The daemon configuration on each host names the revocation file, and the daemon is running with that configuration rather than merely holding it on disk.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Revocation is not reversed. If the wrong key was revoked, issue its owner a new key rather than removing the entry, because a list that has been published and then relaxed is no longer trustworthy.
↶If the daemon fails to start after the configuration change, restore the configuration copy taken during the pre-checks and reload through the second session or the console. Never leave the daemon stopped while you investigate.
↶If the revocation list was rebuilt without the historical entries, restore the copy taken during the pre-checks, merge the new entry into the full record, and rebuild once from that record.
↶If a sweep removed an authorised keys entry that was still needed, restore it from the copy taken before the edit, and confirm through the second session before closing the first.
↶If the replacement certificate is refused, inspect it before reissuing: the usual causes are an empty principal list, a validity interval that has not started, and a clock difference between the issuing host and the target.
↶If distribution reached only part of the estate, do not roll back the hosts that received it. Complete the distribution instead, and treat the partial state as an open incident until it is finished.

6 · Escalation

When the runbook isn't enough, contact:

· The authentication logs show a session you cannot attribute to a known operator: this is an intrusion, and the security incident responder takes over while this procedure continues.
· The compromised key is a certificate authority key rather than a user or host key: containment through the revocation list is not sufficient, and the far larger authority compromise procedure applies instead.
· The key opened an account with unrestricted privilege escalation on any host: assume the host is compromised rather than merely accessed, and hand it to whoever owns host rebuilds.
· Agent forwarding was in use and an intermediate host is not fully trusted: every host reachable from that intermediate is in scope, and scoping that belongs to the platform owner rather than to the responder.
· A host cannot receive the revocation list because it is unreachable or unmanaged: it remains an open exposure. Escalate it as a named risk with an owner rather than closing the incident around it.
· The estate still accepts raw keys with no revocation list configured at all: that is a structural gap rather than an incident finding, and it needs an owner and a date.

An SSH private key is a bearer credential with no expiry unless someone gave it one. Copying it is undetectable, using it looks exactly like the legitimate owner using it, and the key itself carries no record of where it has been. The only durable facts you have are the fingerprint, the hosts that trust it and the authentication logs on those hosts.

SSH revocation is unlike certificate revocation everywhere else in this course. There is no chain, no intermediate, no online status responder and no certificate revocation list. A certificate carries exactly one signature from exactly one authority, and the only revocation mechanism is a key revocation list, or a plain list of keys, that each daemon is configured to consult. That makes revocation reliable, because nothing soft fails, and makes distribution the entire problem: a host that never received the list is a host where nothing was revoked.

The order below reflects that. Revoke, distribute, prove refusal from the server logs, and only then sweep the files. Sweeping first feels productive and leaves every certificate issued from the key still valid on every host you have not reached yet.

When this runbook applies, and when it does not

It applies when an SSH private key, or a certificate issued from one, is believed to be held by anyone other than its intended holder: a lost or stolen device, a key found in a repository or an image, a key copied to a shared host, or a key whose passphrase was disclosed.

It does not apply when:

The key is a certificate authority key. Revoking one certificate does nothing about an authority that can mint more of them. That incident is far larger and has its own procedure.
Only the public key was exposed. Public keys are published by design, sit in authorised keys files and are printed in logs. Confirm which half you are holding before taking an outage.
A host key changed unexpectedly. That is a host identity problem and a possible interception, and the response is to establish what the host key should be, not to revoke a user credential.
The key never had access to anything. A key present in no authorised keys file and signed by no authority is a hygiene finding. Confirm that with a sweep rather than assuming it.

Blast radius

Action	Reversible?	What it costs if wrong
Adding an entry to the revocation list	In principle, by rebuilding without it	Every authentication using that key stops. Sessions already open are unaffected, which is why you keep one.
Rebuilding the list from scratch	Only if you kept a copy	Every previously revoked key becomes acceptable again, silently, across the whole estate.
Editing the daemon configuration and reloading	Only if you kept a copy and still hold a session	A parse failure with no second path in means a console or out-of-band visit to every affected host.
Removing entries from an authorised keys file	Only if you kept a copy	You can remove your own access, on the host you are removing it from.
Rotating the user certificate authority key	No	Every certificate that authority ever signed stops being accepted at once, everywhere.
Disabling the account the key opened	Yes	Automation running as that account stops. Weigh it against evidence of misuse rather than doing it reflexively.

Step 1 - Identify the key and the trust path

Read-only / Safeon the responder workstation— Record the fingerprint and, if a certificate exists, its serial and principals.

$ INCIDENT=/run/incident
install -d -m 0700 "$INCIDENT"

# The fingerprint is what the daemon and the revocation list match on.
ssh-keygen -l -f "$INCIDENT/compromised.pub" | tee "$INCIDENT/fingerprint"

# If a certificate was issued from this key, read it in full. The
# serial is what a serial based revocation entry names.
ssh-keygen -L -f "$INCIDENT/compromised-cert.pub" \
| tee "$INCIDENT/certificate"

The certificate output names the signing authority, the key identity, the serial, the validity interval and the principals. Read the validity interval carefully: a certificate signed without an explicit interval runs from the Unix epoch into the distant future, so a credential the team believes is short lived may in fact never expire. Read the principals as carefully: from version 10.3 an empty principal list matches nothing, where older releases treated it as matching anything.

Step 2 - Revoke, from the full historical record

Configuration changeon the host that owns the revocation list— Append to the historical record, rebuild the list from it, then add the raw key.

$ INCIDENT=/run/incident
KRL=/etc/ssh/revoked.krl
CA=/etc/ssh/user_ca.pub
RECORD=/etc/ssh/revocations.txt

# Keep a copy of both before changing either.
cp -a "$KRL" "$INCIDENT/revoked.krl.before"
cp -a "$RECORD" "$INCIDENT/revocations.txt.before"

# Add this incident to the record, then rebuild from the whole record.
cat "$INCIDENT/serials-to-revoke.txt" >> "$RECORD"
ssh-keygen -k -f "$KRL" -s "$CA" -z 2 "$RECORD"

# A raw key with no certificate is revoked by naming its public key.
ssh-keygen -k -f "$KRL" -u "$INCIDENT/compromised.pub"

The record file holds one specification per line, in the form the revocation list accepts, such as a serial entry naming the certificate serial. Rebuilding from that record rather than from the incident is the whole point of keeping it: a list regenerated with today’s entry alone quietly reinstates every key any previous incident revoked, and nothing in the output tells you that it happened.

Step 3 - Distribute the list and make the daemon read it

Service impact possibleon each host, with a second session open— Confirm the daemon is told where the list is, validate, then reload.

$ # The daemon consults the list only if it is configured to.
grep -n '^RevokedKeys' /etc/ssh/sshd_config

# Parse the configuration without applying it. Do this every time.
/usr/sbin/sshd -t

# Reload the unit that runs the daemon: it is named ssh on
# Debian family systems and sshd on Red Hat family systems.
systemctl reload ssh
systemctl is-active ssh

Distribution is the control, so treat it as one. Push the list with the same configuration management that owns the rest of the daemon configuration, record which hosts have which list version, and treat any host that did not converge as an open exposure rather than a tidying task. Keep the second session open until the reload has completed and you have authenticated afresh through a new connection.

Step 4 - Prove refusal from the server side

Read-only / Safeon a target host, after the reload— Ask the revocation list directly, then read the daemon log for the refusal.

$ INCIDENT=/run/incident
KRL=/etc/ssh/revoked.krl

# The list answers directly for a given key or certificate file.
ssh-keygen -Q -f "$KRL" "$INCIDENT/compromised-cert.pub"

# Then attempt the authentication and read the server, not the client.
# Since version 10.0 the daemon is three binaries, so match all names.
journalctl --since "$(cat "$INCIDENT/revoked-at")" \
-t sshd -t sshd-session -t sshd-auth \
| grep -E 'revoked by file|Certificate invalid|Accepted'

A revoked certificate reports as revoked, and the daemon records a line stating that the authentication key was revoked by the named file. On the client the same event appears only as Permission denied, which is the identical message produced by a certificate that expired, by a principal that is not listed, and by an account that is locked in the password database. Only the server log distinguishes them, which is why this step reads the server.

Step 5 - Sweep every place the key is trusted

Read-only / Safeon each host in the inventory— Match authorised keys on fingerprint, after reading where they are read from.

$ INCIDENT=/run/incident
FP=$(awk '{print $2}' "$INCIDENT/fingerprint")

# Authorised keys are not always in home directories, and may be
# produced by a command rather than read from a file.
grep -n -E '^(AuthorizedKeysFile|AuthorizedKeysCommand)' /etc/ssh/sshd_config

for f in /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys \
       /etc/ssh/authorized_keys.d/*; do
[ -r "$f" ] || continue
ssh-keygen -l -f "$f" 2>/dev/null | grep -F "$FP" >/dev/null \
  && echo "match: $f"
done

Fingerprint matching is what makes this reliable: the same key reformatted, re-commented or wrapped differently is textually different and cryptographically identical. Then leave the hosts and sweep the trust points that are not files you own: deployment keys registered on the code hosting platform, keys registered with cloud providers, keys held in pipeline credential stores, and keys baked into container images. Each of those is a place the key is trusted that no host sweep will ever find.

Step 6 - Answer what the key could reach

Read-only / Safeon each host that trusted the key— Attribute every accepted session in the window, then read the account privilege.

$ INCIDENT=/run/incident
FP=$(awk '{print $2}' "$INCIDENT/fingerprint")
FROM=$(cat "$INCIDENT/exposed-at")

# Sessions accepted with this key, in the exposure window.
journalctl --since "$FROM" -t sshd -t sshd-session -t sshd-auth \
| grep -F "$FP"

# Verbose logging records the certificate identity and serial next to
# the accepted key, which is what attributes a session to a person.
grep -n '^LogLevel' /etc/ssh/sshd_config

# What the account could do once the session was open.
getent passwd deploy
grep -rn 'deploy' /etc/sudoers /etc/sudoers.d 2>/dev/null

The blast radius of an SSH key is the union of everything reachable from every account it opened, on every host that trusted it, including onward connections made from those hosts. Write that set down explicitly rather than describing it as the hosts the key was for. If any account it opened could escalate to root without constraint, treat the host as compromised rather than merely accessed, and hand it to whoever owns rebuilds.

Step 7 - Replace the key and remove the standing raw key

Configuration changeon the machine that will use the replacement— Generate locally, then sign with explicit validity and explicit principals.

$ INCIDENT=/run/incident

# Generate where it will be used. A private key that has travelled has
# been exposed to every hop it travelled through.
ssh-keygen -t ed25519 -f "$INCIDENT/replacement" \
-C 'deploy at web-01, reissued during incident'

# Sign with an explicit interval and explicit principals. Without the
# interval the certificate runs from the Unix epoch into the future.
ssh-keygen -s /etc/ssh/user_ca -I 'deploy@web-01' -n deploy \
-V -5m:+8h -z 1102 "$INCIDENT/replacement.pub"

ssh-keygen -L -f "$INCIDENT/replacement-cert.pub"

Read the issued certificate back before distributing it. The three fields that matter are the validity interval, which should be hours rather than years, the principal list, which must name the accounts the holder is entitled to, and the serial, which is what a future revocation will name. A certificate whose serial you did not record is one you cannot revoke by serial later.

Common pitfalls

Symptom	Cause	Action
The key still authenticates after the list was updated	The daemon was not reloaded, or its configuration names a different file	Read the configured path, validate, reload, then re-test from the server side
Permission denied with no certificate message in the log	The target account is locked in the password database, which refuses key authentication with no certificate error	Check the account state before continuing to chase the certificate
Revoked on one host, accepted on another	The list reached only part of the estate	Treat distribution as the control, record list versions per host, and converge the rest
A certificate with no principals is accepted on an old host	Before version 10.3 an empty principal list matched anything; from 10.3 it matches nothing	Reissue with explicit principals and bring the old hosts forward
A certificate is valid for decades	It was signed with no validity interval, so it runs from the Unix epoch	Revoke that serial and reissue with an explicit interval
Log searches return nothing on a recent host	Since version 10.0 the daemon is split into three binaries and the lines may carry any of the three names	Match all three identifiers when searching
Old revocations stopped working after this incident	The list was rebuilt from the incident rather than from the historical record	Restore the previous list, merge, and rebuild once from the full record

Verification

Ask the revocation list directly on a host other than the one that built it, and confirm both the certificate and the raw key report as revoked. Attempt a live authentication and read the daemon log on the target, confirming the refusal names the revocation file rather than inferring it from the client message. Repeat that check on a host chosen at random from the inventory, because a success on the host you configured proves configuration rather than distribution. Sweep every authorised keys file across the inventory by fingerprint, including any file named by a non default setting, and confirm no match remains. Authenticate with the replacement and inspect its certificate, confirming an explicit validity interval and explicit principals. Read the authentication logs for the window on every host that trusted the key and attribute each accepted session to a known operator. Finally, compare entry counts between the previous historical record and the current one to confirm nothing was lost in the rebuild.

Rollback

Revocation is not reversed. If the wrong key was revoked, issue that person a new key rather than removing the entry, because a published list that is later relaxed cannot be trusted by anyone reading it. If the daemon fails to start after the configuration change, restore the configuration copy taken during the pre-checks and reload through the second session or the console, never leaving the daemon stopped while you investigate. If the list was rebuilt without its history, restore the copy taken during the pre-checks, merge the new entry into the full record and rebuild once from that record. If a sweep removed an entry that was still needed, restore it from the pre-edit copy and confirm through the second session before closing the first. If the replacement certificate is refused, inspect it before reissuing: the usual causes are an empty principal list, a validity interval that has not started yet, and a clock difference between the issuing host and the target. If distribution reached only part of the estate, complete it rather than rolling back the hosts that already have it.