Runbook: SSH access incident - lockout, brute force, key compromise

This runbook triages SSH access incidents. The three common cases are lockout, brute force, and key compromise. Each has a distinct procedure.

When to use this runbook

Use this runbook when:

• An operator is locked out of SSH.
• sshd logs show many failed authentication attempts.
• A private key is suspected to be compromised.
• An unauthorised login is detected.

Inputs

Gather before starting:

• The host name and IPs affected.
• Recent activity logs (auth log, sshd log).
• The list of users with active sessions.
• The list of keys in authorized_keys (suspect or all).
• For compromise: the compromised public key file, not only its fingerprint. The fingerprint identifies the key in logs; only the key itself matches what is written on disk.
• For lockouts: console or out-of-band access available.

Procedure: Lockout

A lockout means no one can SSH in. The most common causes:

• Wrong sshd_config (e.g. PermitRootLogin no but no admin user set up).
• Wrong authorized_keys permissions.
• Firewall change blocking the management network.
• Host key mismatch and StrictHostKeyChecking yes.

Step 1: Get console access

For cloud hosts:

AWS: EC2 > Instances > Connect > Serial Console or Session Manager
Azure: Virtual Machines > Serial Console
GCP: Compute Engine > VM > Serial Console

For on-prem: IPMI, iDRAC, iLO, hypervisor console.

Log in with break-glass credentials.

Step 2: Inspect sshd

sudo sshd -t                              # check config syntax
sudo journalctl -t sshd -t sshd-session -n 50 --no-pager   # recent logs
sudo cat /etc/ssh/sshd_config             # inspect config

Identify the wrong setting (most often an over-restrictive ACL or a typo in AllowGroups).

Step 3: Fix the config

For example, if AllowGroups is wrong:

sudo usermod -aG ssh-users <admin-user>
sudo systemctl reload sshd

Step 4: Verify from a known-good source

From your workstation or another host:

ssh user@host

Procedure: Brute force

sshd logs show many failed attempts from one or more sources.

Step 1: Identify the source

sudo journalctl -t sshd -t sshd-session --since "1 hour ago" --no-pager \
| grep -E 'Failed|Invalid' | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn

Output shows the top source IPs and attempt counts. Use -t sshd -t sshd-session, not -u sshd: on Debian-family hosts the unit form returns nothing at all, so the attacker IP is never extracted. See the note in the lockout procedure above.

Step 2: Verify fail2ban is active

sudo fail2ban-client status sshd

If fail2ban is not active or not configured, install it:

sudo apt install fail2ban
sudo systemctl enable --now fail2ban

Step 3: Add manual ban if needed

# Insert at the TOP of the chain - an appended rule sits behind the existing accepts
sudo nft insert rule inet filter input ip saddr <attacker-ip> counter drop

# iptables equivalent: -I with an explicit position, not -A
sudo iptables -I INPUT 1 -s <attacker-ip> -j DROP

A new drop rule does not affect sessions that are already established, because ct state established,related accept still matches them. Cut them explicitly:

sudo conntrack -D -s <attacker-ip>

Verify that the rule really is first and really is matching. The counter keyword is what makes this checkable:

sudo nft -a list chain inet filter input | head -5

The drop rule must appear before the SSH accept, and its counter must climb while the brute force continues. A rule that is present but shows zero packets after a minute of sustained attempts is a rule that is not being reached.

Step 4: Alert upstream if relevant

If the source IP belongs to a known attacker or a service you have a relationship with (e.g. a customer’s network), notify them.

Procedure: Key compromise

A private key has been disclosed. The impact depends on where the key is trusted.

Step 1: Confirm the compromise

Sources of confirmation:

• The user reports losing their laptop.
• An audit log shows the key used from an unexpected IP.
• The user shared the key accidentally.

Step 2: Identify the blast radius

First pin down the two identifiers you need. Take the public key from the incident report, or derive it from the private key:

ssh-keygen -y -f compromised_key > compromised_key.pub    # only if you hold the private key
FP=$(ssh-keygen -lf compromised_key.pub | awk '{print $2}')   # SHA256:... for comparison
BLOB=$(awk '{print $2}' compromised_key.pub)                  # base64 blob for matching

Next, on each host, find out where sshd actually reads authorised keys from. Do not assume ~/.ssh/authorized_keys:

ssh user@host "sudo sshd -T | grep -iE 'authorizedkeysfile|authorizedkeyscommand|trustedusercakeys|revokedkeys'"

Read the answer carefully:

• authorizedkeysfile may point somewhere central, such as /etc/ssh/authorized_keys/%u.
• authorizedkeyscommand other than none means keys come from a helper (LDAP, a key server, a bastion CA). Editing files on the host will not revoke anything.
• sshd -T does not evaluate Match blocks unless you give it a connection to match. Use sudo sshd -T -C user=deploy,host=web01,addr=10.0.0.5 for any account that a Match block covers, because a Match block can override AuthorizedKeysFile.

Then fingerprint every authorised key for every account on the host, not just the one you logged in as:

ssh user@host 'sudo bash -s' <<'EOF'
for akf in /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys; do
[ -f "$akf" ] || continue
printf '== %s\n' "$akf"
ssh-keygen -lf "$akf"
done
EOF

ssh-keygen -lf reads a whole authorized_keys file and prints one fingerprint per key, options prefix and all. Compare that output against $FP. A host is in the blast radius if the fingerprint appears anywhere in it.

Step 3: Revoke the key

Delete by the base64 blob, because that is what the file contains.

# Run once per affected host. $BLOB comes from Step 2.
ssh user@host "sudo bash -s -- '$BLOB'" <<'EOF'
set -eu
BLOB=$1
IR=/var/tmp/ssh-ir-$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p "$IR" && chmod 700 "$IR"
for akf in /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys; do
[ -f "$akf" ] || continue
grep -qF "$BLOB" "$akf" || continue
before=$(wc -l < "$akf")
cp -a "$akf" "$IR/$(printf '%s' "$akf" | tr / _)"
tmp=$(mktemp)
grep -vF "$BLOB" "$akf" > "$tmp" || true
cat "$tmp" > "$akf"
rm -f "$tmp"
printf 'removed %s line(s) from %s\n' "$((before - $(wc -l < "$akf")))" "$akf"
done
EOF

Three details in that loop are the difference between a revocation and a near miss:

• grep -F matches the blob literally. Base64 contains + and /, which a regex match would misread.
• The backup goes to /var/tmp, not next to the original. A copy of the compromised key left inside ~/.ssh is evidence parked in the blast radius, and a stray authorized_keys.bak invites someone to restore it later.
• cat "$tmp" > "$akf" rewrites the existing file in place, so owner and mode survive. Moving a root-owned temp file over a user’s authorized_keys leaves it root-owned, and sshd then refuses it - you have swapped a breach for a lockout.

Verify on the same host, in the same session, before you move on:

ssh user@host "sudo bash -s -- '$FP'" <<'EOF'
set -eu
FP=$1
rc=0
for akf in /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys; do
[ -f "$akf" ] || continue
ssh-keygen -lf "$akf" | grep -qF "$FP" || continue
printf 'STILL TRUSTED: %s\n' "$akf"
rc=1
done
exit $rc
EOF

Exit status 0 with no output is the only acceptable result. Record that result per host; a host you could not reach is a host that still trusts the key.

If Step 2 reported an authorizedkeyscommand, revoke at the source as well - the directory, the key server, or the identity provider that the helper queries. Files on the host are not authoritative there.

For CA-signed certificates, revoke through the key revocation list:

# -u merges into an existing KRL; omit it to create a new one
sudo ssh-keygen -k -f /etc/ssh/revoked_keys -u compromised_key.pub

# Confirm. -Q prints REVOKED and exits 1 for a revoked key - that is the pass condition
ssh-keygen -Q -f /etc/ssh/revoked_keys compromised_key.pub

If you only have the fingerprint from a log line and never held the key, a KRL specification can revoke by fingerprint. This is the one place a SHA256: string is valid input. Write the specification to revoke.spec:

hash: SHA256:4MIUFL4jGTgeFbz89DK/ipBxeoZNIA/ZZDer812G9G8

Then feed the specification to the same -k command:

sudo ssh-keygen -k -f /etc/ssh/revoked_keys -u revoke.spec
ssh-keygen -Q -l -f /etc/ssh/revoked_keys    # list what the KRL now revokes

Distribute the updated revoked_keys file to every host, then confirm each host is configured to consult it:

ssh user@host "sudo sshd -T | grep -i revokedkeys"

revokedkeys none means sshd ignores the list and the certificate still authenticates. Set RevokedKeys /etc/ssh/revoked_keys in sshd_config, run sudo sshd -t, then reload sshd.

Step 4: Generate a new key

The user generates a new SSH keypair:

ssh-keygen -t ed25519 -C "user@host - replaced <old-fingerprint>"

Distribute the new public key to every host that previously trusted the old key.

Step 5: Audit access

For each host, review recent activity from the compromised key:

sudo lastb -f /var/log/btmp    # bad logins
sudo journalctl -t sshd -t sshd-session --since "30 days ago" --no-pager | grep -F "$FP"

Here the fingerprint is the right thing to grep for. sshd logs the key it accepted - Accepted publickey for deploy from 203.0.113.9 port 51234 ssh2: ED25519 SHA256:... - so the log is the one artefact that carries the fingerprint. The file that grants the access never does. Keep the two straight: fingerprints for logs and comparisons, blobs for authorized_keys.

Look for unauthorised logins, command execution, or privilege escalation.

Step 6: Notify stakeholders

For confirmed compromise:

• Notify the security team.
• Notify any affected users (other users on the same hosts).
• Document the incident timeline.

Common patterns

Escalation

Escalate when:

• Console access is unavailable.
• Key compromise is confirmed (security incident).
• Brute force continues despite fail2ban.
• Same lockout recurs (process issue).

Bring: timeline, evidence, attempted fixes, current state.