Lab: Harden SSH on a host and validate the result

This lab applies a production-grade SSH hardening to a host and validates the result with external tools. By the end you will have a documented baseline for the host.

Objective

By the end of this lab, you can:

• Generate modern SSH host keys.
• Apply a hardened sshd_config.
• Validate the hardening with ssh-audit.
• Test that password authentication is rejected.

Architecture

You need:

• A host with sshd installed and running.
• A second host (or workstation) for testing.
• An existing SSH keypair for authentication.

Tasks

Task 1: Generate modern host keys

Do not stop sshd. Regenerating host keys does not require it, and the running daemon is your only way back in if something goes wrong. The listener loaded the old private keys into memory at startup and hands them to each connection it forks, so replacing the files on disk changes nothing until you reload in Task 3. That is exactly the safety margin you want.

# Timestamped so repeat runs never collide; -a preserves
# mode, owner and timestamps (host keys must stay 0600).
BACKUP="/etc/ssh.backup-$(date +%Y%m%d-%H%M%S)"
sudo cp -a /etc/ssh "$BACKUP"

# Prove the backup is real BEFORE removing anything.
if [ -f "$BACKUP/sshd_config" ] && ls "$BACKUP"/ssh_host_*_key >/dev/null 2>&1; then
  echo "Backup usable: $BACKUP"
  sudo rm -f /etc/ssh/ssh_host_*
else
  echo "Backup incomplete - stop here. Old keys left in place."
fi

Only continue once the backup was reported usable:

sudo ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -N ""
sudo ssh-keygen -t rsa -b 4096 -f /etc/ssh/ssh_host_rsa_key -N ""

Note the value of $BACKUP somewhere outside the session. You need it in Cleanup, and a shell variable does not survive a lost connection.

Task 2: Apply hardened sshd_config

Save the following to /etc/ssh/sshd_config:

Port 22
AddressFamily any
ListenAddress 0.0.0.0
ListenAddress ::

SyslogFacility AUTH
LogLevel VERBOSE

PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no
KbdInteractiveAuthentication no
UsePAM yes
AuthenticationMethods publickey
MaxAuthTries 3
MaxSessions 10
LoginGraceTime 30

KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com

HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key

UseDNS no

ClientAliveInterval 300
ClientAliveCountMax 2

AllowGroups ssh-users

Banner /etc/issue.net

Create the ssh-users group if missing:

# Substitute your own login before running:
TARGET_USER=alice

sudo groupadd ssh-users
sudo usermod -aG ssh-users "$TARGET_USER"

# Confirm BEFORE you restrict sshd to this group, or you lock yourself out
id -nG "$TARGET_USER" | tr ' ' '\n' | grep -qx ssh-users && echo 'membership ok'

Create the banner:

sudo tee /etc/issue.net <<EOF
WARNING: Authorised access only. All activity is monitored and recorded.
EOF

Task 3: Validate and reload

Validate first. sshd -t parses the config and loads the host keys without touching the running daemon, so a typo costs you nothing:

sudo sshd -t

Should exit with no errors. If it prints anything, fix the config and run it again. Do not reload on a config that fails this check.

sudo systemctl reload ssh 2>/dev/null || sudo systemctl reload sshd

Reload re-reads the config and keeps existing sessions alive, so your second terminal stays connected while you test the new rules.

Task 4: Re-trust the new host key, with verification

The host key has changed, so every client now refuses to connect. The response is a two-sided procedure: read the fingerprint on the server first, then match it on each client.

Step 1, on the server, over the console or IPMI - not over the SSH session you are about to invalidate. Record these fingerprints somewhere outside the session:

for k in /etc/ssh/ssh_host_*_key.pub; do sudo ssh-keygen -lf "$k"; done

256 SHA256:XkPz7...9aQ root@web01 (ED25519)
3072 SHA256:9mFa2...Rk4 root@web01 (RSA)

Step 2, on each client. Remove the stale entry first; appending a new key beside the old one still trips the mismatch check:

ssh-keygen -R <host>

Step 3, fetch the new key to a scratch file - not to known_hosts - and print its fingerprint:

# Substitute your own values before running:
HOST=server01.example.com

ssh-keyscan -t ed25519 "$HOST" > /tmp/new_hk
ssh-keygen -lf /tmp/new_hk

Step 4, compare that fingerprint, character by character, against the one you recorded in step 1. Only if they match:

cat /tmp/new_hk >> ~/.ssh/known_hosts
rm -f /tmp/new_hk

If they do not match, stop. Do not retry, do not connect. You are either talking to the wrong host or to something sitting between you and it.

Task 5: Test authentication

From the client:

# Should succeed (with valid key)
ssh user@host

# Should fail (no password allowed)
ssh -o PubkeyAuthentication=no user@host

# Should fail (root login not allowed)
ssh root@host

Verify each behaves as expected.

Task 6: Run ssh-audit

# Substitute your own values before running:
HOST=server01.example.com

# Prefer the distribution package
sudo apt install ssh-audit     # Debian/Ubuntu
# sudo dnf install ssh-audit   # RHEL/Fedora
# If it is not packaged: pipx install ssh-audit
# A bare `pip install` fails with externally-managed-environment
# on Debian 12+, Ubuntu 23.04+ and Fedora 38+ (PEP 668).

ssh-audit "$HOST"

Expected: A grade or A+ grade with no critical findings.

Task 7: External scan

nmap -sV -p 22 <host>

Expected: shows OpenSSH 9.x with the SSH-2 protocol.

ssh -vvv user@host 2>&1 | grep -E 'kex: |cipher: |mac:'

Expected: modern algorithms only (no diffie-hellman-group1, no aes-cbc, no hmac-md5).

Task 8: Document the baseline

SSH HARDENING BASELINE
======================
Host: <host>
Date: 2026-08-09

Algorithms:
- Kex: curve25519-sha256, curve25519-sha256@libssh.org, diffie-hellman-group16-sha512
- Ciphers: chacha20-poly1305@openssh.com, aes256-gcm@openssh.com, aes128-ctr
- MACs: hmac-sha2-512-etm@openssh.com, hmac-sha2-256-etm@openssh.com

Authentication:
- Publickey: yes
- Password: no
- Root login: no

Logging:
- LogLevel: VERBOSE
- fail2ban: enabled

Validation:
- ssh-audit grade: A
- nmap SSH version: OpenSSH 9.x

Save for future comparison.

Validation

• The hardened config is in /etc/ssh/sshd_config.
• sudo sshd -t exits clean.
• A second, freshly opened SSH session authenticates.
• Password auth is rejected.
• Root login is rejected.
• ssh-audit gives an A grade.
• Algorithms are modern only.
• The new host key fingerprint recorded on the server matches the one each client added to known_hosts.

Cleanup

Step 1. Choose the newest backup and prove it is usable. Nothing is deleted yet:

BACKUP=$(ls -1dt /etc/ssh.backup-* 2>/dev/null | head -n1)

if [ -n "$BACKUP" ] && [ -f "$BACKUP/sshd_config" ] && ls "$BACKUP"/ssh_host_*_key >/dev/null 2>&1; then
  echo "Usable backup: $BACKUP"
else
  echo "No usable backup. Do NOT touch /etc/ssh - fix the backup first."
fi

Step 2. Restore in place. rsync replaces the contents of /etc/ssh without the directory ever ceasing to exist. -a preserves mode, owner and timestamps, -A preserves ACLs, -X preserves extended attributes including SELinux labels, and --delete removes files the backup does not have:

sudo rsync -aAX --delete "$BACKUP"/ /etc/ssh/

If rsync is not installed, sudo cp -a "$BACKUP"/. /etc/ssh/ gets the file modes and ownership right, but it does not remove files the backup lacks. Delete any drop-ins the lab added, such as anything under /etc/ssh/sshd_config.d/, by hand afterwards. Do not add --preserve=context to that cp: it fails outright on a kernel without SELinux. Step 3 fixes the labels instead.

Step 3. Relabel on SELinux systems. Harmless where restorecon is absent:

command -v restorecon >/dev/null && sudo restorecon -R -v /etc/ssh

Step 4. Validate the restored config. Do not skip this:

sudo sshd -t && echo 'restored config OK' || echo 'INVALID - do not restart'

Step 5. Only if step 4 printed restored config OK, restart so the original host keys are served again:

sudo systemctl restart ssh 2>/dev/null || sudo systemctl restart sshd

Step 6. Open a brand-new SSH session from a second terminal and confirm it works before closing the session you have been working in. Clients still hold the lab host keys, so clear them once more:

ssh-keygen -R <host>

Step 7. Remove the backups only after the new session succeeded:

sudo rm -rf /etc/ssh.backup-*

What you learned

• A hardened sshd_config is multiple layers of defence.
• Validation tools catch regressions before they ship.
• Documenting the baseline lets future operators compare.
• sshd -t before every reload. It is the cheapest check you will ever run and the only one that runs before you are locked out.
• A revert that deletes before it can restore is not a revert. Prove the backup is usable, restore in place, then validate.
• A host key is re-trusted by comparing a fingerprint read out of band, never by appending whatever ssh-keyscan returns. accept-new does not help here: it refuses hosts whose key has changed.