Lab 16: Configure SSH public-key authentication correctly

Objective

By the end of this lab you will have generated an Ed25519 key pair, installed its public half into a disposable server’s authorized_keys, authenticated with it, and then broken the setup in the two ways that account for most real support tickets: wrong file modes, and a right-looking key whose private half you do not hold.

The point is not that you can log in. Most engineers can already do that. The point is that you will be able to answer, from evidence rather than from memory, what the server checked before it let you in. A public key sitting in authorized_keys grants nothing on its own. It is a lookup table entry. What authenticates is a signature the client produces over data that is unique to this one connection, which the server verifies with that stored public key.

You will also see why the server’s refusal is deliberately uninformative to the client, and where the actual reason lives.

Architecture

One disposable Alpine container runs sshd on a loopback-bound port. Everything else stays on your host: the key pair, the client configuration, and a lab-local known_hosts file so that nothing this lab does touches your own ~/.ssh.

flowchart LR
    K["Ed25519 private key\nyour host, mode 0600"] --> C["ssh client signs\nsession-specific data"]
    P["Public key installed as\n/home/deploy/.ssh/authorized_keys"] --> D["sshd: look up the key,\nthen verify the signature"]
    C --> D
    D --> A["Accepted publickey for deploy"]
    D --> R["Permission denied\npublickey, keyboard-interactive"]

The two arrows into sshd are the whole lesson. One carries a stored public key that anybody may read and copy. The other carries a fresh signature that only the holder of the private key could have produced. Both must line up, and only the second one is a secret-dependent operation.

Requirements

• OpenSSH 10.x client on the host, with ssh, ssh-keygen and ssh-keyscan on the PATH. Anything from 8.2 onward will follow the steps; the algorithm listing in Task 2 is the 10.x one.
• Docker 29.x, able to pull alpine:3.22 and to publish a port on 127.0.0.1.
• About 50 MB of disk and a working directory under $HOME.
• No out-of-band access requirement. This lab never touches your host’s sshd, your ~/.ssh directory, your firewall or your primary interface. The only SSH server involved is inside a container, reachable only on 127.0.0.1:12216.

Scenario

You have joined a team that has been handing round a single shared SSH key for two years. The key has no passphrase, its private half is in three people’s home directories and one CI runner, and nobody can say who generated it. Before you can argue for replacing it, you need to be able to state precisely what that key does and does not prove, and to demonstrate the mechanics on something disposable rather than on production.

Tasks

Task 1 — Record the starting state

LAB="$HOME/rbpki-lab-16"
rm -rf "$LAB"
mkdir -p "$LAB/client" "$LAB/server"
cd "$LAB"

# Record what Cleanup must restore: no rbpki- container, no rbpki- network.
{
  echo "containers:"
  docker ps -a --filter name=rbpki- --format '{{.Names}}'
  echo "networks:"
  docker network ls --filter name=rbpki- --format '{{.Name}}'
} > "$LAB/state.pre-lab"
cat "$LAB/state.pre-lab"

Both lists should be empty. If either is not, you already have resources using the rbpki- prefix and you must rename this lab’s resources before continuing, or you will delete something you meant to keep.

Task 2 — Ask the client what it can actually do

Never take an algorithm list from documentation when the binary in front of you will tell you the truth. ssh -Q queries the build you are holding.

$ ssh -Q key

ssh-ed25519
ssh-ed25519-cert-v01@openssh.com
sk-ssh-ed25519@openssh.com
sk-ssh-ed25519-cert-v01@openssh.com
ecdsa-sha2-nistp256
ecdsa-sha2-nistp256-cert-v01@openssh.com
ecdsa-sha2-nistp384
ecdsa-sha2-nistp384-cert-v01@openssh.com
ecdsa-sha2-nistp521
ecdsa-sha2-nistp521-cert-v01@openssh.com
sk-ecdsa-sha2-nistp256@openssh.com
sk-ecdsa-sha2-nistp256-cert-v01@openssh.com
ssh-rsa
ssh-rsa-cert-v01@openssh.com

There is no ssh-dss in that list, and no amount of configuration will bring it back. OpenSSH 10.0 removed DSA support outright, finishing a deprecation that began in 2015; the --enable-dsa-keys build option and the DSAKEY make variable are both gone. If a runbook in your estate still tells someone to generate a DSA key, that runbook is now a dead end rather than a weak default.

Now ask the same build about signature algorithms, which are a different list.

$ ssh -Q key-sig

ssh-ed25519
sk-ssh-ed25519@openssh.com
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
sk-ecdsa-sha2-nistp256@openssh.com
webauthn-sk-ecdsa-sha2-nistp256@openssh.com
ssh-rsa
rsa-sha2-256
rsa-sha2-512

That output is trimmed to the plain forms; the real listing interleaves a -cert-v01@openssh.com variant after most entries. The important observation is that ssh-rsa appears in both lists, meaning two different things. As a key type it is an RSA key and is perfectly fine. As a signature algorithm it means RSA with SHA-1, which you should never negotiate. rsa-sha2-512 and rsa-sha2-256 are already the RSA defaults in HostKeyAlgorithms, CASignatureAlgorithms and PubkeyAcceptedAlgorithms, so this is not something you enable. It is something you avoid undoing.

Save both listings, because they are the first deliverable.

LAB="$HOME/rbpki-lab-16"
{
  ssh -V 2>&1
  echo "--- ssh -Q key ---"
  ssh -Q key
  echo "--- ssh -Q key-sig ---"
  ssh -Q key-sig
} > "$LAB/ssh-algorithms.txt"
grep -c ssh-dss "$LAB/ssh-algorithms.txt" || echo "ssh-dss absent, as expected"

Task 3 — Generate the key pair

LAB="$HOME/rbpki-lab-16"
ssh-keygen -t ed25519 -f "$LAB/client/rbpki16" -C "rbpki16-lab-key" -N ""
ls -l "$LAB/client"

ssh-keygen prints five lines: what it generated, where the private key went, where the public key went, the SHA-256 fingerprint, and a block of randomart. The randomart is a visual aid for humans comparing fingerprints by eye and carries no information the fingerprint line does not. The fingerprint is what matters, because that string is what every sshd log line about this key will contain.

-N "" sets an empty passphrase. That is correct for a disposable lab key and wrong for anything else, which is the subject of the callout below.

Look at the modes ssh-keygen chose: the private key is 0600 and the public key is 0644. Those defaults exist because the client refuses to use a private key that other users can read.

Task 4 — Start the disposable server

Write the server’s configuration and start script first, so that every setting is visible in your working directory rather than buried in a container command line.

LAB="$HOME/rbpki-lab-16"

cat > "$LAB/server/sshd_config" <<'SSHDCONF'
Port 22
PubkeyAuthentication yes
PasswordAuthentication no
AuthorizedKeysFile .ssh/authorized_keys
StrictModes yes
LogLevel VERBOSE
PidFile /tmp/sshd.pid
SSHDCONF

cat > "$LAB/server/start.sh" <<'START'
#!/bin/sh
set -e
apk add --no-cache openssh-server >/dev/null
adduser -D deploy
passwd -u deploy
ssh-keygen -A >/dev/null
install -d -m 0700 -o deploy -g deploy /home/deploy/.ssh
cp /etc/ssh/lab/sshd_config /tmp/sshd_config
exec /usr/sbin/sshd -D -f /tmp/sshd_config -e
START

docker network create rbpki-net-16
docker run -d --name rbpki-ssh16 --network rbpki-net-16 \
  -p 127.0.0.1:12216:22 \
  -v "$LAB/server:/etc/ssh/lab:ro" \
  alpine:3.22 sh /etc/ssh/lab/start.sh
sleep 8
docker ps --filter name=rbpki-ssh16 --format '{{.Names}} {{.Status}}'

The passwd -u deploy line is not decoration. On Alpine, adduser -D creates the account with no password, which BusyBox records in /etc/shadow as a locked account, and sshd refuses public-key authentication for a locked account. Skipping that one line produces an ordinary Permission denied with no hint anywhere that the account, rather than the key, is the problem. You will meet that failure again in Lab 17.

LogLevel VERBOSE is what makes the rest of this lab possible. At the default level sshd does not log the fingerprint of the key it accepted or rejected.

Task 5 — Install the public key, correctly

LAB="$HOME/rbpki-lab-16"

docker exec -i rbpki-ssh16 sh -c 'cat > /home/deploy/.ssh/authorized_keys' \
  < "$LAB/client/rbpki16.pub"
docker exec rbpki-ssh16 chown deploy:deploy /home/deploy/.ssh/authorized_keys
docker exec rbpki-ssh16 chmod 0600 /home/deploy/.ssh/authorized_keys

docker exec rbpki-ssh16 ls -ld /home/deploy /home/deploy/.ssh \
  /home/deploy/.ssh/authorized_keys | tee "$LAB/permissions-report.txt"

ssh-copy-id performs exactly these steps and is what you should use day to day. Doing it by hand once is worth the two minutes, because it makes the failure in Task 7 obvious rather than magical.

Record what the installed line actually is. An authorized_keys entry is a single line of four possible fields: optional options, the key type, the base64 key, and a free-text comment. The comment is not checked by anything and is therefore the only place a human can record who this key belongs to and when it was issued.

LAB="$HOME/rbpki-lab-16"
{
  echo "installed line:"
  cut -c1-60 "$LAB/client/rbpki16.pub"
  echo "fingerprint sshd will log:"
  ssh-keygen -l -f "$LAB/client/rbpki16.pub"
} > "$LAB/authorized_keys.audit"
cat "$LAB/authorized_keys.audit"

Task 6 — Authenticate, and watch what the server records

LAB="$HOME/rbpki-lab-16"
SSHOPTS=(-o "UserKnownHostsFile=$LAB/client/known_hosts"
         -o StrictHostKeyChecking=accept-new
         -o IdentitiesOnly=yes
         -o BatchMode=yes)

ssh "${SSHOPTS[@]}" -i "$LAB/client/rbpki16" -p 12216 deploy@127.0.0.1 id

IdentitiesOnly=yes stops the client offering every key in your agent, so the log shows one attempt with one key rather than a queue of rejections. UserKnownHostsFile points at a lab-local file, which is why this lab never writes to your real known_hosts.

StrictHostKeyChecking=accept-new accepts a host key on first contact and still refuses a changed one. Use it, and not no, whenever automation must connect to a host it has not seen before.

Now read the server’s side of the same event.

docker logs rbpki-ssh16 2>&1 | grep -E 'Accepted|Failed|authorized' \
  | tee "$HOME/rbpki-lab-16/auth-evidence.log"

A successful login appears as a line beginning Accepted publickey for deploy, followed by the key type and the same SHA-256 fingerprint you recorded in Task 5. That fingerprint is the join key between “who logged in” and “which credential they used”, and it is the reason LogLevel VERBOSE matters.

Task 7 — Break it the way it usually breaks

Loosen the mode on authorized_keys and try again.

LAB="$HOME/rbpki-lab-16"
docker exec rbpki-ssh16 chmod 0666 /home/deploy/.ssh/authorized_keys
docker exec rbpki-ssh16 ls -l /home/deploy/.ssh/authorized_keys \
  | tee -a "$LAB/permissions-report.txt"

ssh -o "UserKnownHostsFile=$LAB/client/known_hosts" \
    -o StrictHostKeyChecking=yes -o IdentitiesOnly=yes -o BatchMode=yes \
    -i "$LAB/client/rbpki16" -p 12216 deploy@127.0.0.1 id

$ ssh -i rbpki16 -p 12216 deploy@127.0.0.1 id

deploy@127.0.0.1: Permission denied (publickey,keyboard-interactive).

That message is the same one you get for a wrong key, a missing key, a locked account and a user that does not exist. The list in brackets is the set of authentication methods the server was still willing to try, not a diagnosis. Treating that string as information is the single commonest way an SSH problem gets misdiagnosed.

The diagnosis is on the server:

docker logs rbpki-ssh16 2>&1 | tail -20

StrictModes yes makes sshd refuse to read authorized_keys when the file, the .ssh directory or the home directory is writable by group or other. Restore the mode and confirm the login works again.

docker exec rbpki-ssh16 chmod 0600 /home/deploy/.ssh/authorized_keys
ssh -o "UserKnownHostsFile=$HOME/rbpki-lab-16/client/known_hosts" \
    -o StrictHostKeyChecking=yes -o IdentitiesOnly=yes -o BatchMode=yes \
    -i "$HOME/rbpki-lab-16/client/rbpki16" -p 12216 deploy@127.0.0.1 hostname

Task 8 — Prove that possession is what counts

Generate a second key pair and attempt to authenticate with the second private key while the first public key is the one installed. Nothing about the server changes.

LAB="$HOME/rbpki-lab-16"
ssh-keygen -t ed25519 -f "$LAB/client/impostor" -C "impostor-lab-key" -N ""

ssh -o "UserKnownHostsFile=$LAB/client/known_hosts" \
    -o StrictHostKeyChecking=yes -o IdentitiesOnly=yes -o BatchMode=yes \
    -i "$LAB/client/impostor" -p 12216 deploy@127.0.0.1 id

The refusal is identical to the one in Task 7. The server side is not.

$ docker logs rbpki-ssh16 2>&1 | grep Failed | tail -1

Failed publickey for deploy from 172.25.0.1 port 52046 ssh2: ED25519 SHA256:/hOjaMxlXaTaWekzkFCIxR4PRU/c7eLkooB565Y7X6I

The address, port and fingerprint above are from the reference capture; yours will differ. The shape will not. Compare the logged fingerprint against authorized_keys.audit from Task 5: when they differ, the client offered a key the server does not recognise. When they match and the login still fails, the key is fine and the problem is the account, the modes or a Match block.

Repeat the failed attempt three or four times in quick succession and then look at the log again.

$ docker logs rbpki-ssh16 2>&1 | grep srclimit | tail -1

srclimit_penalise: ipv4: new 172.25.0.1/32 deferred penalty of 5 seconds for penalty: failed authentication

PerSourcePenalties is on by default and is not something you install. If your automation retries a bad key in a tight loop, OpenSSH will start deferring its connections, and the symptom will present as intermittent slowness rather than as an authentication problem.

Task 9 — Capture the deliverables

LAB="$HOME/rbpki-lab-16"
cd "$LAB"

docker exec rbpki-ssh16 ls -ld /home/deploy /home/deploy/.ssh \
  /home/deploy/.ssh/authorized_keys >> "$LAB/permissions-report.txt"
docker logs rbpki-ssh16 2>&1 | grep -E 'Accepted|Failed|srclimit|authorized' \
  > "$LAB/auth-evidence.log"

ls -l ssh-algorithms.txt authorized_keys.audit auth-evidence.log permissions-report.txt

Validation

• grep -c ssh-dss "$HOME/rbpki-lab-16/ssh-algorithms.txt" returns 0 and exits non-zero. Any other result means you are not on an OpenSSH 10.x build and the DSA discussion above does not apply to your host.
• ssh -o BatchMode=yes -o IdentitiesOnly=yes -i .../rbpki16 -p 12216 deploy@127.0.0.1 true exits 0 with no output. A non-zero exit with Permission denied (publickey,keyboard-interactive). means the file modes were not restored at the end of Task 7.
• grep -c 'Accepted publickey for deploy' auth-evidence.log is at least 1, and grep -c 'Failed publickey for deploy' is at least 2. A zero in the first is a lab that never succeeded; a zero in the second means LogLevel VERBOSE did not take effect.
• The fingerprint printed by ssh-keygen -l -f "$LAB/client/rbpki16.pub" appears verbatim on at least one Accepted publickey line.
• The deliverables ssh-algorithms.txt, authorized_keys.audit, auth-evidence.log and permissions-report.txt all exist and are non-empty.

Expected Outcome

$HOME/rbpki-lab-16/
├── state.pre-lab
├── ssh-algorithms.txt
├── authorized_keys.audit
├── auth-evidence.log
├── permissions-report.txt
├── client/
│   ├── known_hosts
│   ├── rbpki16
│   ├── rbpki16.pub
│   ├── impostor
│   └── impostor.pub
└── server/
    ├── sshd_config
    └── start.sh

You can now answer a question you could not answer before: given a Permission denied (publickey,keyboard-interactive). and nothing else, what do you ask for next. The answer is the server’s log at VERBOSE, and the specific thing you are looking for is whether a fingerprint was logged at all. A logged fingerprint that does not match the installed key is a client problem. A logged fingerprint that does match, or no fingerprint at all, is a server-side problem: modes, account state, or a Match block that never reached your user.

Troubleshooting

The container exits immediately after docker run. Run docker logs rbpki-ssh16. The usual cause is that apk add could not reach the package mirror, in which case the start script fails at its first line. Confirm outbound network from containers with docker run --rm alpine:3.22 apk update before retrying.

Every login fails and the server log contains no fingerprint at all. The deploy account is locked. Run docker exec rbpki-ssh16 passwd -u deploy and try again. This is the same trap described in Task 4, and it produces a completely ordinary Permission denied.

ssh says Bad owner or permissions. That is the client refusing to read its own key, not the server refusing you. Run chmod 0600 "$HOME/rbpki-lab-16/client/rbpki16". It happens when the lab directory has been copied out of an archive that did not preserve modes.

The port is already in use. Something else on your host holds 127.0.0.1:12216. Change the published port in Task 4 and in every ssh -p invocation, or stop the other listener. Do not publish the container on 0.0.0.0.

Cleanup

LAB="$HOME/rbpki-lab-16"

# 1. Stop and forget the container and its network.
docker rm -f rbpki-ssh16
docker network rm rbpki-net-16

# 2. Compare against the Task 1 capture before deleting the evidence.
cat "$LAB/state.pre-lab"
docker ps -a --filter name=rbpki- --format '{{.Names}}'
docker network ls --filter name=rbpki- --format '{{.Name}}'

# 3. Remove the lab directory, both key pairs included.
rm -rf "$LAB"

To confirm the host is as you found it, run docker ps -a --filter name=rbpki- and docker network ls --filter name=rbpki- once more: both must print only their header, matching the empty lists recorded in state.pre-lab. Then run test -d "$HOME/rbpki-lab-16" && echo "still present" || echo "removed". Your own ~/.ssh/known_hosts needs no attention, because the lab wrote its host key to $LAB/client/known_hosts and that file has just been deleted with the rest.

Production notes

• The comment field is your only inventory. Nothing in the protocol carries an owner, an issue date or an expiry. If your authorized_keys comments say user@laptop, you cannot answer “whose key is this and when was it issued”. Write a comment you could grep for during an incident, and accept that this is a convention your tooling has to enforce, not a guarantee.
• authorized_keys scales badly and that is structural. Every added engineer means a write to every server, and every departure means a delete you have to prove happened everywhere. Labs 17 and 18 replace the file with a certificate authority precisely to remove that fan-out.
• Use the option fields. A line may carry restrict to disable every forwarding feature and re-enable only what is needed, from="192.0.2.0/24" to bind a key to a source range, and command="..." to pin it to one operation. OpenSSH 10.5 fixed a case where restrict was not applied to tunnel forwarding, so if you rely on restrict as a boundary, know which release your fleet is on.
• Rotation is a fleet operation, not a key operation. Generating a replacement key takes a second. Proving the old public key is gone from every host, image, golden template and CI secret store is the part that takes a week, and it is the part that decides whether rotation actually happened.

What You Learned

• A public key is a lookup entry, not a credential. The server recognises you by the stored key and authenticates you by a signature over connection-specific data, which only the private key can produce. Copying authorized_keys gives an attacker recognition without the ability to sign.
• The client’s error message is deliberately uninformative. Permission denied (publickey,keyboard-interactive). covers a wrong key, a locked account, bad modes and a nonexistent user. The diagnosis lives in the server’s log at VERBOSE, and the first thing to check is whether a fingerprint was logged at all.
• Modes are part of the trust model. StrictModes refuses a group-writable or world-writable path to authorized_keys, because anyone who can write that file can add their own key. The refusal looks like a rejected key from the client side.
• DSA is gone and ssh-rsa means two different things. OpenSSH 10.0 removed DSA outright. ssh-rsa as a key type is ordinary RSA; ssh-rsa as a signature algorithm is SHA-1 and must not be negotiated. rsa-sha2-512 and rsa-sha2-256 are already the defaults.
• OpenSSH throttles failed authentication on its own. PerSourcePenalties has been on by default since 9.8, so a retry loop against a bad key degrades into apparent network slowness rather than a clean error.