Private keys - passphrases, agents and the keys you cannot rotate

Private keys are the awkward case in secret management. Everything in the previous lessons applies to them - mode, ownership, backups, copies - and then they add two problems of their own.

The first is that a key is used constantly, so it has to be available to a process without a human present. The second is that rotating one is not a config change: an SSH key with forty authorized_keys entries and a TLS key with a certificate issued against it both have consumers who will break when you replace them. Keys are the secrets people delay rotating, which is exactly why they need the tightest handling.

What a passphrase buys

A passphrase encrypts the key file. It protects the key against anyone who obtains the file - a stolen laptop, a backup tape, a repository someone cloned, a disk that left the building without being wiped.

It does not protect against a compromised running system. Once the key is loaded into an agent, or read into a running service, the plaintext key is in memory on that host, and anyone with root on that host has it.

That is the whole distinction, and it drives the decision:

• Interactive human keys: passphrase, always. The threat is a lost device, and the passphrase is exactly the right control.
• Service keys used unattended: a passphrase means either a human types it at every start - which fails at 3am - or the passphrase sits in a file next to the key, which protects nothing. Use an unencrypted key at 0400 owned by the service, on a host you actually secure, and spend the effort on shortening the key lifetime instead.

# Add or change a passphrase on an existing key
ssh-keygen -p -f ~/.ssh/id_ed25519

# Check the format and the KDF work factor when creating one
ssh-keygen -t ed25519 -a 100 -f ~/.ssh/id_ed25519

-a 100 sets the number of bcrypt KDF rounds used to derive the file encryption key from your passphrase; a higher number makes an offline guessing attack proportionally slower. It applies to the OpenSSH key format, which has been the default since OpenSSH 7.8 - a key whose first line reads -----BEGIN OPENSSH PRIVATE KEY----- is in that format, and one reading -----BEGIN RSA PRIVATE KEY----- is in the old PEM format with a much weaker derivation.

head -1 ~/.ssh/id_ed25519

Bounding a loaded key: the agent

ssh-agent holds decrypted keys so you type the passphrase once. Everything it holds is usable by anyone who can reach its socket, so the useful questions are how long, and for what.

# Load with a lifetime: the agent forgets the key after an hour
ssh-add -t 3600 ~/.ssh/id_ed25519

# Require an explicit confirmation for every use of this key
ssh-add -c ~/.ssh/id_ed25519

# What is loaded right now?
ssh-add -l

# Drop everything
ssh-add -D

-t takes a lifetime in seconds, or a time suffix such as 1h. -c makes each authentication attempt prompt for confirmation, which is the control that turns a silent misuse of your key into something you notice. Both are worth setting on any key that reaches production.

Agent forwarding

ssh -A exposes your agent socket on the remote host. For as long as you are connected, root on that host - and the owner of the socket, which is you - can ask your agent to sign anything. They cannot steal the key, but they do not need to: they can authenticate as you to everything the key opens, including hosts you have never mentioned.

$ ssh -J bastion.example.com app01.example.com

Warning: Permanently added 'app01.example.com' (ED25519) to the list of known hosts.
Linux app01 6.1.0-40-amd64 #1 SMP x86_64
Last login: Tue Aug 11 09:14:02 2026 from 192.0.2.14

-J (ProxyJump) is the answer in most cases where people reach for -A. It opens a TCP channel through the bastion and completes the authentication to the final host from your workstation, so the bastion is a router rather than a party to the authentication. No agent socket appears on it.

When you genuinely need forwarding - git operations run on a remote host is the honest case - constrain it:

ssh-add -h git.example.com ~/.ssh/id_ed25519

-h adds a destination constraint: the agent will only use that key to authenticate to the named destination, so a compromised intermediate host cannot use it against anything else. It requires OpenSSH 8.9 or later on both ends.

And set the default the safe way round in ~/.ssh/config:

Host *
    ForwardAgent no
    AddKeysToAgent yes
    IdentitiesOnly yes

Host git.example.com
    ForwardAgent yes

ForwardAgent yes under Host * is the configuration that causes this problem, and it is common because someone needed it once for one host.

TLS keys: matching, and the private directory

The most common TLS incident is not a stolen key. It is a renewal where the certificate and the key stop matching, and the service fails to start with an error that names neither file usefully.

Prove they match by comparing public keys. This works for RSA and EC alike, unlike the older modulus comparison:

openssl pkey -in /etc/ssl/private/server.key -pubout -outform DER | sha256sum
openssl x509 -in /etc/ssl/certs/server.crt -pubkey -noout -outform PEM \
  | openssl pkey -pubin -pubout -outform DER | sha256sum

Two identical hashes mean the certificate was issued against that key. The private key never leaves the first command, and the second command reads only public material.

While you are there, the expiry and the subject:

openssl x509 -in /etc/ssl/certs/server.crt -noout -enddate -subject -issuer

On Debian-family systems /etc/ssl/private is mode 0700, group ssl-cert, and the intended pattern is to add the service account to ssl-cert rather than to loosen the directory. Check what you have:

stat -c '%a %U %G %n' /etc/ssl/private
getent group ssl-cert

If a service cannot read its key, the fix is group membership plus a restart of that service - not chmod 755 on a directory holding every private key on the host.

Keys that got baked into an image

An image or template captured after the SSH host keys were generated gives every clone the same host keys. The consequences are not theoretical:

• Every host presents the same host key, so known_hosts cannot distinguish them and a genuine man-in-the-middle is indistinguishable from a rebuild.
• Anyone who obtains the image - a developer with the template, an old backup, a public marketplace image - can impersonate every host built from it.

# Compare across hosts: these fingerprints must all differ
for h in app01 app02 app03; do
  printf '%s ' "$h"
  ssh-keyscan -t ed25519 "$h.example.com" 2>/dev/null | ssh-keygen -lf -
done

The build procedure must remove the host keys before the image is sealed, and the first boot must generate new ones. cloud-init does this by default; a hand-built template does not:

sudo rm -f /etc/ssh/ssh_host_*        # before sealing the image
sudo ssh-keygen -A                    # regenerate on first boot

ssh-keygen -A generates any missing host key of each supported type, with default parameters, and is what the distribution boot scripts call.

The same applies to any other key material a template can capture: a machine-id (which some tools derive identity from), a Puppet or Salt client certificate, a monitoring agent token.

When a key is compromised

1. Fingerprint the compromised key so you can search for it: ssh-keygen -lf key.pub.
2. Generate and distribute the replacement first, so you do not lock yourself out while removing the old one.
3. Search every authorized_keys on every host for the fingerprint, including service accounts and root.
4. Search the configuration management repository, the CI system and any deploy tooling for the public key.
5. Remove the old key, then verify by attempting an authentication with it and confirming it is refused.
6. Check the auth logs for uses of that key since the suspected compromise: sshd logs the key fingerprint on every accepted publickey authentication.

The last step is the one people skip and the one that answers the question the incident review will ask. sshd records the fingerprint of the key that authenticated:

sudo journalctl -u ssh --since '-30d' --no-pager \
  | grep 'Accepted publickey' | awk '{print $NF}' | sort | uniq -c | sort -rn

The unit is ssh on Debian-family systems and sshd on RHEL-family ones.

Knowledge check