Host keys and trust on first use — what TOFU guarantees

Server authentication happens first in SSH, before any password or key of yours is offered. The server signs a value derived from the key exchange with its host private key, and the client checks that signature against a public key it already holds. The cryptography is solid. The weak joint is how the client came to hold that public key in the first place.

Possession proof and trust decision are two different things

The signature the server produces during key exchange proves one thing only: whoever is on the other end of this connection holds the private half of the host key they just presented. It says nothing about whether that key belongs to the machine you meant to reach.

The client therefore performs a second, entirely separate step. It looks up the hostname you typed in its known-hosts files and compares the presented key against what it finds. Three outcomes exist, and they are not variations of one another.

• Known and matching. The lookup finds an entry for this host and this key type, and the bytes agree. The connection proceeds with no output at all.
• Not known. No entry exists for this host. The client falls back to whatever StrictHostKeyChecking says, which by default means asking a human.
• Known and different. An entry exists and the key does not match. This is the loud failure, and the next lesson is devoted to it, because the correct response is investigation rather than deletion.

flowchart TD
    A["Server signs exchange hash\nwith its host key"] --> B["Client verifies signature"]
    B --> C{"Entry in known_hosts?"}
    C -- "no entry" --> D["StrictHostKeyChecking decides"]
    C -- "entry matches" --> E["Connect silently"]
    C -- "entry differs" --> F["Refuse and warn"]

The diagram makes the split visible: the left half is arithmetic that either works or does not, and the right half is a policy lookup against a local file. A man-in-the-middle does not break the signature, it supplies a different host key and hopes the lookup has nothing to compare against.

Reading a known_hosts line

The format is documented in sshd(8) and has five positional fields: an optional marker, the hostname patterns, the key type, the base64 key, and a comment. Only one marker may appear on a line.

sshd.lab.example,192.0.2.41 ssh-ed25519 AAAAC3Nza...
@cert-authority *.lab.example ssh-ed25519 AAAAC3Nza...
@revoked old-web-01.lab.example ssh-ed25519 AAAAC3Nza...
|1|BASE64SALT|BASE64HASHOFHOSTNAME ssh-ed25519 AAAAC3Nza...

The first line is the ordinary case, recording one key against a name and an address. The second is a certificate authority marker, which the SSH CA lesson later in this part builds on: it replaces per-host entries entirely. The third marks a key as revoked, and a revoked key is never accepted for authentication or as a certification authority, which makes it the right way to retire a key you believe was stolen. The fourth is a hashed entry.

Hashing is controlled by HashKnownHosts, which upstream defaults to no. Several distributions enable it in /etc/ssh/ssh_config, so check before assuming. When it is on, the hostname field is replaced by a salted keyed hash, and the practical consequence is that you cannot grep the file for a hostname any more. The lookup has to be done by the tool that knows how to recompute the hash.

# Find and inspect an entry whether or not the file is hashed.
HOSTNAME=sshd.lab.example
ssh-keygen -F "$HOSTNAME"
ssh-keygen -l -F "$HOSTNAME"

The -F search recomputes the hash for each line, so it works on both forms. The -l variant adds the fingerprint, which is the value you compare against an out-of-band record. Two further defaults are worth knowing because they surprise people: CheckHostIP is no, so the address is not independently tracked unless you turn it on, and UserKnownHostsFile covers ~/.ssh/known_hosts and ~/.ssh/known_hosts2 while GlobalKnownHostsFile adds /etc/ssh/ssh_known_hosts and its 2 variant for entries an administrator distributes.

What trust on first use actually guarantees

Trust on first use guarantees continuity. Having accepted a fingerprint once, you will be told if the key presented under that name ever changes. That is a genuine and useful property: it turns a silent interception into a visible event, and it means an attacker must either steal the host key or accept being noticed.

It guarantees nothing about the first connection. At that moment the client has no reference point, so the fingerprint in the prompt is whatever the far end sent. If the far end is an interceptor, the prompt shows the interceptor’s fingerprint and looks completely normal. Almost nobody compares it against anything.

The authenticity of host '<host> (<ip>)' can't be established.
<TYPE> key fingerprint is: SHA256:....
Are you sure you want to continue connecting (yes/no/[fingerprint])?

That is the upstream prompt with the substitutions the source leaves in place. The bracketed third answer is the part worth teaching: instead of typing yes, you can paste the fingerprint you obtained independently, and the client accepts the connection only if it matches. It turns a rubber-stamp into a comparison, at no cost.

There is a variant of the first line worth recognising. If keys of other types are already recorded for this host, the message says that the authenticity cannot be established but keys of different type are already known for this host. That wording usually means a host key algorithm was added or reordered rather than that anything is wrong, and it is a different situation from the changed-key warning in the next lesson. On acceptance the client confirms with a line beginning Warning: Permanently added, which is worth matching on in automation logs, because it is the trace left every time a machine made a trust decision with nobody watching.

Seeding trust without a first contact

The fix for the first-contact gap is to remove the first contact. Distribute the host keys to clients from a source that is already trusted, so every connection is a match rather than a decision.

ssh-keyscan collects host keys over the network, optionally hashing its output with -H, requesting certificates with -c, emitting SSHFP records with -D, and accepting CIDR ranges. It has a default timeout of five seconds. It also carries an explicit warning in its own manual: “ssh-keyscan cannot verify the authenticity of the host keys it obtains, and an attacker on the network with the ability to intercept traffic could substitute their own host keys to enable impersonation of servers.” Scanning is a convenient way to build a file, not a way to establish trust. The keys must still be checked against something authoritative, such as the values recorded when each host was built.

Better sources exist. A provisioning system that generates the host key can publish the fingerprint into an inventory at build time. A cloud instance can emit its fingerprints to the console log where they can be read through an API rather than over SSH. Configuration management can then write a single /etc/ssh/ssh_known_hosts for every workstation and jump host. Best of all, a host certificate authority collapses the whole file to one line, which is where this part is heading.

DNS offers a third route through SSHFP records, published in the zone and consulted when VerifyHostKeyDNS is enabled. It is off by default, and its assurance is exactly the assurance of your DNS: with DNSSEC and a validating resolver it is a real out-of-band channel, and without them it is a lookup an attacker on the path can answer. ssh-keygen -r generates the records and ssh-keyscan -D collects them from live hosts. One deprecation is already announced: OpenSSH 10.1 and 10.2 both carry notice that SHA-1 SSHFP records will be ignored in a future release and that generation will produce SHA-256 records only, so a zone populated years ago needs regenerating rather than merely reviewing.

Production discipline

1. Record the fingerprint at build time, not at first login. The provisioning run is the only moment when the host key’s provenance is not in question.
2. Ship /etc/ssh/ssh_known_hosts from configuration management. A global file removes the prompt for every user on the machine and gives you one place to audit.
3. Use the fingerprint answer, never yes, when a prompt is unavoidable. Paste the value you were given out of band and let the client do the comparison.
4. Mark retired keys @revoked rather than deleting them. A deleted entry can be re-accepted by mistake; a revoked one cannot be accepted at all.
5. Check whether your distribution hashes known_hosts before writing tooling. A script that greps for a hostname silently returns nothing on a hashed file and reports success.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (Configuration Management) covers distributing a file to every host in a fleet, which is the delivery mechanism a global known-hosts file needs.
• Ansible for Production Sysadmins - Part XX (SSH) covers host key checking in automation, where the accept-or-refuse decision has to be made with no interactive prompt available.
• Git, CI/CD & GitOps for Infrastructure Engineers - Part XLI (Runner Security) covers ephemeral build hosts, which generate a fresh host key on every boot and so break continuity by design.

Quiz