SSH certificate authorities — trusting one key instead of thousands

Distributing public keys does not scale, and the failure is not about effort. With users and hosts both growing, the number of grants to keep synchronised grows as their product, every grant lives on the host it applies to, and removing one is only as reliable as your inventory. A certificate authority replaces that entire structure with two keys and an expiry date.

The arithmetic that forces the change

Two hundred engineers across nine hundred hosts is one hundred and eighty thousand potential grants. Nobody stores it that way, of course; they store group membership in a configuration management system and render authorized_keys from it. The problem is not the size of the file, it is the properties of the result.

• Removal is a convergence race. An engineer leaves and their access ends when the last host runs configuration management, which might be minutes or might be the host that has been failing its runs since March.
• The list has no expiry. Nothing in the file times out, so a host that drops out of management keeps every grant it had at that moment, indefinitely.
• Every host is authoritative. There is no single place to ask who has access, only nine hundred places to survey and reconcile.
• The host side is worse. Nine hundred host keys must reach every client, or every client accepts a first-contact prompt.

A certificate flips the direction of the flow. Instead of pushing a grant to every host in advance, the user carries a signed statement that any host can verify on its own, and the statement expires by itself.

flowchart LR
    UCA["User CA key\noffline"] --> UC["User certificate\nprincipals, validity"]
    HCA["Host CA key\noffline"] --> HC["Host certificate\nhostnames, validity"]
    UC --> S["sshd trusts one\nuser CA public key"]
    HC --> C["client trusts one\nhost CA public key"]

Two authorities, two directions. The user CA lets a host decide whether to admit a person without holding a list of people. The host CA lets a client decide whether a machine is genuine without holding a list of machines. Neither replaces the other, and running one without the other is a common half-migration that leaves the trust on first use problem exactly where it was.

What the certificate actually contains

An SSH certificate is a signed blob wrapping a public key. The field order defined in draft-ietf-sshm-cert is: a nonce, the public key material, a serial number, the certificate role (user or host), a key identifier, the principals list, valid-after and valid-before timestamps, critical options, extensions, a reserved field, the signature key and the signature.

The nonce comes first, before the key material, and that ordering is deliberate: it denies an attacker control of the beginning of the signed message and so blocks chosen-prefix attacks against the signature hash. The specification requires it to be at least sixteen bytes and notes that thirty-two is typical.

Now the omissions, which matter more than the fields.

• There is one signature and one level. No intermediates, no path building, no cross-certification. The ssh-keygen manual states it directly: OpenSSH certificates are “a different, and much simpler, format to the X.509 certificates used in ssl(8).”
• There is no CRL distribution point and no OCSP responder. A certificate carries no pointer to any revocation service, because none exists in the format. Revocation is a list the verifier must already hold, which the next lesson covers.
• There are no extension OIDs and no policy layer. Critical options and extensions are named strings, and the rule is simple: an unrecognised critical option causes the certificate to be refused, while an unrecognised extension must be ignored.

Configuring both directions

The server needs to know which CA it trusts for users, and it needs a certificate for itself. This is the configuration that was executed to verify this lesson, with the lab paths as they were written.

HostKey /path/hostkey
HostCertificate /path/hostkey-cert.pub
TrustedUserCAKeys /path/user_ca.pub
RevokedKeys /path/revoked.krl
PubkeyAuthentication yes
PasswordAuthentication no
AuthorizedKeysFile /dev/null
LogLevel VERBOSE

TrustedUserCAKeys names a file of CA public keys, one per line. A certificate signed by a key in that file may be used to log in as any user named in the certificate’s principals list. If AuthorizedPrincipalsFile is not configured, the default rule applies: the target username itself must appear in the principals list. Two constraints commonly trip people up. Certificates that lack a principals list are never permitted through TrustedUserCAKeys. And AuthorizedPrincipalsFile is only consulted for CAs listed there; a CA trusted through a cert-authority line in authorized_keys uses that line’s own principals= option instead, and where both a certificate restriction and a key option apply, the most restrictive union of the two wins.

Where the CA private key should physically live is the design decision this configuration forces. ssh-keygen supports two answers directly. With -D it will sign using a key held in a PKCS#11 token, which keeps the private key inside hardware that cannot export it. With -U it will sign using a key held in an agent, which allows a dedicated signing host to hold the key in memory behind whatever authentication you place in front of it; OpenSSH 10.2 fixed a defect in exactly that path, so treat 10.2 as the floor if you rely on it. Either way, the signing operation and the authorisation decision should be separable: something must decide that this person may have a certificate for this principal today, and that something is a policy service, not ssh-keygen.

AuthorizedKeysFile /dev/null in the sample is the interesting line. It proves the estate no longer depends on per-host key lists at all. On a real host it also produces a benign log entry on every successful login, User 'deploy' authorized keys '/dev/null' is not a regular file, which is easy to mistake for the cause of a failure it has nothing to do with.

The client side is one line in a known-hosts file, and it replaces every per-host entry the pattern covers.

@cert-authority *.lab.example ssh-ed25519 AAAAC3NzaC1lZDI1NTE5...

Every host whose name matches the pattern and which presents a valid certificate from that CA is accepted with no prompt and no stored key. Deployed globally in /etc/ssh/ssh_known_hosts, this is the line that ends trust on first use for the whole estate.

Reading a successful certificate login

Verbose logging on both ends turns certificate authentication into something you can audit. On the client, the exchange announces both halves:

debug1: Server host certificate: ssh-ed25519-cert-v01@openssh.com SHA256:... serial 0 ID "sshd.lab.example" CA ssh-ed25519 SHA256:... valid from 2026-08-26T21:16:24 to 2027-08-25T21:21:24
debug1: Host 'sshd.lab.example' is known and matches the ED25519-CERT host certificate.
debug1: Offering public key: alice ED25519-CERT SHA256:... explicit
debug1: Server accepts key: alice ED25519-CERT SHA256:... explicit
Authenticated to 127.0.0.1 ([127.0.0.1]:12222) using "publickey".

The host certificate is reported before authentication begins, with its serial, key identifier, issuing CA and validity window. Note the host certificate’s serial of zero, which is the default when -z is omitted and which has consequences for revocation. On the server the same event produces two lines at VERBOSE:

Accepted certificate ID "alice@runbook-lab" (serial 1001) signed by ED25519 CA SHA256:6x39cg8... via /tmp/user_ca.pub
Accepted publickey for deploy from [REDACTED] port 51328 ssh2: ED25519-CERT SHA256:... ID alice@runbook-lab (serial 1001) CA ED25519 SHA256:...

This is the operational payoff. The log names the key identifier chosen at issuance, the serial, the issuing CA and the local account that was entered. A key list can tell you that some key succeeded; a certificate tells you which issuance, from which authority, under which identity, and lets you tie the session back to the request that produced it.

Production discipline

1. Deploy both CAs or accept that you deployed neither. A user CA with per-host known_hosts entries still leaves every client accepting first-contact prompts.
2. Keep CA private keys off general-purpose hosts. Hardware tokens through PKCS#11, or an agent on a dedicated signer, are the two supported paths in ssh-keygen itself.
3. Set LogLevel VERBOSE before you need it. Without it the log records a successful public key and discards the identifier, serial and issuing CA that make the event auditable.
4. Distribute the @cert-authority line globally, not per user. A line in each engineer’s own file is a line each engineer can remove or fail to receive.
5. Plan CA rotation before first issuance. Every host and every client holds the trust anchor, so replacing it is a fleet-wide change that wants a period where both keys are trusted.

Cross-course references

• Linux for Production Sysadmins - Part LI (Fleet Management) covers operating a large host estate, which is where the synchronisation costs described here are actually paid.
• Kubernetes for Production Sysadmins - Part LXXVI (Certificates) covers a cluster PKI built on X.509, which is the useful contrast: chains, intermediates and path building that SSH omits entirely.
• Observability for Production Sysadmins - Part LXIV (TLS Monitoring) covers alerting on expiry horizons, the same discipline a long-lived host certificate needs.

Quiz