Runbook: Rotate an SSH Certificate Authority

An SSH certificate authority is a single key with a single signature and no structure underneath it. There are no intermediates, no chains, no CRL and no OCSP. A server decides whether to accept a user certificate by looking up the signing key in the file named by TrustedUserCAKeys; a client decides whether to accept a host certificate by looking up the signing key in an @cert-authority line in known_hosts. That is the whole mechanism, and it is why rotation is a distribution problem rather than a cryptography problem.

Because the trust anchor is a file on every participant, the only safe rotation has an overlap in the middle. Both authorities are trusted at once, everything is reissued under the new one, and only then is the old one withdrawn. Reversing that order, or shortening the overlap because the change window is tight, locks people out of the hosts they need in order to fix it.

Rotate one authority at a time. A user CA and a host CA have different populations, different files and different symptoms, and doing both in one window means every failure has two candidate causes.

When this runbook applies, and when it does not

It applies to a planned replacement of an SSH user or host certificate authority across an estate that is already using certificates, with a maintenance window and an inventory.

It does not apply when:

• The CA private key is believed to be compromised. An overlap gives an attacker a window in which their forged certificates are still accepted. That is an incident with a deliberately short and disruptive cutover, not a scheduled rotation.
• The estate is not using certificates yet. Rolling out certificates for the first time is a build project with a pilot group. Running a rotation procedure against authorized_keys files achieves nothing.
• You cannot reach the hosts out of band. Every step below assumes a path that does not depend on the trust you are changing. Without it there is no rollback, only an incident.
• The inventory is incomplete. A rotation is only as good as the list of participants. Withdrawing trust from an estate you cannot enumerate strands whatever you missed.

Blast radius

Step 1 - Record what is trusted now, and match it to a fingerprint

The fingerprint is how you will recognise the right authority in a server log later, so capture it before anything changes.

CA_DIR=/etc/ssh/ca
# Fingerprint each authority. Keep both lines in the change record.
ssh-keygen -l -f "$CA_DIR/user_ca_2026.pub"
ssh-keygen -l -f "$CA_DIR/user_ca_2027.pub"
cp -a /etc/ssh/ca/user_ca_trusted "/root/user_ca_trusted.pre-rotation"

The fingerprint printed here is the same string sshd writes into its authentication log and the same string ssh-keygen -L reports on an issued certificate, which is what makes the three observations comparable. Keep the copy of the trust file: it is the rollback.

The server configuration that consumes it is small enough to read in full, and every line in it matters during a rotation:

HostKey /etc/ssh/ssh_host_ed25519_key
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
TrustedUserCAKeys /etc/ssh/ca/user_ca_trusted
RevokedKeys /etc/ssh/ca/revoked.krl
PubkeyAuthentication yes
PasswordAuthentication no
LogLevel VERBOSE

TrustedUserCAKeys names a file that holds one public key per line, which is exactly the property the overlap depends on. LogLevel VERBOSE is what makes the verification step possible; a rotation performed at the default log level cannot be proved.

Step 2 - Publish the new authority alongside the old one

Build the file beside its target and move it into place in a single operation. A trust file caught half-written by an incoming connection is a lockout.

CA_DIR=/etc/ssh/ca
cat "$CA_DIR/user_ca_2026.pub" "$CA_DIR/user_ca_2027.pub" > "$CA_DIR/user_ca_trusted.new"
chmod 0644 "$CA_DIR/user_ca_trusted.new"
mv "$CA_DIR/user_ca_trusted.new" "$CA_DIR/user_ca_trusted"

For a host CA the equivalent lives on the clients, and again the new line is added rather than substituted:

@cert-authority *.lab.example ssh-ed25519 AAAAC3NzaC1lZDI1NTE5...
@cert-authority *.lab.example ssh-ed25519 AAAAC3NzaC1lZDI1NTE6...

Only one marker is permitted per line, and @revoked on a key means it is never accepted for authentication or as a certification authority. That second property is what makes revocation of an old host CA meaningful rather than advisory.

Step 3 - Validate, reload, and test from a different terminal

SSHD_UNIT=ssh          # Debian and Ubuntu; use sshd on the Red Hat family
sshd -t
systemctl reload "$SSHD_UNIT"

Keep the console session open and open a brand new SSH connection from a second terminal to test. Reconnecting the session you are relying on is not a test; it is the thing you are protecting.

Step 4 - Prove the overlap before issuing anything new

Authenticate with an existing certificate signed by the outgoing authority and confirm it still works. If it does not, the trust file is wrong and you have just discovered it while the old authority is still deployed, which is the cheapest possible moment.

Step 5 - Reissue from the new authority, with a window and a serial

$ ssh-keygen -s user_ca -I "alice@runbook-lab" -n deploy -V -5m:+1h -z 1001 alice.pub

Signed user key alice-cert.pub: id "alice@runbook-lab" serial 1001 for deploy valid from 2026-08-26T21:17:09 to 2026-08-26T22:22:09

-V is not optional. Without it a certificate is valid from the Unix epoch to the distant future, which converts a short-lived credential system into a permanent one. -z is equally load-bearing: the default serial is zero, and a revocation list revokes certificates by serial number for values that do not include zero, so a certificate issued without a serial can only be revoked by key identity.

Host certificates use -h, and their principals are hostnames:

ssh-keygen -s host_ca -I "sshd.lab.example" -h -n sshd.lab.example \
  -V -5m:+52w /etc/ssh/ssh_host_ed25519_key.pub

Wildcards in principals are supported for host certificates and are not supported for user certificates. Since OpenSSH 10.3 an empty principals list matches nothing at all rather than acting as a wildcard, and certificates that carry no principals are never accepted through TrustedUserCAKeys.

Step 6 - Confirm the certificate names the authority you expect

$ ssh-keygen -L -f alice-cert.pub

alice-cert.pub:
      Type: ssh-ed25519-cert-v01@openssh.com user certificate
      Public key: ED25519-CERT SHA256:/hOjaMxlXaTaWekzkFCIxR4PRU/c7eLkooB565Y7X6I
      Signing CA: ED25519 SHA256:6x39cg8OAp7PZgisHreCLRO7g0vp6s0VZIw+DcTgKtE (using ssh-ed25519)
      Key ID: "alice@runbook-lab"
      Serial: 1003
      Valid: from 2026-08-26T19:21:30 to 2026-08-26T20:21:30
      Principals:
              deploy
      Critical Options: (none)
      Extensions:
              permit-X11-forwarding
              permit-agent-forwarding
              permit-port-forwarding
              permit-pty
              permit-user-rc

Compare the Signing CA fingerprint against the new CA public key, not against the one you intended to use. During an overlap both keys are present on the signing host and the wrong one is one shell history entry away.

Step 7 - Read the server log for the decisive evidence

The client says it connected. The server says which authority let it in, and which file that authority was read from. Only the second answers the question this rotation exists to settle.

Accepted certificate ID "alice@runbook-lab" (serial 1001) signed by ED25519 CA SHA256:6x39cg8... via /etc/ssh/ca/user_ca_trusted
Accepted publickey for deploy from 192.0.2.41 port 51328 ssh2: ED25519-CERT SHA256:... ID alice@runbook-lab (serial 1001) CA ED25519 SHA256:...

Those two lines are the shape sshd emits at LogLevel VERBOSE, with the file path and the source address of this procedure substituted in. The detail that matters is via, which names the trust file the accepted authority was read from, and the CA fingerprint immediately before it.

Since OpenSSH 10.0 the server is split across sshd, sshd-session and sshd-auth, so these lines may be attributed to any of the three depending on how your logging is configured. Search on the message, not on the process name.

Step 8 - Withdraw the old authority, one group at a time

Only when every certificate signed by the outgoing authority has been reissued or has expired. Remove its line from the trust file using the same atomic write as before, verify the group, and only then move to the next.

Then revoke it, so that a stray copy of an old trust file cannot silently restore it:

printf 'serial: 1001\n' | ssh-keygen -k -f revoked.krl -s user_ca_2026.pub -z 1 -
ssh-keygen -Qf revoked.krl alice-cert.pub

A revocation list can revoke by serial number, by serial range, by key identity or by the key itself. The test command exits zero only when nothing presented to it has been revoked, which makes it usable directly in a verification script.

Common pitfalls

Verification

The rotation is verified when a certificate signed by the new authority authenticates from a client that never held one from the old authority, and when the server log for that login names the new CA fingerprint and the trust file it was read from. Those two together are the independent channel; the signing command’s own success message is not. Confirm that ssh-keygen -L on a reissued certificate reports a Signing CA fingerprint matching ssh-keygen -l on the new public key. Confirm a client carrying only the new host CA connects with no first-contact prompt. After withdrawal, confirm that a certificate signed by the old authority is now refused and that the server log explains the refusal in words rather than leaving you to infer it. Finally, check that configuration management reports the full inventory as converged, and that an out-of-band console login still works on a sample host in each group, because the fallback path is the one thing that must never be collateral damage.

Rollback

Every step has the same rollback: restore the previous file from the copy taken in the pre-checks, run sshd -t, and reload from the console session you deliberately kept open. Adding the old CA public key line back to the trust file undoes a premature withdrawal in seconds, which is why the old public key is retained until the rotation is signed off rather than deleted at the moment trust is removed. If a revocation file has locked out public key authentication on a host, the fix is on the console: confirm the file exists, is readable by the server, and is a valid revocation list. If clients have begun prompting for host keys, restore the previous known_hosts content rather than telling anyone to accept the prompt. If only part of the estate converged, stop, leave both authorities trusted, and treat the divergence as a configuration management problem. If the new private key was handled in any way you cannot fully account for, destroy it and begin again with a fresh one.