Skip to main content
RunBook Academy

Secrets, PKI & CertificatesXI · SSH Keys, Host Trust and SSH CAsSSH

Issuing, constraining and revoking SSH certificates

Advanced⏱ ~25 min🧪 Lab requiredsshssh-keygensshd

What you'll learn

  • Issue a certificate with an identifier, principals, validity and serial
  • Apply the OpenSSH 10.3 rules for empty principals and for wildcards
  • Choose between a critical option and an extension for each restriction
  • Build and distribute a KRL, and read the refusal it produces

Prerequisites

Practice

Verified against OpenSSL 3.5.x teaching target; 3.0+ minimum · OpenSSH 10.x teaching target; 8.2+ minimum for certificate workflows · OpenBao 2.6.x · Smallstep step-ca 0.30.x · Certbot / Pebble Certbot current release; Pebble 2.10.x ACME test server · Kubernetes (cross-course target) 1.36.x · PostgreSQL 17.x · 2026-08-26

Not yet marked complete on this device.

Signing a public key with a CA key takes one command, and the defaults on that command are hostile. Omit one flag and the certificate never expires. Omit another and it can never be revoked by serial. Get the principals wrong in either direction and it either opens too much or, since OpenSSH 10.3, nothing at all.

The four flags that decide everything

This is a real issuance, executed on the authoring host, with its actual output.

$ ssh-keygen -s user_ca -I "alice@runbook-lab" -n deploy -V -5m:+1h -z 1001 alice.pub
Signed user key alice-cert.pub: id "alice@runbook-lab" serial 1001 for deploy valid from 2026-08-26T21:17:09 to 2026-08-26T22:22:09

Four flags carry the meaning, and each one has a failure mode when omitted.

  • -I sets the key identifier. It is a human-readable string that the server records in its log on every use. This is the field that lets an access review answer “who was this” months later, so it should encode the requester and the request, not a serial number you already have.
  • -n sets the principals. These are the names the certificate may act as, and the section below is entirely about them.
  • -V sets the validity interval. Leaving it out is the single most dangerous omission in SSH certificate practice: without -V a certificate is valid from the Unix epoch to the distant future. It does not default to a day, or to the CA key’s own life. It never expires. The -5m start in the example is backdating by five minutes to absorb clock skew between the signer and the target.
  • -z sets the serial. The default serial is zero, and a KRL revokes certificates by serial only for non-zero values. A certificate issued without -z can be revoked by key identifier or by its own key, but it cannot participate in the compact serial-based revocation that makes KRLs cheap.

Inspecting the result shows every field the server will act on.

alice-cert.pub:
        Type: ssh-ed25519-cert-v01@openssh.com user certificate
        Public key: ED25519-CERT SHA256:/hOjaMxlXaTaWekzkFCIxR4PRU/c7eLkooB565Y7X6I
        Signing CA: ED25519 SHA256:6x39cg8OAp7PZgisHreCLRO7g0vp6s0VZIw+DcTgKtE (using ssh-ed25519)
        Key ID: "alice@runbook-lab"
        Serial: 1003
        Valid: from 2026-08-26T19:21:30 to 2026-08-26T20:21:30
        Principals:
                deploy
        Critical Options: (none)
        Extensions:
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

Note the five extensions nobody asked for. They are the defaults, and they are why the constraints section below starts by clearing them.

Principals, and what changed in OpenSSH 10.3

For a user certificate, the principals are the account names the holder may log in as. For a host certificate, they are hostnames. Three rules govern them on current OpenSSH, and two of them contradict most published material.

  • An empty principals list matches nothing. Before 10.3, a certificate with an empty principals section was treated as a wildcard when used through the principals="" option in authorized_keys. That release changed it: an empty principals section now never matches any principal. Tutorials, vendor documentation and answers written before April 2026 still describe the old behaviour, and following them produces a certificate that authenticates nowhere.
  • Wildcards are host certificates only. The same release made wildcard handling consistent: wildcard characters in certificate principals work for host certificates and are not supported for user certificates. *.lab.example in a host certificate covers a domain. admin-* in a user certificate is a literal string that will match an account of that exact name or nothing.
  • The username must be listed unless a principals file says otherwise. With TrustedUserCAKeys and no AuthorizedPrincipalsFile, the account being entered must itself appear in the list. Add a principals file or command and the rule becomes membership of that file instead, which is how role names such as deploy or dba get mapped onto local accounts.

The command form deserves care because it moves an authorisation decision into a program. AuthorizedPrincipalsCommand must be owned by root, must not be writable by group or others, and must be given as an absolute path, which is the server refusing to execute anything an unprivileged user could have replaced. It runs only for valid users, so it cannot be used to probe for account existence. And once either the command or the file is configured, a certificate must contain a principal that is listed, which makes this the mapping layer where group membership from a directory service turns into local access without any of it being written to disk on the host.

Critical options and extensions are not the same thing

Both are named restrictions carried in the certificate, and the difference is what a verifier does when it does not recognise one. An unrecognised critical option causes the certificate to be refused. An unrecognised extension must be ignored. That single rule decides which mechanism a control belongs in.

The specification defines three critical options: force-command, which replaces whatever the client asked to run; source-address, which limits the addresses the certificate may be used from; and verify-required, which demands a user-verification step on a hardware key. Extensions are the permissive flags, including the five permit- entries seen above and no-touch-required.

# Issue a deployment certificate that can do exactly one thing.
CA=/etc/ssh/user_ca
ssh-keygen -s "$CA" -I "deploy-run-4821" -n deploy \
  -V -5m:+15m -z 4821 \
  -O clear \
  -O force-command=/usr/local/bin/deploy-runner \
  -O source-address=192.0.2.0/24 \
  deploy-runner.pub

-O clear is doing real work there. The four no- forms of the forwarding, pty and user-rc options exist because each capability is permitted by default; clearing first and then granting nothing leaves a certificate that can open a session and run one command. Combined with a fifteen-minute validity and a source restriction, the credential is close to useless outside the moment it was issued for. Arbitrary names are allowed on both sides using critical:name=contents and extension:name=contents, with a domain suffix by convention, but note that no standard options are valid for host certificates at present.

Revoking, and reading the refusal

A Key Revocation List is a binary file listing keys and certificates that must be refused, and it is designed to stay small: the manual notes it can take as little as one bit per certificate when revoking by serial number. That efficiency is the reason to allocate serials densely and monotonically from a counter rather than randomly.

# Revoke one serial against a specific CA, then verify the result.
CA_PUB=/etc/ssh/user_ca.pub
KRL=/etc/ssh/revoked.krl
printf 'serial: 1001\n' > /tmp/revoke.spec
ssh-keygen -k -f "$KRL" -s "$CA_PUB" -z 1 /tmp/revoke.spec
ssh-keygen -Q -f "$KRL" alice-cert.pub

The specification file accepts serial: for a number or a range, id: for a key identifier, key: for a whole public key, and sha1:, sha256: or hash: for fingerprints. Adding to an existing list rather than replacing it uses -u. Querying with -Q has an exit status designed for scripting: a zero status is returned only if no key was revoked, so a non-zero result means the certificate you tested is on the list.

$ ssh-keygen -Qf revoked.krl alice-cert.pub
alice-cert.pub (alice@lab): REVOKED

Server-side, the checks run in a fixed order, and a failure at any one of them produces the same message for the user.

flowchart TD
    A["Certificate presented"] --> B{"Signed by a trusted CA?"}
    B -- "no" --> X["Refuse"]
    B -- "yes" --> C{"On the revocation list?"}
    C -- "yes" --> X
    C -- "no" --> D{"Inside the validity window?"}
    D -- "no" --> X
    D -- "yes" --> E{"Principal permitted here?"}
    E -- "no" --> X
    E -- "yes" --> F["Apply options, then admit"]

Three of those refusals look identical to the client and completely different in the log. Every one of them presents the user with Permission denied (publickey,keyboard-interactive)., which is why the log is the only place to diagnose from.

Certificate invalid: name is not a listed principal
Certificate invalid: expired
Authentication key ED25519-CERT SHA256:... revoked by file /tmp/revoked.krl

Repeated failures also trigger OpenSSH’s per-source penalties, on by default since 9.8, which log a line beginning srclimit_penalise: and defer further attempts from that address. During a broken rollout this looks like a network problem and adds delay to every diagnostic attempt, so read it as a symptom of the failures rather than as the fault.

Production discipline

  1. Never sign without -V. Make the issuing tool refuse to run without a validity interval rather than relying on the operator to remember one.
  2. Allocate serials from a counter and record them. A serial you did not store is a certificate you cannot revoke compactly, and the default value of zero is not revocable by serial at all.
  3. Put the request in the key identifier. Ticket, requester and purpose, so the server log answers the access review without a second lookup.
  4. Clear the defaults, then grant. Start every issuance from -O clear so that agent forwarding, port forwarding and a pty are decisions rather than leftovers.
  5. Rehearse KRL distribution before you need it. Time how long a revocation takes to reach the last host, then set certificate validity shorter than the number that frightens you.

Cross-course references

  • Linux for Production Sysadmins - Part XXXI (Audit) covers the host audit subsystem, which records what a certificate holder did once the key identifier has established who they were.
  • Observability for Production Sysadmins - Part XVIII (Alerting Rules) covers turning a validity horizon into a page, which is how a long-lived host certificate stops becoming an outage.
  • Git, CI/CD & GitOps for Infrastructure Engineers - Part XCIII (Credential Rotation) covers scheduled rotation of machine credentials, the pattern a short certificate lifetime automates.

Quiz

Knowledge check · 4 questions

  1. Q1. A certificate is signed with ssh-keygen -s and no -V flag. What is its validity?

  2. Q2. On current OpenSSH, a user certificate with an empty principals list is treated as matching any principal.

  3. Q3. Why does issuing without -z limit your revocation options later?

  4. Q4. Contain an incident where issued certificates must stop working immediately.

    At 09:12 UTC a signer host is found compromised. It issued user certificates with eight-hour validity, serials 8400 to 8462, over the previous six hours. RevokedKeys is configured on all hosts but the current KRL was last regenerated in March and is distributed by a configuration management run that completes in about forty minutes.

Passing score: 75%. Answers are checked in this browser.