Skip to main content
RunBook Academy

← All break/fix scenarios in Secrets, PKI & Certificates

advancedssh-ca~40 min

Two teams lose SSH access to different hosts, and the server logs the same sentence for both

Reported symptoms

  • ●At 06:41 UTC an on-call engineer responding to a page could not reach anything through jump-3.lab.example and lost eleven minutes before trying a different jump host.
  • ●The same engineer reached the same targets through jump-1.lab.example immediately, so the first report described the problem as one broken jump host.
  • ●At 07:24 UTC the deploy pipeline began failing against the application tier, while continuing to succeed against the jump tier and the database tier.
  • ●Two unrelated identities failing within an hour led the channel to conclude that the certificate authority itself had broken, which it had not.
  • ●Certificate issuance was working throughout, and every certificate examined during the incident was well inside its validity window.
  • ●Interactive logins by named engineers, who use their own keys rather than certificates, worked everywhere and continued to work for the whole incident.

Evidence

  • · The sshd journal on jump-3.lab.example carries the message Certificate invalid: name is not a listed principal for every on-call attempt from 06:41 onwards.
  • · The sshd journal on app-14.lab.example carries exactly the same message for every deploy attempt from 07:24 onwards, which rules out a per-host configuration typo as a single explanation.
  • · The client for both sets of attempts reports only Permission denied (publickey,keyboard-interactive), with nothing that distinguishes the two cases.
  • · The issuance service request log shows the oncall identity being signed with an empty principals array, while the deploy identity is signed with a principals array containing the single value deploy.
  • · The jump tier trusts the authority through an authorized keys line carrying the cert-authority marker and a principals option naming oncall, which is a different code path from the one the application tier uses.
  • · The application tier trusts the authority through TrustedUserCAKeys with an AuthorizedPrincipalsFile, and that file on app-14 now contains the single word deployer.
  • · The estate inventory shows jump-1 and jump-2 running OpenSSH 10.2 while jump-3 and jump-4 were rebuilt onto a newer image on 25 August and are running 10.3 or later.
  • · Configuration management applied a service account rename from deploy to deployer across the application tier at 07:18 UTC, six minutes before the deploy failures began.
Diagnosis and resolutionclick to reveal

Root cause

One log message, two entirely separate defects, and neither is a fault in the certificate authority. The on-call failure is a behavioural change. Six weeks ago the oncall issuance profile was created by copying the deploy profile, and the copy lost its principals key. Certificates for that identity have been signed with an empty principals list ever since. Until 25 August every jump host ran a release in which an empty principals section was treated as matching any principal when the authority was trusted through an authorized keys line, so those certificates worked by accident. OpenSSH 10.3 changed that: an empty principals section now matches nothing. Rebuilding jump-3 and jump-4 onto a newer image did not break them, it removed the permissive behaviour that had been concealing a six week old issuance defect. The deploy failure is a plain ordering mistake. Configuration management renamed the service account from deploy to deployer across the application tier at 07:18 and rewrote each AuthorizedPrincipalsFile to contain deployer, while the issuance profile continued to sign certificates carrying deploy. The server was asked to accept a principal it no longer lists. The two defects reached production in the same hour by coincidence, and because they log identically, the incident was diagnosed as one problem for over half an hour.

Remediation

Restore the paging path first, using a route that already works rather than editing server configuration under time pressure. On-call reaches its targets through jump-1 and jump-2, so direct the rota there and say so in the channel before doing anything else. Do not add the on-call public key to an authorized keys file, and do not roll jump-3 and jump-4 back to the previous image: the older behaviour is the thing that hid the defect, and restoring it puts every future rebuild back on the same cliff edge. Fix the issuance profile so the oncall identity is signed with an explicit principals list, reissue, and confirm on a host running the newer release. Treat the deploy rename as a transition rather than a switch. Add deploy back alongside deployer in the AuthorizedPrincipalsFile on the application tier so both are accepted, which restores the pipeline immediately without reissuing anything. Then change the issuance profile to sign deployer, reissue, confirm acceptance, and only then remove deploy from the principals files. Doing those three steps in the other order produces a second outage between the reissue and the file update. Do not attempt to cover the problem by signing a principal of asterisk. Wildcards in principals are supported for host certificates only and are not supported for user certificates, so an asterisk is matched as a literal name and matches nothing, and on the paths where a wildcard did once work it granted every principal at once.

Verification

Verify from the server, because the client is told only that permission was denied and cannot distinguish which of the two defects it hit. On a host running the newer release, read the sshd journal and confirm it now logs an accepted certificate naming the key identity, the serial and the signing authority, rather than logging nothing and closing the connection. Repeat the check on a host running the older release so that both trust paths are exercised, since the jump tier and the application tier reach the authority through different directives. Inspect a freshly issued oncall certificate and read its principals block directly; the issuance log showing a populated array proves what was requested, and only the certificate proves what was signed. For the rename, confirm that a certificate carrying deployer is accepted on app-14 before you remove deploy from the principals file, and confirm the removal afterwards by presenting an old certificate and watching the server refuse it. That final negative test is the one that proves the transition finished, and it is the step teams skip. Finish by confirming that a deliberately principal-free certificate is now refused everywhere, on old and new releases alike.

Prevention

Validate at issuance, because an authority is the only place that sees every certificate before a server has to judge it. Refuse to sign a request whose principals list is empty, refuse a principal of asterisk, and refuse a principals value that is not on the approved list for that profile. Any one of those rules would have stopped this incident six weeks before it happened. Test behaviour, not versions: keep one host in the estate running the newest OpenSSH release you intend to deploy, run the authentication suite against it as a gate on image promotion, and treat a suite failure as a blocked release rather than a note. Write the rename procedure down as additive first, reissue second, subtractive third, with a stated overlap of fourteen days, and require the subtractive step to reference the change that completed the reissue. Alert on the sshd journal directly: any Certificate invalid message pages during business hours, and three within five minutes pages at any time, which would have surfaced the on-call failure at 06:41 rather than at 07:52. Converge the two trust paths onto TrustedUserCAKeys with an AuthorizedPrincipalsFile so that one behaviour is in force estate-wide, and record in the runbook that this path has never accepted a certificate with no principals.

Reported symptoms

This estate runs one SSH certificate authority for machine and human access, and two different ways of trusting it. The jump tier, which predates the migration, trusts the authority through a line in the deploy and oncall account authorized_keys files carrying the cert-authority marker. The application tier, built later, uses TrustedUserCAKeys with an AuthorizedPrincipalsFile. Nobody has thought about the difference for a year, because both worked.

The incident channel filled up faster than the facts did:

  • 06:41, an on-call engineer responding to a page cannot get through jump-3.lab.example. The client says permission denied and nothing else. She tries three times, then tries jump-1.lab.example, which works, and the page is handled eleven minutes late.
  • 06:55, the report is filed as one broken jump host.
  • 07:24, the deploy pipeline starts failing against the application tier. Jump tier and database tier deployments continue to succeed.
  • 07:31, somebody observes that two unrelated identities have failed within an hour, and the channel concludes the certificate authority is broken. An engineer begins drafting a plan to reissue the authority key.
  • 07:52, the first person opens an sshd journal on an affected host.

The authority was never broken. It had been signing one of these two identities incorrectly for six weeks, and the other one correctly right up to the moment the servers stopped agreeing about what the correct principal was called.

Evidence provided

Read-only / Safejump-3.lab.example, on-call attempts from 06:41 UTC
$ journalctl -u ssh --since 06:40 -o cat --grep 'Certificate invalid'
Certificate invalid: name is not a listed principal
Certificate invalid: name is not a listed principal
Certificate invalid: name is not a listed principal

Illustrative output

Read-only / Safeapp-14.lab.example, deploy attempts from 07:24 UTC: the identical sentence, a different tier and a different identity
$ journalctl -u ssh --since 07:20 -o cat --grep 'Certificate invalid'
Certificate invalid: name is not a listed principal
Certificate invalid: name is not a listed principal

Illustrative output

Two hosts, two identities, two trust paths, one message. The message is accurate in both cases and useless for telling them apart, because it describes the outcome of the principal check rather than the reason the check had nothing to match.

{"ts":"2026-08-26T06:38:51Z","profile":"oncall","identity":"oncall@shift-3312","serial":51877,"principals":[],"validity":"-5m:+8h"}
{"ts":"2026-08-26T07:22:40Z","profile":"deploy","identity":"pipeline@deploy-7802","serial":51861,"principals":["deploy"],"validity":"-5m:+1h"}

The authority writes one record per signature. The oncall profile is signing with an empty principals array and has been doing so since the profile was created. The deploy profile is signing deploy, which is exactly what it has always signed.

Read-only / SafeDeploy runner, 07:40 UTC: an unexpired, correctly signed certificate that the application tier refuses
$ ssh-keygen -L -f /run/deploy/deploy-cert.pub
/run/deploy/deploy-cert.pub:
      Type: ssh-ed25519-cert-v01@openssh.com user certificate
      Public key: ED25519-CERT SHA256:Rn8vQ2cLtY6wKdA0fJmX5pBhZ3sEo7NgUcT1iVqM9bY
      Signing CA: ED25519 SHA256:6x39cg8OAp7PZgisHreCLRO7g0vp6s0VZIw+DcTgKtE (using ssh-ed25519)
      Key ID: "pipeline@deploy-7802"
      Serial: 51861
      Valid: from 2026-08-26T07:17:40 to 2026-08-26T08:22:40
      Principals:
              deploy
      Critical Options: (none)
      Extensions:
              permit-X11-forwarding
              permit-agent-forwarding
              permit-port-forwarding
              permit-pty
              permit-user-rc

Illustrative output

Read-only / SafeOn-call laptop: read the principals block of the certificate that jump-3 refused
$ ssh-keygen -L -f /run/oncall/oncall-cert.pub

Now the two trust paths, side by side.

# /home/oncall/.ssh/authorized_keys on the jump tier, one line, unchanged since 2025
cert-authority,principals="oncall" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5... ca@lab.example
# /etc/ssh/sshd_config.d/10-ca.conf on the application tier
TrustedUserCAKeys /etc/ssh/user_ca.pub
AuthorizedPrincipalsFile /etc/ssh/principals/%u
PubkeyAuthentication yes
PasswordAuthentication no
LogLevel VERBOSE
# /etc/ssh/principals/deployer on app-14.lab.example, written by configuration management at 07:18
deployer
# estate inventory: jump tier, 2026-08-26
host                   image              openssh   rebuilt
jump-1.lab.example     base-2026.06       10.2      2026-06-14
jump-2.lab.example     base-2026.06       10.2      2026-06-14
jump-3.lab.example     base-2026.08       10.3+     2026-08-25
jump-4.lab.example     base-2026.08       10.3+     2026-08-25
Read-only / Safejump-1.lab.example: the release on the two jump hosts that still work
$ ssh -V
OpenSSH_10.2p1 Ubuntu-2ubuntu3.5, OpenSSL 3.5.5 27 Jan 2026

Illustrative output

Work the evidence before reading on

Two failures that log the same sentence are usually one failure. These are not. The evidence separates them cleanly, but only if you stop reading the message and start reading what each side of each check actually held.

  1. The principal check compares a name the server will accept against the list of names inside the certificate. For the on-call case, which side of that comparison is empty, and where in the evidence is that shown?
  2. For the deploy case, neither side is empty. Which side changed, at what time, and what changed it?
  3. jump-1 and jump-3 run the same configuration and the same authority. What differs between them, and why would that difference change the result of a comparison against an empty list?
  4. The deploy certificate in the inspection block is unexpired, correctly signed and carries the principal it has always carried. Given that, what must be true of the server for it to be refused?

Before continuing: name the two defects separately, say which one is six weeks old, and explain why rebuilding a host made a certificate stop working without changing the certificate.

Root cause

The on-call profile has been signing certificates with no principals since July

The oncall issuance profile was created by copying the deploy profile and editing it. The edit dropped the principals key. Every oncall certificate since then has been signed with an empty principals list, which the issuance log records plainly as an empty array. Nobody noticed, because on-call engineers only ever connect through the jump tier, and on the jump tier the authority is trusted through an authorized_keys line with a principals= option. Until 25 August, a certificate with an empty principals section was treated on that path as matching any principal. The certificates were accepted for the wrong reason for six weeks.

OpenSSH 10.3 stopped treating an empty principals list as a wildcard

The rebuild of jump-3 and jump-4 moved them to a release in which an empty principals section matches nothing rather than everything. That is a deliberate change, and it also removed wildcard matching from user certificate principals while keeping it for host certificates. Nothing about the certificate changed. The permissive interpretation that had been carrying it disappeared. This is the reason jump-1 still works and jump-3 does not, and it is why rolling the image back would restore service by restoring the bug.

The deploy rename changed the server side without changing the issuer

At 07:18 configuration management renamed the application tier service account from deploy to deployer and rewrote each AuthorizedPrincipalsFile accordingly. The issuance profile still signs deploy. From 07:24 the application tier was asked to accept a principal it no longer lists, and it correctly refused. The certificate in the inspection block is unexpired and properly signed; it simply names something the server stopped recognising six minutes earlier.

Resolution

  1. Restore the paging path before anything else. On-call reaches every target through jump-1 and jump-2, so redirect the rota to those two hosts and announce it in the channel. This costs nothing, is instantly reversible, and stops the incident from delaying the next page while you work.
  2. Do not roll jump-3 and jump-4 back to the previous image. The older release accepted these certificates because of behaviour that was removed on purpose, so a rollback restores service by restoring the defect and guarantees the same outage on the next rebuild, with less context and probably at a worse hour.
  3. Add the missing principals list to the oncall issuance profile, reissue, and test against jump-3 specifically. Testing against jump-1 proves nothing here, because jump-1 would accept the broken certificate as well as the fixed one.
  4. Restore the deploy pipeline additively. Write both deploy and deployer into the AuthorizedPrincipalsFile on the application tier. This accepts the certificates already in circulation without reissuing anything, and it is the only step in this incident that fixes a tier in one action.
  5. Change the deploy issuance profile to sign deployer, reissue, and confirm acceptance on app-14 from the server journal. Until this lands, the pipeline is running on the compatibility entry you just added rather than on a correct certificate.
  6. Remove deploy from the principals files only after the reissued certificates are confirmed in use, and no earlier than the end of the longest certificate validity currently outstanding. Removing it first creates a second outage in the gap between the file change and the reissue, which is exactly what happened at 07:18.
  7. Converge the two trust paths. Move the jump tier from the authorized_keys cert-authority route to TrustedUserCAKeys with an AuthorizedPrincipalsFile, so one matching rule is in force across the estate and no host can be accidentally more permissive than its neighbours.
  8. Add issuance-time validation so this cannot recur from a profile edit: reject an empty principals list, reject an asterisk, and reject any principal not on the approved list for that profile. Deploy the rule before you close the incident, while the reason for it is still obvious to reviewers.

Verification

  1. On jump-3, read the sshd journal for the on-call connection and confirm the server logs an accepted certificate naming the key identity, the serial and the signing authority. A successful shell prompt is not the same observation; the journal is what proves certificate authentication was the mechanism.
  2. Repeat the same journal check on jump-1, which runs the older release. Both trust paths and both releases must accept the reissued certificate, otherwise you have fixed one half of a fleet and left a cliff edge on the other.
  3. Inspect a freshly issued oncall certificate directly and read its principals block. The issuance record proves what was requested; only the certificate proves what was signed, and this incident began with those two things disagreeing.
  4. Present a certificate carrying deployer to app-14 and confirm acceptance in that server journal before removing deploy from any principals file. Sequence matters here: the confirmation is what authorises the subtractive step.
  5. After the removal, present an old certificate carrying deploy to app-14 and confirm the server refuses it. This negative test is what proves the transition actually finished rather than leaving a permanently permissive file behind.
  6. Present a deliberately principal-free certificate to one host of each release and confirm both refuse it. Uniform refusal is the property you are buying with the issuance validation, and it is worth measuring once rather than assuming.
  7. Confirm the alert fires. Generate a single Certificate invalid message on a test host and check that it reaches the on-call channel, because the whole point of the change is that nobody should discover this from a missed page again.

Prevention

  • Validate principals at issuance. The authority sees every certificate before any server has to judge one. Reject an empty list, reject an asterisk, and reject names outside the profile allow list. Any of the three would have caught the on-call defect in July.
  • Keep a canary on the newest release. Run one host on the newest OpenSSH you intend to deploy, run the authentication suite against it, and gate image promotion on the result. This incident was a behavioural change shipped in a base image with no behavioural test.
  • Make renames additive, then subtractive. Write the procedure as add the new name, reissue, confirm, remove the old name, with a fourteen day overlap and a required reference from the subtractive change to the confirming one.
  • Alert on the sshd journal, not on tickets. Any Certificate invalid message pages during business hours and three within five minutes page at any time. The on-call failure at 06:41 was visible in a log for seventy minutes before a human looked.
  • Run one trust path. Two ways of trusting the same authority means two matching rules, and one of them will always be the one nobody remembers. Standardise on TrustedUserCAKeys with an AuthorizedPrincipalsFile and record that it has never accepted a principal-free certificate.
  • Do not build on permissive defaults. Behaviour that works because a check found nothing to compare is not a design, it is an accident waiting for a release note. Treat any credential that works without a positive match as a defect to be fixed today.