A deploy job loses SSH access halfway through, and one host lost it four minutes early

Reported symptoms

This estate replaced per-host authorized_keys files with an SSH certificate authority fourteen months ago. A small issuance service signs a user certificate for the deploy identity on request, sshd trusts the authority through TrustedUserCAKeys, and no host holds a copy of anybody public key. It has been the least interesting part of the platform since the migration.

At 08:58:04 UTC the deploy job logged its first failure of the day. The incident channel filled up faster than the facts did:

• 08:58, web-07.lab.example refuses the deploy connection. The client reports Permission denied (publickey,keyboard-interactive) and nothing else.
• 09:00, an engineer opens an interactive session to web-07 from a laptop and it works, which is read as proof that SSH on the host is fine and the pipeline account is broken.
• 09:01, web-07 is drained and queued for a rebuild. The channel agrees the incident is contained to one host.
• 09:02, every remaining host in the run fails, all of them within one second of each other.
• 09:04, somebody asks whether anything had changed at 09:02, and the answer is no. Nothing changed at 09:02. Something stopped.

The rebuild of web-07 was cancelled at 09:06. By then the run had completed eleven of forty hosts, the migration it was carrying was half applied, and the team had spent twenty minutes investigating a host whose only fault was that its clock was four minutes fast.

Evidence provided

$ journalctl -u ssh --since 09:00 --until 09:05 -o cat --grep 'Certificate invalid'

Certificate invalid: expired
Certificate invalid: expired
Certificate invalid: expired

$ ssh -o BatchMode=yes deploy@web-12.lab.example true

deploy@web-12.lab.example: Permission denied (publickey,keyboard-interactive).

The two blocks are the same event. The server knows exactly what is wrong and writes it down. The client is told only that the server declined, because telling an unauthenticated peer why its credential was rejected is an information leak. If your incident process starts from client output, certificate faults will always look like account faults.

$ ssh-keygen -L -f /run/deploy/deploy-cert.pub

/run/deploy/deploy-cert.pub:
      Type: ssh-ed25519-cert-v01@openssh.com user certificate
      Public key: ED25519-CERT SHA256:Rn8vQ2cLtY6wKdA0fJmX5pBhZ3sEo7NgUcT1iVqM9bY
      Signing CA: ED25519 SHA256:6x39cg8OAp7PZgisHreCLRO7g0vp6s0VZIw+DcTgKtE (using ssh-ed25519)
      Key ID: "pipeline@deploy-7741"
      Serial: 40219
      Valid: from 2026-08-26T07:57:11 to 2026-08-26T09:02:11
      Principals:
              deploy
      Critical Options: (none)
      Extensions:
              permit-X11-forwarding
              permit-agent-forwarding
              permit-port-forwarding
              permit-pty
              permit-user-rc

# ca-1.lab.example, issuance log for run deploy-7741
2026-08-26T08:02:11Z Signed user key deploy-cert.pub: id "pipeline@deploy-7741" serial 40219 for deploy valid from 2026-08-26T07:57:11 to 2026-08-26T09:02:11

One line, one signing, one serial. The authority was never asked for a second certificate during the eighty-one minutes the job ran.

# /etc/ssh-ca/issue.conf on ca-1.lab.example
identity_template = pipeline@%RUN_ID%
principals        = deploy
validity          = -5m:+1h
extensions        = default

The validity interval is the only line in that file that matters today, and it is written in the same notation the signing tool accepts. The value before the colon is where the window opens, expressed as an offset from the moment of signing, and the value after it is where the window closes. Both are converted to absolute timestamps at signing time and written into the certificate, which is why the certificate carries two dates and no duration at all. Nothing downstream of the authority can widen a window that has already been signed.

# pipeline log, run deploy-7741 (runner clock)
08:02:03  run start, 40 hosts queued
08:02:11  certificate acquired, serial 40219
08:02:14  step 1 of 3: package roll, hosts 1-11, complete 08:14:52
08:14:55  step 2 of 3: schema migration, single host, complete 08:57:58
08:58:04  step 3 of 3: service roll, host 12 web-07.lab.example FAILED
09:02:11  host 13 web-12.lab.example FAILED
09:02:11  host 14 web-13.lab.example FAILED

$ journalctl -u ssh --since 08:55 --until 09:05 -o short-iso-precise --grep 'Certificate invalid'

2026-08-26T09:02:11.418632+00:00 web-07 sshd-session[3311]: Certificate invalid: expired

Work the evidence before reading on

The interesting thing is not that a certificate expired. Certificates are supposed to expire. It is that two hosts refused the same certificate two hundred and forty-seven seconds apart, and that the certificate itself carries only one expiry time.

1. Take the issuance line and the validity interval in issue.conf and reconstruct the window arithmetically. Where do 07:57:11 and 09:02:11 come from, and how long is the window really?
2. Compare the job step durations in the pipeline log against that window. At what point in the run did the credential stop being valid, and how many hosts were still queued?
3. The runner logged the web-07 failure at 08:58:04. web-07 logged the same connection at 09:02:11. Only one of those clocks can be right relative to the certificate. Which one, and how do you tell?
4. Every other host failed at 09:02:11 by its own clock. What does that agreement rule out, and what does it leave?

Before continuing: state which of the two defects made this an outage and which one only made it confusing, and say what would have happened on 25 August, when the same pipeline ran for twenty-two minutes.

Root cause

The validity window was sized against the median job, not the longest step

The issuance template signs with -5m:+1h. That start offset back-dates the certificate five minutes before signing, and the end offset runs one hour forward from signing, so the real window is sixty-five minutes wide and it begins the moment the authority signs, not the moment the job starts using it. Serial 40219 was signed at 08:02:11 and therefore ceased to be valid at 09:02:11. The run took eighty-one minutes because step two, a schema migration, took seventy-one minutes on its own. Median job duration in this pipeline is thirty-one minutes, which is why the window had never been tested. Nothing failed at 09:02:11. Something that had been failing to renew since 08:02:11 finally became visible.

The pipeline fetched a credential once and treated it as a constant

There is no renewal logic anywhere in the run. The certificate is acquired at job start, written to /run/deploy/deploy-cert.pub, and read by every subsequent SSH invocation. A credential with a lifetime shorter than the process that holds it must be refreshed by that process, and this one had no code path that could refresh it. That is the defect which made a correctly enforced expiry into an outage.

web-07 verified the same window against a clock that was fast

Certificate validity is an absolute interval expressed in seconds since the Unix epoch. Both endpoints are timestamps, not durations, so every verifier compares them against its own idea of now. web-07 had been rebuilt with its time synchronisation unit masked and its clock was two hundred and forty-seven seconds fast, so it reached 09:02:11 while the rest of the world was at 08:58:04. It refused a certificate that was still valid, first, and alone. That is why the first twenty minutes of the incident went into a host that had nothing to do with the outage.

Resolution

1. Check the clock on ca-1.lab.example before issuing anything. A signing host whose clock is fast produces certificates that expire early on every correctly synchronised verifier at once, which turns a single host incident into a fleet incident. Confirm convergence against the reference, not merely that the service is running.
2. Restore service by having the pipeline acquire a new certificate and resume from the last completed host. The migration in step two is already applied, so the resumed run must skip it rather than repeat it. Record the new serial before the run starts.
3. Do not bypass certificate authentication to unblock the run. Adding the deploy public key to an authorized_keys file replaces a credential the authority can stop issuing with one that has to be removed from every host by hand, and estates that do this once are still finding the leftovers years later.
4. Add renewal to the pipeline. The correct shape is to request a certificate immediately before each SSH step, or to read the remaining validity and reissue whenever fewer than five minutes remain. Write the threshold as a named constant next to the validity interval so the two are reviewed together.
5. Unmask and start the time synchronisation unit on web-07, then wait for the offset to converge before returning the host to service. A host that reports the correct time within a second of being started has not converged; it has been stepped.
6. Audit the fleet for the same masked unit. The mask came from a build image, so it will be present on every host built from that image since the change, and web-07 is simply the first one to have been asked a question that depends on time.
7. Shorten the validity interval once renewal is in place. With per-step issuance, a window of minus five minutes to plus fifteen minutes costs nothing and limits the value of a stolen certificate file to a quarter of an hour.
8. Make the issuance service reject a request that carries no validity interval. Signing without an interval does not produce a default of one hour; it produces a certificate valid from the Unix epoch to the distant future, which is the worst possible outcome of a code path nobody tested.

Verification

1. Read the sshd journal on a target host and confirm it logs an accepted certificate naming the key identity, the serial and the signing authority. The server saying it accepted the credential is a different observation from the client saying the session opened, and only the first one proves certificate authentication was used.
2. Confirm the serial in that server log line is the newly issued one and not 40219. A resumed run that still presents the old serial has not picked up the new certificate, it has simply reached hosts that had not yet been tried.
3. Inspect the new certificate and check that its validity window opens after the previous window closed, and that the width matches the interval currently configured. Two overlapping windows mean the reissue happened but the old file was not replaced.
4. Compare host clocks from outside the hosts. Have every host log a line in response to one broadcast event and read the timestamps centrally; a host that disagrees with its peers is the one to fix, and this method does not depend on trusting the host under test to report its own offset.
5. Run a job deliberately engineered to exceed the validity window, with each issuance logged. It must renew mid-run and complete. A run shorter than the window proves nothing at all, because that case was already working the day before the incident.
6. Send an issuance request with no validity interval and confirm the authority refuses it. If it succeeds, inspect what it produced before doing anything else, and revoke it.
7. Re-run the failed hosts from step three of the original job and confirm the service roll completes, then check the migration state on the single host that ran step two to make sure the half-applied state was resolved rather than left.

Prevention

• Renew rather than extend. Issue with a fifteen minute window and reissue when fewer than five minutes remain. Short credentials are only painful when something has to notice they are expiring, and the something should be code, not an engineer at 09:02.
• Alert on clock offset as a first class signal. Warn at one second of divergence and page at five, on every host and especially on the certificate authority. Treat a masked time synchronisation unit as an image defect and fix it in the image.
• Refuse issuance without an explicit interval. An unqualified signing operation yields a certificate valid from the Unix epoch to the distant future. The authority is the right place to make that impossible, because the caller will eventually get it wrong.
• Log issuance and detect stale presentation. Record serial, identity, principals and window for every signature, and alert when a serial is presented more than fifteen minutes after signing. That single rule detects any client that has stopped renewing, before a long job finds it.
• Grant only the extensions the workflow uses. The default five include agent forwarding, port forwarding and X11. A deploy identity that needs none of them should be issued with them cleared, so a stolen certificate cannot be used to pivot.
• Review the window against the duration distribution monthly. Compare the ninety-ninth percentile job duration against the configured interval in the same meeting. If the two are never reviewed together, the window will always be sized against the median.
• Fail a run that cannot renew, rather than letting it continue. A job that has lost its credential should stop at the next step boundary with a clear message, not attempt twenty-eight more hosts and report twenty-eight separate authentication failures. The distinction matters at three in the morning, when the difference between one error and twenty-eight is whether anybody reads them.