Clock skew, validity windows and why a correct certificate fails on one host

One host in a fleet refuses a certificate that forty others accept. Nothing was deployed, the file is byte-identical everywhere, and the certificate has months left. The instinct is to reissue. The instinct is wrong, and acting on it converts a five minute time fix into an afternoon of certificate churn with no record of what actually happened.

A window is a comparison, not a property

An X.509 certificate carries two instants, and they are absolute points on the universal timeline rather than anything local:

notBefore=Aug 26 21:19:00 2026 GMT
notAfter=Nov 24 21:19:00 2026 GMT

Note the suffix. These values are expressed in universal time and the certificate contains no timezone of interest, so the timezone configured on a host has nothing to do with the outcome. A machine displaying local time in Tokyo and a machine displaying local time in Lisbon reach the same verdict provided their underlying clocks agree.

What decides the verdict is the verifier’s own wall clock at the moment of the handshake. Validity is therefore a property of a pairing, not of a file, and two verdicts can differ without either certificate being wrong.

Two directions of skew, two different symptoms

• The verifier is ahead of true time. Its clock has already passed notAfter, so a certificate with weeks remaining is reported as expired. This is the direction people recognise, because the word expired appears and matches an existing mental model.
• The verifier is behind true time. Its clock has not yet reached notBefore, so a certificate issued minutes ago is refused as not yet valid. This is the direction that wastes hours, because the certificate is provably new and everyone concludes the issuance was faulty.

Skew only matters near the edges of a window. In the middle of a ninety day certificate a four minute offset is invisible. That is precisely why these failures cluster around change events: issuance, renewal cutover, and the first boot of a rebuilt host.

Freshly issued credentials fail first

The notBefore value is typically the instant of issuance. A verifier even thirty seconds behind the issuer will therefore reject a credential the moment it is handed over, and the credential will start working on its own a little later, which produces the least helpful bug report in operations: it fixed itself.

Mature tooling anticipates this by backdating. Consider a real SSH certificate issuance, where the validity argument asks for a window starting five minutes in the past:

ssh-keygen -s user_ca -I "alice@runbook-lab" -n deploy \
  -V -5m:+1h -z 1001 alice.pub

The issuance reports the resulting window:

Signed user key alice-cert.pub: id "alice@runbook-lab" serial 1001 for deploy valid from 2026-08-26T21:17:09 to 2026-08-26T22:22:09

Sixty five minutes wide for a one hour credential. Those extra five minutes are not generosity; they are an explicit tolerance for the clocks of hosts that the issuer does not control. Note also that the validity argument is not optional in practice: without one, this tool issues a credential valid from the Unix epoch to the distant future, which trades a skew problem for a much worse one.

Diagnosing skew without changing anything

Three read-only commands settle it. Run the first on the verifier and on the issuer within the same minute, and read the answers side by side.

# Run the identical command on both hosts inside the same minute.
date -u +%Y-%m-%dT%H:%M:%SZ

# Ask systemd whether this clock is disciplined at all.
timedatectl show --property=NTPSynchronized --value

# Ask chrony how far it currently sits from its sources.
chronyc tracking

The first command gives the offset directly. The second answers a different and equally important question: whether anything is keeping this clock honest, because an undisciplined clock that happens to be correct today will not be correct next month. The third gives the current correction, which distinguishes a host that has drifted from a host that has never been synchronised.

flowchart TD
    A["a credential is refused as outside its window"] --> B["compare date -u on issuer and verifier"]
    B -- "the two agree" --> C["the window really has passed: renew"]
    B -- "verifier is behind" --> D["not yet valid: correct time sync"]
    B -- "verifier is ahead" --> E["appears expired early: correct time sync"]
    D --> F["re-test the same credential unchanged"]
    E --> F

The branch that matters is the one that leads back to the same credential. If skew was the cause, the existing certificate starts working the moment the clock is corrected, with no reissue, no deployment and no reload. That is the outcome to aim for, because it leaves the estate exactly as it was.

Two environmental facts narrow the search quickly. A container does not have its own wall clock; it reads the host’s, so time inside a container is corrected on the host and nowhere else. A virtual machine that has been suspended, migrated or restored from a snapshot can resume with a clock hours behind, and until its time service catches up it will refuse credentials that the rest of the cluster accepts.

The same fault outside TLS

Skew is not a certificate problem. It is a validity problem, and almost every credential in a modern estate has a window.

SSH certificates are the worst offender because the client is told nothing useful. With verbose server logging the daemon records Certificate invalid: expired, while the user sees only Permission denied (publickey,keyboard-interactive). The word time appears nowhere on the client, so unless someone reads the server log the investigation starts in the wrong place entirely.

Secret manager leases carry issue and expiry instants, and a client whose clock disagrees will schedule its renewal at the wrong moment: too late, and the lease has already gone; too early, and it churns credentials for no reason. Dynamic database credentials add a third clock, because the database enforces the account expiry itself. A role created with an expiry recorded as 2026-08-26 21:25:48+00 disappears at that instant according to the database server, whatever the application host believes.

Signed API requests are stricter still: the signature covers a timestamp and the receiving service rejects anything outside a narrow tolerance, so a few minutes of skew produces authentication failures that look like a bad key.

The direction of travel makes this worse rather than better. Credential lifetimes across the industry are shortening deliberately, because short lifetimes are the control that actually limits the damage from a compromised key. A ninety day certificate tolerates a clock that is minutes wrong for almost all of its life. A credential measured in hours spends a much larger fraction of its existence near one of its own boundaries, and a workload identity token measured in minutes is essentially always near one. Every reduction in lifetime raises the accuracy that the verifying fleet must actually achieve, which makes time synchronisation a dependency of the security programme rather than a housekeeping detail.

Production discipline

1. Make the clock check the first step of every expiry-shaped incident. It is read-only, it takes seconds, and it is the only thing that distinguishes two very different causes.
2. Require a disciplined time source on every host that verifies anything. An unsynchronised clock is a latent outage with no alert attached to it.
3. Backdate issuance deliberately. A short tolerance on notBefore absorbs the skew of hosts you do not control, and it should be a property of the issuance profile, not of the operator running it.
4. Alert on synchronisation state, not only on offset. A host whose time service has stopped will look correct for hours before it looks wrong.
5. Never reissue as a diagnostic. A reissue in response to skew fails identically and destroys the clarity of the record.

Cross-course references

• Linux for Production Sysadmins - Part XXIV (Time) covers configuring and verifying a disciplined time source, including the behaviour of slewing against stepping that decides whether a correction is gradual or instantaneous.
• Observability for Production Sysadmins - Part CVIII (ClockSkew) covers detecting divergence across a fleet before a credential refuses, and why skewed hosts also corrupt the ordering of the telemetry you would use to investigate.
• Kubernetes for Production Sysadmins - Part LXXVI (Certs) covers a cluster whose components each verify short-lived credentials, where a single skewed node produces authentication failures that read as authorisation problems.

Quiz