One host rejects every valid certificate and every live token

Reported symptoms

batch-07 runs the nightly reconciliation. It reads from a message broker over mutual TLS, writes results to an object store, and takes its credentials from the secret manager. It has done this every night for two years.

At 02:07 UTC it fails at start-up. The agent logs a TLS verification failure against the broker, and the failure names the validity window of the certificate the broker presented. Nobody believes it, because the broker is serving forty other hosts in the same minute and none of them have complained.

Over the next hour the picture gets stranger rather than clearer:

• The internal CA issuance rate for the workload identity batch-07 climbs from one certificate a day to roughly forty in an hour. Something on that host is asking for a new certificate over and over.
• Every signed upload from batch-07 to the object store is rejected with an error naming the difference between the request timestamp and the server time. This is the point at which the investigation acquires a second team and a second theory.
• A log query for batch-07 between 01:30 and 02:30 UTC returns nothing. Not an error, not an empty result from a broken shipper. Nothing, while the host is visibly producing errors.
• Every Prometheus panel for batch-07 looks completely normal. Rates, saturation, error counts, all indistinguishable from its peers.
• Nothing has been deployed to the host for eleven days. The only change on the record is a reboot onto a newly commissioned hypervisor at 21:40 UTC, done as part of a capacity rebalance and considered routine.

Three subsystems, three teams, and one host. The reconciliation misses its window and the morning reports go out incomplete.

Evidence provided

The cheapest question nobody had asked yet is what time it is.

$ for h in batch-07 peer-03 peer-04; do
printf '%s ' "$h"
ssh "$h" date -u
done

batch-07 Thu Aug 27 08:16:04 UTC 2026
peer-03 Wed Aug 26 02:04:11 UTC 2026
peer-04 Wed Aug 26 02:04:12 UTC 2026

Thirty hours and twelve minutes. Everything that follows is a consequence, but the consequences are worth reading individually because each one looked like a separate incident.

Here is the certificate the agent was issued at 01:58 UTC, six minutes before it started failing.

$ openssl x509 -in /etc/pki/workload/batch-07.crt -noout -subject -serial -dates

subject=CN=batch-07.internal.example.com
serial=4E71C8A5D3026FB9147E0AC3852D6F10
notBefore=Aug 26 01:58:00 2026 GMT
notAfter=Aug 27 01:58:00 2026 GMT

One file, one command, two hosts, two answers. This pair is the entire diagnosis.

$ openssl x509 -in /etc/pki/workload/batch-07.crt -noout -checkend 0

Certificate will expire

$ openssl x509 -in /tmp/batch-07.crt -noout -checkend 0

Certificate will not expire

The certificate itself is sound, which rules out the CA rotation that happened two days ago and which was the first hypothesis in the channel.

$ openssl verify -CAfile /etc/pki/ca/root-g1.crt -untrusted /etc/pki/ca/issuing-ca-g2.crt /tmp/batch-07.crt

/tmp/batch-07.crt: OK

The secret manager tells the same story from the other side. The lease the agent has been discarding is alive and has most of its life left.

$ bao lease lookup database/creds/batch-readonly/nRk4Vt2QwLpZ8mCd7XeYbHs3

expire_time     2026-08-26T02:22:31.117402913Z
issue_time      2026-08-26T02:02:31.117402742Z
renewable       true
ttl             18m20s

Two diagnostic commands remain, and neither of them has output worth quoting because what matters is what you look for. On batch-07, ask the time daemon whether it considers the system clock synchronised, and ask it what sources it can reach:

$ timedatectl status

The daemon is running. It reports the clock as unsynchronised. And on the hypervisor, the same question has the same answer, which is where the trail ends:

$ ssh hv-11 timedatectl status

Work the evidence before reading on

One host, three subsystems, and no bad credential anywhere in the evidence.

1. The same certificate file answers checkend 0 differently on two hosts. What is checkend actually comparing, and what does that tell you about where a validity decision is made?
2. The lease has eighteen minutes left according to the secret manager, and the agent on batch-07 keeps re-authenticating. The lease record contains an absolute expiry timestamp. What does the agent do with it?
3. Prometheus panels for batch-07 are normal while its logs are missing from the incident window. What is different about how each of those two signals acquires its timestamp?
4. The object store, the broker and the CA are operated by three different teams and all three rejected this host. What is the single property they have in common, and why does that property make the failure total rather than partial?

Before continuing: the workload certificate is valid for twenty-four hours and the clock is thirty hours out. State the general relationship between credential lifetime and skew tolerance, and say which way the estate has been trading it.

Root cause

The hypervisor was commissioned without a time path

hv-11 joined the estate three weeks ago in a capacity expansion. Its data-plane networking was tested, its storage was tested, and its management network was configured from the standard template. That template routes to the monitoring and configuration endpoints. It does not route to the internal time servers, which sit in a different segment, and nothing in the acceptance procedure checked.

So hv-11 has been holding whatever wall-clock value it was given at first power-on, thirty hours and twelve minutes ahead of reality. It has hosted no guests until now, so nothing has noticed. batch-07 was rebooted onto it at 21:40 UTC and took its initial clock from the hypervisor, which is what a guest does at boot.

That is the whole fault. Everything else in this incident is a correct rejection.

The guest declined to correct itself, quietly

The time daemon on batch-07 started, measured an offset of more than a day, and did nothing about it.

That is not a bug. The fleet default was written for the database estate, where stepping a clock underneath a running transaction is genuinely dangerous, so the policy is to slew after start-up and to refuse corrections beyond a bounded size. The daemon applied that policy, wrote a line to the journal explaining that it would not correct an offset of this magnitude, and settled into a healthy running state. Every process-level check in the estate reports it as fine, because it is fine. It is simply not doing the one thing its name suggests it is doing.

This is why the fault survived four and a half hours of investigation rather than resolving itself at the next poll.

Every validity window is evaluated against the local clock

The broker, the CA, the object store and the secret manager did not coordinate to reject this host. They have no relationship with each other. What they share is that each of them expresses authority as a window of time, and windows of time are compared against whichever clock is doing the comparing.

The certificate the broker presented was rejected by batch-07 because its notAfter lies in the past according to batch-07. The certificate batch-07 itself was issued was discarded by its own agent for the same reason, six minutes after it was issued, which produced the renewal loop: request a certificate, receive a twenty-four hour certificate that is already six hours expired according to the local clock, discard it, request another. Forty times an hour.

The secret-manager lease failed differently but for the same reason. The lease record carries an absolute expire_time, and an agent that decides when to renew by comparing that timestamp with its own clock sees 2026-08-26T02:22:31Z as more than a day in the past. It concludes the credential is dead on arrival and re-authenticates on every call. The object store rejected the uploads because a signed request carries the time it was signed and the server enforces a tolerance window around it; thirty hours is outside every such window ever configured.

The logs went missing because the host stamped them

The last symptom is the one that made the incident hard to search. A log line carries the timestamp written by the process that emitted it. Every line batch-07 produced during the incident is filed thirty hours in the future, so a query bounded to the incident window contains none of them.

Metrics behaved differently because they are stamped by the scraping server rather than the target, which is why every panel looked normal. Two signals, two clocks, and only one of them was wrong.

Resolution

1. Take batch-07 out of the work rota before touching its clock. Correcting by thirty hours moves the wall clock discontinuously under anything in flight, which corrupts the timestamps of running work and can trip timers and timeouts in ways that are tedious to unpick. A batch host drains cleanly; use that.
2. Fix the hypervisor first, because it is the source. Give hv-11 a route from its management network to the internal time servers and confirm the path with a measurement that reports without adjusting the clock. Repairing the guest while the hypervisor is still wrong means the next guest to boot there inherits the same fault.
3. Correct the guest clock inside the drained window. An offset of thirty hours cannot be slewed in any useful period, so it must be stepped. Record the exact time of the step in the incident log: you are deliberately inserting a discontinuity into this host's history, and whoever reads its logs next year will otherwise treat it as evidence.
4. Let the renewal loop settle and then account for it. Collect the serials of the certificates the agent consumed during the loop, confirm each is a twenty-four hour certificate that ages out on its own, and record the list. Do not attempt a mass revocation: nothing in this estate enforces one, and the lifetime already bounds the exposure more tightly than a CRL would.
5. Restore the workload and confirm it authenticates rather than assuming it will. The agent must obtain one certificate and keep it, and must obtain one lease and renew it rather than re-authenticating on every call.
6. Change the image default so a large offset is corrected at start-up. Slewing after start-up is the right policy for a running database host; refusing to correct a day of error at boot is not, and the two are separable settings.
7. Sweep every other guest on hv-11 and every host commissioned from the same template in the last quarter. The template is the defect, so the blast radius is the template, not the host.
8. Close the object store and log-shipping tickets against this incident so the next engineer searching for a rejected signed request finds the cause rather than a fourth theory.

Verification

1. Re-run the cross-host time reading, and run it across every guest on hv-11 rather than only the repaired one. All of them must agree with the fleet median to within a second. This is the check that does not trust the time daemon, which is the component that failed to act.
2. Confirm the guest daemon now considers the clock synchronised and can reach at least two sources, and confirm the same on the hypervisor. Both statements are needed: a guest can be correct today and inherit a wrong clock at the next boot.
3. Ask the same certificate file the same question on both hosts. The checkend result must now be identical on batch-07 and on peer-03, which is the direct inverse of the evidence that identified the fault.
4. Establish a real TLS connection from batch-07 to the broker and confirm the handshake completes with verify return code 0 against the production trust store. The clock being right and the credential path working are separate claims.
5. Watch the CA issuance rate for the batch-07 identity return to one certificate per day. A rate that stays elevated means the agent is still discarding valid material, which would point at a cached expiry rather than at the clock.
6. Upload a file to the object store with a signed request and confirm it is accepted. This is the independent third-party check, made by a system that has no knowledge of your PKI at all.
7. Emit a deliberate log line from batch-07, note the wall time, and confirm it lands in the correct window in the log store. Historical lines stay where they were filed; a corrected clock does not repair them.
8. Skew a staging host on purpose and confirm the new clock-sanity check alerts within its window. A guard that has only ever been green has never been tested.

Prevention

• Commission a time path before a hypervisor accepts guests. A hypervisor is a time source for everything that boots on it, so its synchronisation belongs on the acceptance checklist beside networking and storage, with a recorded proof rather than an assumption.
• Alert on whether the clock is disciplined, not on the offset. A daemon that cannot reach a source stops measuring, and a metric that stops being updated looks exactly like a metric reporting zero error. The useful question is whether anything is steering this clock at all.
• Compare hosts with each other on a schedule. One sweep that reads the time from every host and flags any that disagrees with the median by more than a second catches this whole class of fault, including on the hosts nobody added to monitoring, which are exactly the hosts that drift.
• Make the workload fail fast and say why. An agent that compares its own clock against the notBefore of the certificate it was just issued, and refuses to start when that instant is in its future, converts four hours of cross-team investigation into one legible log line.
• Treat skew tolerance as an input to credential lifetime. The shorter the validity, the smaller the clock error the estate can absorb. Twenty-four hour certificates are a good control and they make a thirty-hour skew an instant total failure. Decide that consciously and set the monitoring threshold accordingly.
• Put a clock check in the post-migration and post-restore checklists. Both operations set a guest wall clock from somewhere other than the network, and both are performed under time pressure by people who are thinking about something else.
• Linux for Production Sysadmins - Part XXIV (Time) covers chrony architecture, client configuration, guest timekeeping and the operational impact of skew, which is the depth this bullet list deliberately does not attempt.