Skip to main content
RunBook Academy

← All break/fix scenarios in Docker & Containers

intermediateRuntime~20 min

Break/Fix 20: Token and TLS validation fail on one host after a VM restore

Reported symptoms

  • ●Authentication fails on one host only: "token used before issued", "JWT nbf claim is in the future".
  • ●Outbound TLS to an internal service fails with "certificate is not yet valid" against a certificate issued minutes ago.
  • ●Log lines from this host arrive in the central store minutes ahead of everything else, so request correlation across services breaks.
  • ●Someone previously added an NTP client to the application image to "fix time", and it changed nothing.

Evidence

  • · `docker exec app date -u` and `date -u` on the host return the same wrong time, which rules out the container as the source.
  • · `timedatectl` shows `System clock synchronized: no` and `NTP service: inactive`.
  • · `chronyc tracking` reports `Leap status: Not synchronised` and a system-time offset of several minutes.
  • · `readlink /proc/1/ns/time` inside the container is identical to the host's, confirming there is no separate container clock to fix.
Diagnosis and resolutionclick to reveal

Root cause

Containers do not have their own wall clock. Docker does not place them in a separate time namespace, and Linux time namespaces virtualise only CLOCK_MONOTONIC and CLOCK_BOOTTIME — CLOCK_REALTIME is deliberately not namespaced — so every container on a host reads that host's clock and nothing else. This host is a VM that was restored from a snapshot; its chronyd never re-synchronised afterwards and the clock ran several minutes fast. Certificate `notBefore` checks and JWT `nbf` and `iat` claims are absolute-time comparisons, so a fast clock rejects credentials that are perfectly valid, and the errors name the issuer rather than the reader.

Remediation

Fix time on the host, not in the image. Confirm a reachable source with `chronyc sources -v`, then `systemctl enable --now chrony` and let it slew, or `chronyc makestep` for a one-off step when the offset is too large to slew and the workload tolerates a discontinuity. Remove the NTP client from the container image: without CAP_SYS_TIME its clock_settime call is denied, and with CAP_SYS_TIME it would be silently resetting the clock for every other container on the box. Restart any process that cached a bad notion of time.

Verification

`timedatectl` reports `System clock synchronized: yes`. `chronyc tracking` shows Leap status Normal and a system-time offset within a few milliseconds. `docker exec app date -u` matches an external reference. The authentication and TLS failures stop with no change to the identity provider, the CA, or any certificate.

Prevention

Monitor clock offset as a first-class host metric and page on `System clock synchronized: no` — node_exporter exposes node_timex_sync_status and node_timex_offset_seconds for exactly this. Never run an NTP client inside a container. Configure the hypervisor's guest time handling and re-check time after every snapshot restore, suspend or resume. Allow a small validation leeway (30-60 seconds) on token `nbf` and `iat` checks so ordinary skew degrades gracefully instead of becoming an outage.

Reported symptoms

One host out of nine starts rejecting logins. The application log is specific and completely misleading:

ERROR auth: token validation failed: token used before issued
  (iat=2026-08-11T14:02:07Z, now=2026-08-11T13:58:12Z)
ERROR upstream: tls: failed to verify certificate: x509: certificate
  has expired or is not yet valid: current time 2026-08-11T13:58:12Z
  is before 2026-08-11T14:01:44Z

The team pages the identity platform team, then re-issues an internal certificate that was not broken. Both errors name a remote component. Neither is about a remote component.

Diagnosis

Two numbers in that error message are the answer, and they are sitting next to each other. now is four minutes behind iat, and the certificate notBefore is in the future. A clock disagrees with the rest of the estate, and only one clock is involved.

Ask the container and the host the same question:

docker exec app date -u
date -u

If those agree with each other and disagree with the world, the container is not the problem and never could have been:

timedatectl
chronyc tracking
               Local time: Tue 2026-08-11 13:58:12 UTC
System clock synchronized: no
              NTP service: inactive

Reference ID    : 00000000 ()
Stratum         : 0
Leap status     : Not synchronised
System time     : 243.881 seconds fast of NTP time

Four minutes fast, no sync, no NTP service running. Correlate with the change log: this VM was restored from a snapshot the previous evening, and chronyd was not restarted afterwards.

Confirm that there is no container-level clock to fix:

docker exec app readlink /proc/1/ns/time
readlink /proc/1/ns/time

Both return the same inode. The container shares the host’s time namespace — and even if it did not, that would not help.

Resolution path

  1. Confirm the host has a reachable time source before you change anything.
  2. chronyc sources -v — a host that cannot reach any NTP server will not stay fixed, and stepping it manually just resets the countdown.
  3. If every source is unreachable, this is a firewall or routing finding (UDP 123, or NTS on TCP 4460) and belongs to the network team.
  4. Start the time service.
  5. systemctl enable --now chrony (or chronyd/systemd-timesyncd, per distribution).
  6. Decide between slewing and stepping.
  7. Slewing corrects gradually and never moves time backwards; it takes roughly 20 minutes per second of offset, so four minutes takes far too long here.
  8. sudo chronyc makestep applies the correction immediately.
  9. Restart processes that cached the wrong time.
  10. Some TLS stacks, token caches and schedulers read the clock once. docker compose restart the affected services after the step, in a rolling fashion.
  11. Remove the NTP client from the image.
  12. It has never worked and it hides the fact that nothing is managing time. Delete the package and the entrypoint call, and rebuild.
  13. Close the loop on the snapshot restore.
  14. Add "verify timedatectl output" to the VM restore runbook, and enable the hypervisor guest time-sync or RTC handling for this guest.

Verification

  1. The host is synchronised. timedatectl reports System clock synchronized: yes and a running NTP service.
  2. The offset is small and stable. chronyc tracking shows Leap status Normal and a system-time offset in milliseconds, re-checked five minutes later.
  3. The container agrees. docker exec app date -u matches an external reference.
  4. The original errors are gone with no change to the identity provider, the CA, or any certificate — that is what proves the diagnosis rather than a coincidence.
  5. The rest of the fleet is checked. Run the same timedatectl check on every host; a snapshot restore is rarely done to exactly one machine.

Prevention

  • Alert on clock sync state, not just on clock offset. timedatectl reporting System clock synchronized: no is a binary, unambiguous precursor to this incident and is trivially scrapeable via node_timex_sync_status.
  • Never install an NTP client in a container image. It cannot work, and its presence convinces the next reader that time is handled.
  • Put a time check in the VM restore and migration runbooks. Snapshot restore, suspend/resume and live migration are the three events that produce this failure, and all three are planned work with a checklist attached.
  • Give token validation a small leeway. Thirty seconds of tolerance on nbf and iat costs nothing in security terms and converts routine sub-minute skew from an outage into a non-event.