Time skew operational impact - what breaks when clocks drift

Time skew causes some of the most confusing production incidents because the symptoms look like network, security, or application failures. Knowing what time skew breaks - and how to recognise it from error messages - is the difference between a five-minute fix and a half-day investigation.

What breaks

TLS certificates

TLS certificates have a notBefore and notAfter field. A client (or server) rejects certificates if the current time is outside this range. With 5 minutes of skew, certificates fail to validate:

error: x509: certificate has expired or is not yet valid: current time
       2026-08-09T14:30:00Z is before 2026-08-10T00:00:00Z

OpenSSL words the same condition as certificate is not yet valid or certificate has expired.

Kerberos

Kerberos uses timestamps for ticket validity. The default allowed clock skew is 5 minutes:

kinit: Clock skew too great while getting initial credentials

The fix is to synchronise clocks; do NOT relax the configuration.

Logs

When timestamps are off across hosts, log correlation becomes impossible:

host1: 2026-08-09 14:30:00 ERROR ...
host2: 2026-08-09 14:25:00 ERROR ...    (5 minutes earlier on host2)

Investigating an incident across multiple hosts becomes guesswork.

Distributed databases

Distributed databases use timestamps for ordering. A skewed clock can cause writes to be rejected or reordered:

Cassandra: Write request failed: OperationTimedOut
Cassandra: Timestamp out of range

Cron and systemd timers

A skewed clock causes jobs to run at unexpected times:

• A backup runs at 14:30 local time but the host thinks it is 14:35, so the backup is logged at the wrong time.
• A maintenance job runs twice because the system clock jumps back.

Two-factor authentication (TOTP)

Time-based OTP codes (RFC 6238) assume client and server clocks are within 30 seconds. A 1-minute skew causes authentication failures:

TOTP validation failed

Database replication

Some databases (MySQL, PostgreSQL with logical replication) require monotonic time. A clock jump can cause replication to fail or replay.

How to recognise time skew

Error messages:

• “Clock skew too great” (Kerberos)
• “x509: certificate has expired or is not yet valid” (TLS)
• “Timestamp out of range” (databases)
• “TOTP validation failed” (auth)

Diagnostic checks:

# Compare the local clock to a known-good reference
date -u
ssh user@ntp1.example.com date -u
ssh user@other-host date -u

# Check chrony tracking
chronyc tracking

# Check timedatectl
timedatectl status

A healthy host shows <100 ms skew from a peer. Anything

1 second is a problem.

How to detect skew in monitoring

Alert on:

• chronyc tracking | grep 'System time' returns >100 ms.
• timedatectl status | grep synchronized returns “no”.
• The NTP service is not active.

Sample Prometheus alert:

- alert: ClockSkew
  expr: abs(node_timex_offset_seconds) > 0.1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.instance }} clock offset exceeds 100 ms"
- alert: ClockSkewCritical
  expr: abs(node_timex_offset_seconds) > 5
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "{{ $labels.instance }} clock offset exceeds 5 s"
- alert: ClockNotSynchronised
  expr: node_timex_sync_status == 0
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.instance }} is not synchronised to any time source"

node_timex_offset_seconds is the offset from NTP, exposed by the node_exporter. It is the kernel timex struct’s offset field, so it is signed: a host running behind true time reports a negative value.

Alert on sync status as well as offset. When a host has lost all its sources, chronyd stops disciplining the clock and the reported offset freezes at its last value - so the offset can look perfect on a host that has been free-running for a week. node_timex_sync_status is 0 in that state; it is the alert that catches the failure the offset rules cannot see.

How to fix skew

Small skew (<1 second): NTP will correct it via slewing. Wait for NTP to catch up.

Large skew (>1 second): NTP will slew slowly. To force a step:

sudo chronyc makestep              # chrony
sudo systemctl restart systemd-timesyncd    # timesyncd

Or stop NTP, set the time manually, and restart:

sudo systemctl stop chronyd
sudo chronyd -q 'server time.cloudflare.com iburst'
sudo systemctl start chronyd

Persistent skew despite NTP: the NTP source is unreachable, or the network is filtering NTP. Check with chronyc sources or tcpdump -i eth0 udp port 123.

Common skew sources

A daylight-saving transition is not on that list, and it does not belong there. Linux holds CLOCK_REALTIME in UTC; DST changes only the local-time rendering applied at display, so no offset appears in chronyc tracking, node_timex_offset_seconds does not move, and no host becomes skewed relative to any other. If a host really does jump an hour at a DST boundary, its hardware clock is set to local time rather than UTC:

timedatectl show -p LocalRTC        # LocalRTC=yes is the defect
sudo timedatectl set-local-rtc 0    # store the RTC in UTC

NTP and security

NTP has historically had weak authentication. Attackers can spoof NTP responses and shift a host’s clock. The consequences:

• Skew-based attacks on TLS, Kerberos.
• Time-based cryptography that depends on accurate time.
• Causality confusion in logs.

Mitigations:

• Use NTS (Network Time Security, RFC 8915) where supported.
• Use authenticated NTP with shared keys.
• Restrict NTP access at the firewall.
• Allow NTP only to trusted sources.

Operational checklist

Daily:

• NTP service is active on every host.
• Clock offset is <100 ms from reference.
• No alerts for clock skew.

Monthly:

• Audit NTP source list.
• Verify NTP source reachability.
• Test skew detection by simulating (chaos test).

After incidents:

• Verify the NTP service was not affected.
• If a time jump was made, document the cause.

Knowledge check