NTP skew cluster impact - the silent time bomb · Linux

NTP skew is the silent time bomb in a cluster. Many services depend on synchronised time; skew breaks them silently. This lesson covers the impact and the mitigation.

What breaks with NTP skew

Kerberos: rejects tickets with timestamp skew >5 min. Auth fails cluster-wide.
TLS: certificates appear not-yet-valid or expired.
Database replication: rejects writes with timestamps in the future or too far in the past.
Cluster membership and fencing: corosync token, consensus and failure-detection timers are driven by the clock. A backward step is the dangerous case.
Distributed locks: lease expiration may be off.
Logs: timestamp correlation breaks across hosts.
Backup consistency: snapshot ordering depends on timestamps.

A cluster with even small skew (1 second) can have sporadic issues. With large skew (minutes), it is unusable.

A backward clock step fences the node

Gradual drift and a deliberate step are not the same event. Stepping the wall clock backwards by an hour makes every outstanding corosync timer appear to be an hour in the future, so the node stops sending and acknowledging tokens inside the window its peers expect. The peers declare it lost and the cluster fences it - on real hardware that means powered off mid-write, with whatever the node was committing left incomplete.

This is why time work on a cluster member is a change, not a diagnostic. Before any step, makestep, date -s, or a timezone or NTP-source change on a live member:

Put the node in standby and stop cluster services (pcs node standby node2, then pcs cluster stop node2).
Confirm with pcs status that resources have moved off.
Do the time work, then return the node to service.

Or do it on a host that is not a cluster member at all. linux-lab-time-sync-cluster walks through both safe paths.

Forward steps are survivable more often, because a timer that has already expired simply fires. They are not safe: a large forward jump expires every lease and token deadline at once, which looks to the peers like a node that went away and came back. Treat any step on a live member as the same class of change.

Symptoms

Sporadic “Clock skew too great” errors in Kerberos.
TLS handshake failures with “certificate not yet valid”.
Database replication errors with timestamps in the future.
Log entries from one host show times in the past compared to another.
Backups appear to be from the future.
A node is fenced with no application fault and no network fault, shortly after someone touched time on it. Check journalctl -u corosync for a lost token around the step.

Mitigation

Multiple NTP sources

Each host syncs to at least two NTP sources. A single source is a single point of failure.

Local NTP server

For larger clusters, run local NTP servers that sync to external sources. Cluster hosts sync to the local servers.

Chrony with leap-second handling

Chrony handles leap seconds correctly. systemd-timesyncd is simpler but may not handle all edge cases.

Monitor offset

Alert on NTP offset > 100 ms. Catching skew early prevents silent failures.

Test for skew

Stop NTP and observe. The unit name is distro-specific, so resolve it rather than guessing:

# RHEL / Fedora / SUSE : chronyd.service  and /etc/chrony.conf
# Debian / Ubuntu      : chrony.service   and /etc/chrony/chrony.conf
CHRONY_UNIT=$(systemctl list-unit-files 'chrony*.service' --no-legend \
  | awk 'NR==1{print $1}')

sudo systemctl stop "$CHRONY_UNIT"

# Watch the offset
chronyc tracking

# Restore
sudo systemctl start "$CHRONY_UNIT"

If a host with skew has Kerberos issues, the cluster recovery from skew is to resync NTP and re-authenticate.