Clock Skew Anatomy

A TLS handshake fails for one in every two hundred requests between the application and the database. The certificate is valid. The application is not. The application host clock is 1.4 seconds ahead of true time. The certificate’s notBefore field falls just outside the application’s window for a window of every twenty-minute NTP polling cycle. The application logs the rejection as x509: certificate is not yet valid. The on-call engineer spends ninety minutes on certificate rotation before checking chronyc tracking. The system time is 1.412 seconds fast of NTP time.

This lesson is the anatomy of that 1.4 seconds. What it is. Why it accumulates. Where it lives. The fix is one package install and two server lines; the lesson is knowing what to ask the host to do.

What it is

Clock skew is the difference between the wall-clock time reported by two systems that are supposed to agree. In a production fleet the two systems are a host and an upstream NTP source, or two hosts that share no NTP source. The difference is measured in seconds at the application layer and in milliseconds at the disciplined-host layer.

Two words worth distinguishing:

• Offset — the instantaneous difference between the local wall clock and the upstream reference. chronyc tracking reports this as System time. node_exporter reports it as node_timex_offset_seconds. A healthy offset on a modern Linux host with chrony is below ten milliseconds.
• Drift — the rate at which the local clock runs faster or slower than true time. A typical x86 system clock drifts by roughly one part per million when undisciplined, which is about 86 milliseconds per day or roughly 2.6 seconds per month. Drift is what produces offset. Offset is the symptom; drift is the cause.

The common shape is the silent drift of an undisciplined host. The host has no NTP daemon, or the daemon cannot reach its upstream, or the upstream is itself misconfigured. The clock runs at the crystal’s natural rate. The offset grows linearly. Nothing in the application stack notices until a downstream system rejects the timestamp.

Why a sysadmin cares

Clock skew is the silent foundation of distributed systems. Five operational consequences ride on it:

1. TLS validation. A certificate’s notBefore and notAfter are absolute timestamps. A host whose wall clock is ninety seconds fast rejects certificates that have not yet become valid. A host whose wall clock is ninety seconds slow accepts certificates that have already expired.
2. Database replication. Most relational databases (PostgreSQL, MySQL) require the replica’s clock to be within a small bound of the primary’s. A replica that drifts beyond the bound pauses replication. The error message is in the replica’s log; the cause is the host’s clock.
3. Log correlation. Two services that share a request must produce log lines with consistent timestamps. A host whose clock is one second behind writes a log line whose timestamp is one second before the request actually arrived. The investigator pivots on the wrong time.
4. Cron and scheduled tasks. cron triggers on the host’s wall clock. Two hosts whose clocks disagree execute the same job twice or skip it once.
5. File timestamps. mtime, ctime, atime are wall-clock values. A backup that compares timestamps across hosts produces wrong deltas when the clocks disagree.

The fix is upstream of every one of these. The NTP daemon disciplines the clock. The cost is one package install and two configuration lines per host.

How it works

Every Linux host has two clocks: the hardware clock (the real-time clock, or RTC, on the motherboard) and the system clock (a kernel-maintained counter). The system clock is read by clock_gettime(CLOCK_REALTIME); the monotonic clock is read by clock_gettime(CLOCK_MONOTONIC). The two are independent measurements.

   RTC (hardware clock, battery-backed)
       |
       |  read on boot
       v
   Kernel timekeeping subsystem
       |
       |  maintained via TSC / HPET / acpi_pm
       v
   System clock (CLOCK_REALTIME)
       |
       |  read by applications
       v
   Application timestamp

The system clock is initialised from the RTC at boot. After boot, the user-space NTP daemon (chronyd, systemd-timesyncd, ntpd) periodically polls an upstream source, measures the offset, and writes the correction to the kernel via adjtimex(2) or clock_adjtime(2). The kernel applies the correction by adjusting the clock rate (the frequency field) rather than stepping the clock. A slewed correction is gradual; a stepped correction is instantaneous.

   Upstream NTP source
       |
       |  NTP poll (UDP 123)
       v
   chronyd
       |
       |  compute offset
       |  write to kernel via adjtimex(2)
       v
   Kernel timekeeping
       |
       |  adjust tick rate
       v
   System clock disciplined

The arithmetic of drift is worth memorising. A drift of one part per million produces:

• 86 milliseconds per day
• 602 milliseconds per week
• 2.6 seconds per month
• 31 seconds per year

The numbers are small at the day scale and enormous at the year scale. A fleet of undisciplined hosts drifts by a minute per host per year, in different directions. The drift is the cause; the offset is the symptom.

How to configure it

chrony is the canonical NTP daemon on Ubuntu 24.04, Debian 12, and RHEL 9. The configuration file is /etc/chrony/chrony.conf. A production baseline:

# /etc/chrony/chrony.conf
# Three internal NTP sources. The pool directive resolves to
# multiple addresses, providing resilience against a single
# host failure.
pool ntp1.internal.example.com iburst maxsources 4
pool ntp2.internal.example.com iburst maxsources 4

# Public fallback. Used only when the internal sources are
# unreachable.
server pool.ntp.org iburst

# Step the clock immediately if the offset exceeds 1 second.
# The 3 is the maximum number of steps before the directive
# expires; after that chrony slews only.
makestep 1.0 3

# Sync the hardware clock to the system clock every 11 minutes.
rtcsync

# Drift file for cold-start. chrony reads this to seed the
# frequency adjustment.
driftfile /var/lib/chrony/chrony.drift

# Allow the data centre monitoring subnet to query our offset.
allow 10.0.0.0/8

The directives worth memorising:

• pool — a hostname that resolves to multiple A records. The maxsources directive caps the number of sources used.
• makestep 1.0 3 — step the clock if the offset exceeds 1 second, but only for the first three updates. After that chrony slews only.
• rtcsync — write the system clock back to the hardware clock every 11 minutes. Without this directive, the RTC drifts and the next boot reads the wrong time.
• driftfile — a file that chrony reads on startup to seed the frequency adjustment. A cold-start without a drift file takes longer to converge.

systemd-timesyncd is a thin client for hosts that only need to consume NTP (not serve it). The configuration is /etc/systemd/timesyncd.conf:

# /etc/systemd/timesyncd.conf
[Time]
NTP=ntp1.internal.example.com ntp2.internal.example.com
FallbackNTP=pool.ntp.org
RootDistanceMaxSec=5
PollIntervalMinSec=32
PollIntervalMaxSec=2048

systemd-timesyncd is the default on Ubuntu. It is sufficient for a host that only needs to be a client. The chrony daemon is the production choice for a host that may serve NTP, or for a host that needs fine-grained control over the source selection algorithm.

How to validate it

The first command reads the user-space view of the offset:

# SEVERITY: READ-ONLY
chronyc tracking

Expected output (illustrative):

Reference ID    : C0A80101 (192.168.1.1)
Stratum         : 3
Ref time (UTC)  : Mon Jan 13 03:00:00 2026
System time     : 0.000012345 seconds fast of NTP time
Last offset     : -0.000006789 seconds
RMS offset      : 0.000012345 seconds
Frequency       : 12.345 ppm (slow)
Residual freq   : 0.001 ppm
Skew            : 0.012 ppm
Root delay      : 0.015 seconds
Root dispersion : 0.008 seconds
Update interval : 64.1 seconds
Leap status     : Normal

The relevant fields:

• System time — the offset between the system clock and the NTP source. Production baseline is below 0.001 seconds.
• Frequency — the kernel’s adjustment to the clock rate. A value of 12.345 ppm means the kernel is slowing the clock by 12.345 parts per million.
• Stratum — the distance from the reference clock. A value of 3 is typical; a value above 5 is a degraded source.
• Leap status — Normal is the default. Insert or Delete is the leap second announcement.

The second command reads the source state:

# SEVERITY: READ-ONLY
chronyc sources -v

Expected output (illustrative):

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = not combined
| /             'x' = may be in error, '~' = too variable
||                                                 .- xxxx  yyyy
|| Reach  Register (last measured)  offset  +/-   yyyy
|| ===========================================
^* ntp1.internal.example.com   3   6   377    -0.0001  0.0001  0.0001
^+ ntp2.internal.example.com   3   6   377    +0.0002  0.0002  0.0002

The ^* prefix indicates the current best source. The Reach column should be 377 (all probes succeeded in the last eight intervals). The offset should be small.

The third command reads the kernel-side view:

# SEVERITY: READ-ONLY
curl -s http://localhost:9100/metrics | grep '^node_timex_offset_seconds'

Expected output (illustrative):

node_timex_offset_seconds -0.000001234

A value above 0.1 seconds is a hard incident. A value above 0.01 seconds is a warning.

How it can fail

Six failure modes appear repeatedly in production.

1. No NTP daemon installed. The host runs systemd and a workload, but chrony or ntpd was never installed. The clock drifts from the moment of boot. Symptom: chronyc tracking returns “no NTP source”; node_timex_offset_seconds grows linearly.
2. NTP is firewalled. The chrony daemon is installed and running, but port 123 (NTP) or 323 (chrony) is blocked outbound. The daemon cannot reach the time server. Symptom: chronyc activity shows “unreachable”; the offset grows over time.
3. The VM was suspended and resumed. Suspend on a virtualised host pauses the wall clock but not the monotonic clock. The gap between resume and NTP re-sync is a clock skew window. Symptom: a small number of timestamps in a window of time are minutes off.
4. The container runtime did not sync the clock. The container’s clock is inherited from the host. If the host is wrong, every container on it is wrong. Symptom: the Kubernetes node has a drifted clock; all pods on the node emit wrong timestamps.
5. The hardware clock has drifted. The RTC is not disciplined. The system clock is correct after the daemon starts, but the RTC is wrong. The next boot shows the wrong time. The fix is the rtcsync directive.
6. The leap second has not been handled. The kernel does not support smoothtime, and the leap second causes a one-second jump. Symptom: applications that depend on monotonic time see a discontinuity. The fix is to update the kernel and to enable smoothtime.

How to troubleshoot it

The diagnostic order:

1. Is chrony running? systemctl status chrony. The simplest answer is the most common one.
2. What is the offset? chronyc tracking. A healthy offset is under 10 ms. A degraded offset is over 100 ms.
3. Is the offset stable? chronyc sourcestats -v. A growing offset is a host that is not being disciplined.
4. Are the sources reachable? chronyc activity. An “unreachable” source is a network problem.
5. Is the offset bounded across the fleet? A Prometheus query against node_timex_offset_seconds shows every host’s offset. The distribution should be tight.
6. Is the kernel view consistent with the chrony view? A divergence between node_timex_offset_seconds and chronyc tracking indicates a daemon-side issue.

Security implications

NTP traffic is UDP on port 123. It is a denial-of-service vector for a misconfigured host — an attacker can flood NTP requests and saturate the daemon — and an amplification vector if the daemon is open to the public internet. Bind the NTP daemon to the internal network only, or use authenticated NTS (Network Time Security) where the upstream supports it.

A second-order risk is around the accuracy of the wall clock for logging. A wrong clock produces wrong timestamps in logs and metrics, which complicates forensics. The remediation is the same: NTP discipline.

A third-order risk: an attacker who can write to the wall clock can forge the timestamps on every span and log line the host emits. The remediation is NTS authentication and kernel- level audit of clock changes.

Performance implications

The chrony daemon is a small, fast process. The CPU cost is sub-millisecond on a modern host. The memory cost is a few megabytes. The metric cost is one series per host on the Prometheus side.

The dominant operational cost is the network path to the upstream NTP source. The production baseline is to use a local NTP server (the organisation’s NTP pool) to avoid WAN jitter.

Production guidance

• Configure multiple NTP sources. Three is the production baseline; one is brittle.
• Use the organisation’s internal NTP pool. The pool.ntp.org public pool is for individual hosts; the production fleet should sync to the internal pool.
• Set makestep 1.0 3 to handle large offsets on boot.
• Set rtcsync to keep the hardware clock disciplined.
• Alert on node_timex_offset_seconds above 0.1 seconds.

Verification

You should now be able to answer:

• What is the difference between offset and drift, and which one does node_timex_offset_seconds measure?
• Why is a typical x86 system clock off by 86 ms per day when undisciplined?
• Which kernel interface does chronyd use to write a frequency adjustment?
• What is the production-baseline threshold for the offset alert in Prometheus?
• Why is monotonic time unsuitable for cross-host ordering of spans or log lines?

Quiz