Time Sync Observability

The application’s logs are timestamped one second ahead of the database’s logs. The cron jobs that should run at 02:00 run at 01:59:58 on the database and 02:00:02 on the application. The TLS certificate validation fails intermittently on the application because the certificate’s notBefore is one second in the future relative to the application’s clock. The team spends two days investigating before they look at the host’s clock.

Time sync observability is the discipline of measuring the time offset between a host and the upstream NTP source, the rate of change of that offset, and the quality of the NTP source. The metrics are a small, bounded set; the failure mode is silent until something else breaks. The most common operational incident is the host whose clock has drifted enough that log correlation is wrong, TLS validation fails, and database replication complains.

What it is

Time sync observability on Linux is the practice of exposing two classes of measurement:

• NTP offset - the difference between the system clock and the upstream NTP source. node_timex_offset_seconds is the metric; chronyc tracking is the canonical command.
• Clock discipline - the rate at which the kernel is correcting the clock. node_timex_frequency_adjustment_ratio is the metric; chronyc tracking shows the System time and Frequency fields.

The Linux kernel synchronises the clock via adjtimex(2) and clock_adjtime(2). The user-space daemon (chronyd, systemd-timesyncd, ntpd) reads the upstream NTP source and adjusts the kernel. node_exporter reads the kernel’s idea of the state and emits the node_timex_* metrics.

The alternative is to assume the clock is correct. The cost is silent skew that breaks TLS, log correlation, and replication.

Why a sysadmin cares

Time skew is the silent failure mode of distributed systems. The clocks on two hosts diverge by milliseconds per day; the divergence eventually crosses a threshold (TLS validation, replication, log correlation); the team debugs the symptom without looking at the clock. The cost is hours of investigation that could have been avoided with a single metric.

A second reason: the NTP source itself can fail. The upstream NTP server is unreachable; the host falls back to the local clock; the clock drifts. The chain of failure is invisible to the application. The right observability exposes the upstream source, the offset, and the stratum.

A third reason: the chrony configuration is the difference between a healthy clock and a brittle one. A host with one NTP source is brittle; a host with three is production. The discipline is to configure chrony with multiple sources and to alert on the offset.

How it works

chrony is the canonical NTP daemon on modern Linux distributions (Ubuntu 24.04, Debian 12, RHEL 9). systemd-timesyncd is the default on Ubuntu (a thin client that does not serve NTP to peers). The two metrics sources:

                  Time Sync Signals
                  ==================

   chronyc tracking                 node_exporter metric
   ----------------                 -------------------
   Reference ID                     (no direct metric; see sources)
   Stratum                          (no direct metric; see sources)
   System time (offset, seconds)    node_timex_offset_seconds
   Last offset (seconds)            (aggregated into offset)
   RMS offset (seconds)             (no direct metric)
   Frequency (ppm)                  node_timex_frequency_adjustment_ratio
   Residual freq (ppm)              (no direct metric)
   Skew (ppm)                       (no direct metric)
   Root delay (seconds)             (no direct metric)
   Root dispersion (seconds)        (no direct metric)
   Update interval (seconds)        (no direct metric)
   Leap status                      (no direct metric; see kernel)

   chronyc sources                  node_exporter metric
   ----------------                 -------------------
   per-source state                 (no direct metric; see configuration)
   per-source offset                (no direct metric)
   per-source stratum               (no direct metric)

node_exporter exposes the kernel-side state via the timex collector. The chronyc commands expose the user-space view. The two together are the production picture.

The canonical chronyc tracking output:

Reference ID    : C0A80101 (192.168.1.1)
Stratum         : 3
Ref time (UTC)  : Mon Jan 13 03:00:00 2026
System time     : 0.000012345 seconds fast of NTP time
Last offset     : -0.000006789 seconds
RMS offset      : 0.000012345 seconds
Frequency       : 12.345 ppm (slow)
Residual freq   : 0.001 ppm
Skew            : 0.012 ppm
Root delay      : 0.015 seconds
Root dispersion : 0.008 seconds
Update interval : 64.1 seconds
Leap status     : Normal

The relevant fields:

• System time - the offset between the system clock and the NTP source. Production baseline is below 0.001 seconds.
• Frequency - the kernel’s adjustment to the clock rate. A value of 12.345 ppm means the kernel is slowing the clock by 12.345 parts per million.
• Stratum - the distance from the reference clock. A value of 3 is typical; a value above 5 is a degraded source.
• Leap status - the leap second status. Normal is the default; Insert or Delete is the leap second announcement.

The node_timex_offset_seconds metric is the canonical Prometheus signal. The value is the system clock’s offset, in seconds.

Under the hood

How to configure it

chrony is the canonical configuration on Ubuntu 24.04, Debian 12, and RHEL 9. The configuration file is /etc/chrony/chrony.conf:

# /etc/chrony/chrony.conf
# Use NTP sources from the project NTP pool.
pool pool.ntp.org iburst maxsources 4

# Use a local NTP source if available.
server ntp1.internal.example.com iburst
server ntp2.internal.example.com iburst
server ntp3.internal.example.com iburst

# Allow the local network to query this host.
allow 10.0.0.0/8

# Tweak the maximum offset.
makestep 1.0 3
rtcsync

# Logging.
log tracking measurements statistics
logdir /var/log/chrony

The directives worth memorising:

• pool pool.ntp.org iburst maxsources 4 - the public pool, with up to 4 sources, and the iburst flag for fast initial sync. Replace with the organisation’s internal NTP pool.
• server ntp1.internal.example.com iburst - a single server, with the iburst flag. A host should have multiple server or pool entries.
• makestep 1.0 3 - if the offset is above 1 second, step the clock immediately. The 3 is the maximum number of steps before the directive is disabled. The production baseline.
• rtcsync - sync the hardware clock to the system clock every 11 minutes. The directive prevents the hardware clock from drifting when the host is rebooted.

For systemd-timesyncd, the configuration is /etc/systemd/timesyncd.conf:

# /etc/systemd/timesyncd.conf
[Time]
NTP=ntp1.internal.example.com ntp2.internal.example.com ntp3.internal.example.com
FallbackNTP=pool.ntp.org
RootDistanceMaxSec=5
PollIntervalMinSec=32
PollIntervalMaxSec=2048

The directives are the systemd equivalent. The NTP list is the primary sources; FallbackNTP is the fallback when the primary is unreachable.

For the node_exporter configuration, the timex collector is enabled by default:

# /etc/systemd/system/node_exporter.service.d/override.conf
[Service]
ExecStart=
ExecStart=/opt/node_exporter/node_exporter \
  --web.listen-address=0.0.0.0:9100 \
  --collector.timex

How to validate it

The first check is the chrony status:

# SEVERITY: READ-ONLY
chronyc tracking

Expected output (illustrative):

Reference ID    : C0A80101 (192.168.1.1)
Stratum         : 3
System time     : 0.000012345 seconds fast of NTP time
Last offset     : -0.000006789 seconds
RMS offset      : 0.000012345 seconds
Frequency       : 12.345 ppm (slow)
Leap status     : Normal

The System time value should be below 0.001 seconds. The Stratum should be below 5. The Leap status should be Normal.

The second check is the chrony sources:

# SEVERITY: READ-ONLY
chronyc sources -v

Expected output (illustrative):

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = combined
| / .- '*' = current best, '+' = combined, '-' = not combined
| /             'x' = may be in error, '~' = too variable
||                                                 .- xxxx  yyyy
|| Reach  Register (last measured)  offset  +/-   yyyy
|| ===========================================
^* ntp1.internal.example.com   3   6   377    -0.0001  0.0001  0.0001
^+ ntp2.internal.example.com   3   6   377    +0.0002  0.0002  0.0002
^+ ntp3.internal.example.com   3   6   377    -0.0000  0.0001  0.0001

The ^* prefix indicates the current best source. The Reach column should be 377 (all probes succeeded in the last 8 intervals). The offset should be small.

The third check is the Prometheus metric:

# SEVERITY: READ-ONLY
curl -s http://localhost:9100/metrics | grep '^node_timex_offset_seconds'

Expected output (illustrative):

node_timex_offset_seconds -0.000001234

A value above 0.1 seconds is a hard incident. A value above 0.01 seconds is a warning.

The fourth check is the alert rule:

# /etc/prometheus/rules/timesync.rules.yml
groups:
- name: timesync.offset
  interval: 30s
  rules:
  - alert: HostTimeSkewHigh
    expr: |
      abs(node_timex_offset_seconds) > 0.1
    for: 5m
    labels:
      severity: ticket
      team: platform
    annotations:
      summary: 'Time skew on {{ $labels.instance }} above 0.1s for 5m'
      description: 'Check chronyc tracking and chronyc sources on the host.'
      runbook_url: 'https://runbooks.example.com/host/time-skew'

The threshold of 0.1 seconds is the production baseline. A value above 0.01 seconds is the warning.

How it can fail

Five failure modes appear repeatedly in production.

1. The upstream NTP source is unreachable. The host falls back to a single source or the local clock. The clock drifts. Symptom: chronyc tracking shows the offset increasing; reachability drops. The fix is to add multiple sources and to alert on the upstream source reachability.
2. The host’s clock has drifted beyond the makestep threshold. The makestep 1.0 3 directive has expired (after 3 steps). The kernel is slewing the clock. The offset can take hours to converge. The fix is to manually step the clock (chronyc makestep) or to restart the daemon.
3. The chrony daemon is not running. The systemd unit is inactive or failed. Symptom: the clock is free-running; the offset grows. The fix is to start the daemon and to coinvestigate the failure.
4. The hardware clock has drifted. The RTC is not disciplined. The system clock is correct after the daemon starts, but the RTC is wrong. The next boot shows the wrong time. The fix is the rtcsync directive.
5. The leap second has not been handled. The system’s kernel does not support smoothtime, and the leap second causes a 1-second jump. Symptom: applications that depend on monotonic time see a discontinuity. The fix is to update the kernel and to enable smoothtime.

How to troubleshoot it

The diagnostic order when a host has time skew:

1. Inspect chronyc tracking. What is the System time?
2. Inspect chronyc sources -v. What is the reach state?
3. Inspect node_timex_offset_seconds. Is the value above the threshold?
4. Inspect systemctl status chrony. Is the daemon running?
5. Inspect journalctl -u chrony. What does the log say?
6. Inspect the upstream NTP source. Is the upstream reachable from the host?
7. Inspect the firewall. Is UDP port 123 open?
8. Inspect /var/log/chrony/. What does the tracking log say?

Each step confirms or rules out a layer. Steps 1-3 answer the “what is the offset?” question; step 4 is the daemon status; step 5 is the daemon log; step 6 is the upstream reachability; step 7 is the network path; step 8 is the historical record.

Security implications

The NTP source is a security boundary. A malicious NTP server can shift the host’s clock by a small amount. The host’s TLS validation may accept a certificate that would otherwise be rejected. The mitigation is to use multiple sources and to detect anomalous offsets.

chrony supports NTS (Network Time Security), which provides authenticated NTP. The configuration is ntsserver directives in the configuration file. The production baseline is to use NTS where the upstream supports it.

The chronyc command can be used to step the clock or to change the source. The production baseline is to restrict the chronyc access to the root user (/etc/chrony/chrony.keys).

The node_exporter node_timex metric does not contain the upstream source’s IP or the cryptographic state. The PII surface is low.

Performance implications

The chrony daemon is a small, fast process. The CPU cost is sub-millisecond on a modern host. The memory cost is a few megabytes. The metric cost is one series per host.

The node_timex collector reads /proc/timer_list and the adjtimex return value. The cost is a few microseconds per scrape. The scrape interval is not a performance concern.

The dominant operational cost is the network path to the upstream NTP source. The production baseline is to use a local NTP server (the organisation’s NTP pool) to avoid WAN jitter.

Production guidance

• Configure multiple NTP sources. Three is the production baseline; one is brittle.
• Use the organisation’s internal NTP pool. The pool.ntp.org public pool is for individual hosts; the production fleet should sync to the internal pool.
• Set makestep 1.0 3 to handle large offsets on boot.
• Set rtcsync to keep the hardware clock disciplined.
• Alert on node_timex_offset_seconds above 0.1 seconds.
• Coinvestigate the upstream NTP source reachability. The offset is the symptom; the source is the cause.
• Use NTS where the upstream supports it. The encryption is the production baseline.

Verification

You should now be able to answer:

• What is the difference between node_timex_offset_seconds and the chronyc tracking System time field, and what question does each answer?
• Why is the silent-clock-skew failure shape the most common time-sync incident in production?
• What is the right configuration for chrony with multiple sources and the makestep directive?
• What is the diagnostic order when a host has time skew?
• Why is NTS the production baseline for NTP authentication?

Quiz