Detecting Skew

A team has just been paged. The alert is HostClockSkewCritical. The metric is node_timex_offset_seconds. The value is 1.412. The host is in production traffic. The on-call engineer opens the runbook. The runbook says “check chronyc tracking on the host”. The engineer SSHes to the host, runs chronyc tracking, and reads the output. The output contradicts the metric: System time : 0.000012345 seconds fast of NTP time. The metric says 1.4 seconds; chrony says 12 microseconds. Which is right?

The answer is that both are right, and the contradiction is the signal. The Prometheus metric was scraped twelve minutes ago; the chrony command is right now. The offset was 1.4 seconds twelve minutes ago, and the daemon slew the clock back into discipline. The detection layer must catch both the transient and the steady state. This lesson is the discipline of the detection layer — what to ask, in what order, with what threshold, and what to do when the two views disagree.

What it is

Detection of clock skew is the discipline of asking three questions on every host that emits telemetry, at intervals short enough to catch the wrong-time symptom before it corrupts a downstream pipeline:

1. What is the offset right now? The difference between the local wall clock and the upstream NTP source. The answer is chronyc tracking on the host and node_timex_offset_seconds in Prometheus.
2. Is the offset stable or drifting? The rate of change of the offset over the last N samples. The answer is chronyc sourcestats -v on the host and a Prometheus query that takes the derivative of the offset metric.
3. Are the sources reachable? Whether the upstream NTP source is responding. The answer is chronyc activity on the host and an alert on the daemon’s reach counter.

The three questions together answer the operational question “is this host’s clock trustworthy?”.

The detection layer has two parts:

• Host-local detection. The chrony user-space daemon exposes the offset via chronyc tracking, the source state via chronyc sources -v, and the source reachability via chronyc activity. The kernel exposes the offset via the node_timex_offset_seconds metric on the node_exporter.
• Fleet-wide detection. Prometheus scrapes the node_timex_* metrics from every host and evaluates alert rules against the fleet. The Grafana dashboard renders the offset per host and the distribution across the fleet.

The two parts are complementary. The host-local view is immediate; the fleet-wide view is historical and aggregated.

Why a sysadmin cares

Clock skew is the silent failure mode of distributed systems. The symptom does not appear at the host; it appears at the downstream pipeline (TLS validation, log correlation, trace ordering, database replication). The detection layer is the only mechanism that catches the wrong-time symptom before it corrupts a downstream pipeline.

The cost of not detecting skew early is paid in incident response. A team that does not alert on the offset metric discovers the skew when the on-call engineer opens a trace with a child before its parent, or when the database replication pauses, or when the application’s TLS validation fails. The detection layer is the canary.

A second reason: the right threshold for the alert is non-obvious. Too tight, and the alert fires on every transient (slew-induced microsecond jitter) and the team ignores it. Too loose, and the alert fires after the trace timeline is already fiction. The right threshold is the threshold that catches the wrong-time symptom before it corrupts a downstream pipeline.

How it works

The detection chain has four steps:

   chronyd                            kernel
   --------                           ------
   poll upstream NTP sources          read adjtimex(2) return
       |                                  ^
       | compute offset                    | write frequency adjustment
       v                                  |
   update kernel via clock_adjtime(2) ---+
       |
       | read offset
       v
   node_exporter timex collector
       |
       | emit node_timex_offset_seconds
       v
   Prometheus scrape
       |
       | evaluate alert rule
       v
   AlertManager
       |
       | page on-call
       v
   On-call engineer

The chain is straightforward; the operational discipline is knowing which link is the right one to alert on.

   chronyc tracking                 node_exporter metric
   ----------------                 -------------------
   System time (offset, seconds)    node_timex_offset_seconds
   Last offset (seconds)            (aggregated into offset)
   RMS offset (seconds)             (no direct metric)
   Frequency (ppm)                  node_timex_frequency_adjustment_ratio
   Residual freq (ppm)              (no direct metric)
   Skew (ppm)                       (no direct metric)
   Root delay (seconds)             (no direct metric)
   Root dispersion (seconds)        (no direct metric)
   Update interval (seconds)        (no direct metric)
   Leap status                      (no direct metric; see kernel)

   chronyc sources                  node_exporter metric
   ----------------                 -------------------
   per-source state                 (no direct metric; see configuration)
   per-source offset                (no direct metric)
   per-source stratum               (no direct metric)

The Prometheus metric node_timex_offset_seconds is the kernel-side view of the offset; the chronyc tracking command is the user-space view. A divergence between the two indicates that the daemon has stale data, that the metric was scraped recently and the daemon has not yet re-polled, or that the kernel has been adjusted by an admin outside of chrony.

How to configure it

The detection layer has three components: the node_exporter collector, the Prometheus alert rules, and the Grafana panel.

The node_exporter timex collector is enabled by default:

# /etc/systemd/system/node_exporter.service.d/override.conf
[Service]
ExecStart=
ExecStart=/opt/node_exporter/node_exporter \
  --web.listen-address=0.0.0.0:9100 \
  --collector.timex

The Prometheus alert rules:

# /etc/prometheus/rules/ntp.yaml
groups:
  - name: ntp
    interval: 30s
    rules:
      - alert: HostClockSkewWarning
        expr: |
          abs(avg_over_time(node_timex_offset_seconds[5m])) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Host {{ $labels.instance }} clock skew exceeds 100 ms'
          description: |
            The wall clock on this host has drifted by more than
            100 ms from NTP. Distributed-trace timestamps from
            this host are unreliable. Investigate the chrony
            daemon.

      - alert: HostClockSkewCritical
        expr: |
          abs(avg_over_time(node_timex_offset_seconds[5m])) > 0.5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'Host {{ $labels.instance }} clock skew exceeds 500 ms'
          description: |
            The wall clock on this host has drifted by more than
            500 ms from NTP. TLS validation, log correlation,
            and trace ordering are all at risk. Investigate
            immediately.

      - alert: HostNoNtpSource
        expr: |
          node_timex_status == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'Host {{ $labels.instance }} has no synchronised NTP source'
          description: |
            The kernel reports that the clock is not synchronised
            with any NTP source. The host is free-running. Run
            chronyc tracking on the host.

A Grafana panel for the fleet-wide view:

# Average offset per host, last 5 minutes
avg by (instance) (avg_over_time(node_timex_offset_seconds[5m]))

# Fleet distribution (50th, 95th, 99th percentile)
quantile by (instance) (0.5, avg_over_time(node_timex_offset_seconds[5m]))
quantile by (instance) (0.95, avg_over_time(node_timex_offset_seconds[5m]))
quantile by (instance) (0.99, avg_over_time(node_timex_offset_seconds[5m]))

How to validate it

The validation reads three surfaces: the chrony user-space view, the kernel-side view, and the Prometheus alert.

READ-ONLY: confirm the daemon-side view.

chronyc tracking
# Reference ID    : C0A80101 (ntp1.internal.example.com)
# Stratum         : 3
# System time     : 0.000012345 seconds fast of NTP time
# Frequency       : 12.345 ppm (slow)
# Leap status     : Normal

The System time value should be below 0.001 seconds. The Stratum should be below 5. The Leap status should be Normal.

READ-ONLY: confirm the source state.

chronyc sources -v

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = not combined
| /             'x' = may be in error, '~' = too variable
||                                                 .- xxxx  yyyy
|| Reach  Register (last measured)  offset  +/-   yyyy
|| ===========================================
^* ntp1.internal.example.com   3   6   377    -0.0001  0.0001  0.0001
^+ ntp2.internal.example.com   3   6   377    +0.0002  0.0002  0.0002

The ^* prefix indicates the current best source. The Reach column should be 377 (all probes succeeded in the last eight intervals).

READ-ONLY: confirm the kernel-side view.

curl -s http://localhost:9100/metrics | grep '^node_timex_offset_seconds'
# node_timex_offset_seconds -0.000001234

A value above 0.1 seconds is a hard incident. A value above 0.01 seconds is a warning.

READ-ONLY: confirm the alert rule fires on a known bad host.

promtool check rules /etc/prometheus/rules/ntp.yaml

The promtool command validates the rule syntax and the PromQL. A non-zero exit code is a syntax error; fix the rule before reloading Prometheus.

How it can fail

Five failure modes appear repeatedly in detection.

1. No node_exporter collector enabled. The timex collector is enabled by default in recent versions of node_exporter. An older binary or a custom build may have the collector disabled. Symptom: the node_timex_* metrics are absent. The fix is to enable the collector and reload the daemon.
2. Alert threshold too tight. A 1 ms threshold fires on every slew-induced jitter. The team ignores the alert; the alert becomes wallpaper. The fix is to set the threshold at 100 ms for warning and 500 ms for critical.
3. Alert threshold too loose. A 10-second threshold fires only after the trace timeline is fiction. The fix is to set the threshold at 100 ms for warning and 500 ms for critical.
4. No fleet-wide panel. The alert fires per host, but the team cannot see the distribution. A long tail of misconfigured hosts is invisible. The fix is to add a panel that shows the offset distribution per host.
5. Daemon-side view contradicts kernel-side view. The chronyc tracking says “12 microseconds”; the metric says “1.4 seconds”. The contradiction is expected after a slew; a persistent contradiction is a sign of stale daemon data. The fix is to restart the daemon or to investigate the discrepancy manually.

How to troubleshoot it

The diagnostic order when an alert fires:

1. Confirm the metric. Query node_timex_offset_seconds directly in Prometheus. The value should match the alert.
2. Confirm the daemon. systemctl status chrony. The daemon should be active.
3. Confirm the offset on the host. chronyc tracking on the host. The value should match the metric (within a few seconds).
4. Confirm the source state. chronyc sources -v. The ^* prefix indicates the current best source; the Reach column should be 377.
5. Confirm the upstream reachability. chronyc activity. The output should show the active sources.
6. Coinvestigate the source reachability. A non-zero unreachable count is a network problem.

The fix is rarely in the alert rule. The fix is at the host.

Security implications

The detection layer is read-only. The chronyc commands require root or the chrony access list; the node_exporter metrics are exposed on a configurable port. The right discipline is to restrict the node_exporter endpoint to the Prometheus source IP and to require authentication for the chrony access list.

A wrong threshold (too loose) hides a real skew. The threshold is the security boundary. The team that ignores the alert becomes the team that pages on the downstream symptom.

Performance implications

The chrony daemon is a small, fast process. The node_exporter timex collector reads /proc/timer_list and the adjtimex return value; the cost is a few microseconds per scrape. The Prometheus scrape interval is not a performance concern.

The dominant cost is the alert evaluation. A fleet of one thousand hosts with a 30-second scrape interval produces one thousand time series per metric. The Prometheus cost is the storage and the evaluation. The cost is small.

Production guidance

• Alert at 100 ms for warning, 500 ms for critical. The thresholds catch the wrong-time symptom before it corrupts a downstream pipeline.
• Alert on the absence of the metric. A missing metric is as bad as a wrong value.
• Display the fleet-wide distribution on a dashboard. The long tail is the place to look for misconfigured hosts.
• Coinvestigate the source reachability. The offset is the symptom; the source is the cause.
• Test the alert in staging. A alert that does not fire on a known bad host is a broken alert.

Verification

You should now be able to answer:

• What is the difference between the chronyc tracking view and the node_timex_offset_seconds view?
• What is the right threshold for the Prometheus alert at warning and at critical?
• Why is alerting on the absence of the metric as important as alerting on the value?
• How do you read chronyc sources -v to confirm a healthy host?
• What is the difference between the offset alert and the source-reachability alert?

Quiz