Prevention

A team maintains a fleet of three hundred hosts. Each host emits traces and logs. The team’s runbook says “install chrony on every host that emits traces”. The team’s base image includes chrony. The team’s IaC playbook configures three NTP sources. The team’s Prometheus alert fires at 100 ms. The team’s audit catches the host that escaped the base image.

A team that does not have this discipline has a different shape. One host was provisioned from a manual kickstart that skipped the chrony package. One host was rebuilt from an old image after a disk failure. One host has chrony installed but the daemon is masked by a misapplied systemd override. The drift on each of these hosts is invisible until a trace that crosses the host boundary shows a child before its parent.

Prevention is the discipline that prevents the second team from existing.

What it is

Prevention of clock skew is the operational discipline that ensures every host in the fleet has a disciplined wall clock from the moment of boot. Three pillars:

1. The base image includes the NTP daemon. chrony is installed and enabled in the image; a fresh host boots with NTP discipline from the first second.
2. The IaC playbook configures three NTP sources. Each host has at least three upstream sources, the makestep and rtcsync directives, and a reachable firewall path.
3. The monitoring layer enforces the discipline. Prometheus alerts at 100 ms warning and 500 ms critical; the absence of the node_timex_* metrics fires an alert.

The common shape is the silent escape. A host is provisioned outside the IaC pipeline; the chrony package is missing; the clock drifts. The team’s audit catches the host before the downstream pipeline breaks.

Why a sysadmin cares

Clock skew is the silent failure mode of distributed systems. The cost of prevention is roughly five minutes per host at provisioning time. The cost of recovery is paid in incident response: the on-call engineer pages, the trace visualisation breaks, the log correlation is wrong, the database replication pauses.

A second reason: the prevention discipline is the only mechanism that catches the host that escaped the base image. The IaC pipeline provisions most hosts correctly; the manual kickstart, the disk rebuild, the misapplied override are the exceptions. The fleet-wide audit is the mechanism that catches the exceptions.

A third reason: the right NTP discipline is the foundation for TLS validation, log correlation, trace ordering, database replication, and cron jobs. The discipline is upstream of every other observability signal. A host without NTP discipline emits wrong timestamps on every signal.

How it works

The prevention discipline has three layers:

   Base image                    IaC pipeline                   Monitoring layer
   -----------                   ------------                   ----------------
   chrony installed              3 NTP sources per host         alert on offset > 100 ms
   chrony enabled                makestep 1.0 3                 alert on offset > 500 ms
   drift file present            rtcsync                        alert on absent metric
   node_exporter installed       allow subnet                   dashboard on distribution

The base image is the first layer. A host provisioned from the image has chrony installed and enabled. The drift file is present (or absent, depending on the image). The node_exporter is installed and the timex collector is enabled.

The IaC pipeline is the second layer. A host provisioned through the pipeline has three NTP sources, the makestep directive, the rtcsync directive, and a reachable firewall path. The configuration is idempotent: re-running the playbook does not change the configuration if it is already correct.

The monitoring layer is the third layer. Prometheus scrapes the node_timex_* metrics from every host. The alert rules fire at 100 ms warning and 500 ms critical. The dashboard shows the offset distribution per host.

   Boot                          Steady state                   Audit
   ----                          ------------                   -----
   chrony starts                 chronyd polls upstream          weekly: every host
   iburst burst                  measures offset                 offset < 10 ms
   fast initial sync             adjusts frequency               node_timex present
   makestep if offset > 1s       writes to kernel via adjtimex   chrony active
   rtcsync every 11 minutes      kernel applies adjustment       sources reachable

How to configure it

The prevention configuration has four parts: the chrony configuration that every host receives, the systemd-timesyncd configuration for hosts that do not need to serve NTP, the Prometheus alert rules, and the IaC playbook that distributes the configuration.

The chrony configuration (the production baseline):

# /etc/chrony/chrony.conf
# Three internal NTP sources. The pool directive resolves to
# multiple A records, providing resilience.
pool ntp1.internal.example.com iburst maxsources 4
pool ntp2.internal.example.com iburst maxsources 4

# Public fallback. Used only when the internal sources are
# unreachable.
server pool.ntp.org iburst

# Step the clock immediately if the offset exceeds 1 second,
# for the first 3 updates. After that, slew only.
makestep 1.0 3

# Sync the hardware clock to the system clock every 11 minutes.
rtcsync

# Drift file for cold-start.
driftfile /var/lib/chrony/chrony.drift

# Allow the data centre monitoring subnet to query our offset.
allow 10.0.0.0/8

# Use NTS (Network Time Security) where the upstream supports it.
ntsserverdir /var/lib/chrony

For systemd-timesyncd on hosts that only consume NTP:

# /etc/systemd/timesyncd.conf
[Time]
NTP=ntp1.internal.example.com ntp2.internal.example.com ntp3.internal.example.com
FallbackNTP=pool.ntp.org
RootDistanceMaxSec=5
PollIntervalMinSec=32
PollIntervalMaxSec=2048

The Prometheus alert rules:

# /etc/prometheus/rules/ntp.yaml
groups:
  - name: ntp
    interval: 30s
    rules:
      - alert: HostClockSkewWarning
        expr: |
          abs(avg_over_time(node_timex_offset_seconds[5m])) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Host {{ $labels.instance }} clock skew exceeds 100 ms'
          description: |
            The wall clock on this host has drifted by more than
            100 ms from NTP. Investigate the chrony daemon.

      - alert: HostClockSkewCritical
        expr: |
          abs(avg_over_time(node_timex_offset_seconds[5m])) > 0.5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'Host {{ $labels.instance }} clock skew exceeds 500 ms'

      - alert: HostNoNtpSource
        expr: |
          absent(node_timex_offset_seconds{instance=~".+"}) == 1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'Host {{ $labels.instance }} has no NTP offset metric'
          description: |
            The node_exporter timex collector is missing or the
            chrony daemon is not running. Investigate the host.

The Ansible playbook (illustrative):

# /etc/ansible/ntp.yml
- name: Configure NTP discipline on every host
  hosts: all
  become: true
  tasks:
    - name: Install chrony
      ansible.builtin.package:
        name: chrony
        state: present

    - name: Deploy chrony configuration
      ansible.builtin.template:
        src: chrony.conf.j2
        dest: /etc/chrony/chrony.conf
        owner: root
        group: root
        mode: '0644'
      notify: Restart chrony

    - name: Enable and start chrony
      ansible.builtin.systemd:
        name: chrony
        enabled: true
        state: started

    - name: Allow UDP port 123 outbound
      ansible.builtin.ufw:
        rule: allow
        direction: out
        proto: udp
        to_port: '123'

  handlers:
    - name: Restart chrony
      ansible.builtin.systemd:
        name: chrony
        state: restarted

How to validate it

The validation reads four surfaces: the per-host offset, the fleet-wide distribution, the daemon status, and the configuration drift.

READ-ONLY: confirm the per-host offset.

chronyc tracking
# Reference ID    : C0A80101 (ntp1.internal.example.com)
# Stratum         : 3
# System time     : 0.000012345 seconds fast of NTP time
# Leap status     : Normal

READ-ONLY: confirm the fleet-wide distribution.

curl -s http://prometheus.monitoring.svc:9090/api/v1/query \
  --data-urlencode 'query=topk(10, abs(avg_over_time(node_timex_offset_seconds[5m])))'

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"checkout-01.internal"},"value":[1736822645.123,"0.000234"]},{"metric":{"instance":"checkout-02.internal"},"value":[1736822645.123,"0.000189"]}]}}

The top 10 should all be below 10 ms.

READ-ONLY: confirm the daemon is running on every host.

ansible all -m shell -a 'systemctl is-active chrony'

A non-zero exit code on any host is a host without NTP discipline.

READ-ONLY: confirm the configuration is consistent.

ansible all -m shell -a 'md5sum /etc/chrony/chrony.conf' \
  | awk '{print $1}' \
  | sort -u \
  | wc -l

A value greater than 1 is configuration drift across the fleet. The fix is to identify the divergent hosts and apply the playbook.

How it can fail

Five failure modes appear repeatedly in prevention.

1. The base image is stale. The chrony package is removed from the image by an unrelated cleanup. New hosts boot without NTP discipline. Symptom: a fleet-wide alert as new hosts come online. The fix is to update the image and rebuild.
2. The IaC playbook has an exception. A specific host group is excluded from the playbook for a legacy reason. The host group drifts. Symptom: a specific subset of hosts has the wrong offset. The fix is to remove the exception and apply the playbook.
3. The NTP source is unreachable from a subnet. A new subnet is provisioned without an NTP source. Symptom: hosts in the subnet have no NTP source. The fix is to deploy an NTP source in the subnet or to allow the subnet to reach the existing source.
4. The drift file is corrupted. The drift file on a host is corrupted by a disk failure. The host takes longer to converge. Symptom: the offset is high for a few minutes after boot. The fix is to remove the drift file and let chrony rebuild it.
5. The chrony version is incompatible with NTS. A new chrony version changes the NTS configuration format. Hosts on the old version do not authenticate. Symptom: the chronyc authdata command fails. The fix is to update the chrony version across the fleet.

How to troubleshoot it

The diagnostic order when the prevention discipline is not working:

1. Confirm the base image. Build a host from the image and inspect the chrony package and configuration. A stale image is the most common cause.
2. Confirm the IaC playbook. Run the playbook against a test host and inspect the result. An exception in the playbook is the second most common cause.
3. Confirm the fleet-wide audit. Run the weekly audit and inspect the top 10 offsets. A long tail is a sign that some hosts have escaped the discipline.
4. Confirm the upstream sources. Run chronyc activity on every host that has drifted. An unreachable source is a network problem.
5. Confirm the firewall. UDP port 123 must be open from every host to the upstream. A blocked port is the most common cause of an unreachable source.

The fix is rarely in the alert rule. The fix is at the provisioning layer or the network layer.

Security implications

The prevention discipline is read-mostly. The chrony daemon listens on UDP port 123; the configuration allows queries from the monitoring subnet. The right discipline is to restrict the allow directive to the monitoring subnet only and to bind the daemon to the internal network interface.

A second-order risk: NTP traffic on the public internet is an amplification vector for attacks on others. The fix is to use NTS where the upstream supports it; NTS authenticates the NTP exchange and prevents the host from being used as an amplification relay.

A third-order risk: an attacker who can write to the wall clock can forge the timestamps on every span and log line the host emits. The remediation is NTS authentication and kernel-level audit of clock changes.

Performance implications

The prevention discipline is cheap. The chrony daemon is a small, fast process; the node_exporter timex collector reads /proc/timer_list and the adjtimex return value; the cost is a few microseconds per scrape.

The dominant cost is the operational cost of running the periodic audit. A weekly audit of three hundred hosts takes roughly thirty minutes of operator time. The cost is small compared to the cost of an incident caused by a drifted host.

Production guidance

• Bake chrony into the base image. A host provisioned from the image boots with NTP discipline.
• Configure three internal NTP sources per host. Three is the production baseline; one is brittle.
• Set makestep 1.0 3 and rtcsync in the chrony configuration. The two together handle the boot-time large-offset and the RTC-drift cases.
• Alert on node_timex_offset_seconds above 100 ms warning, 500 ms critical. The thresholds catch the wrong-time symptom before it corrupts a downstream pipeline.
• Run a weekly fleet-wide audit. The audit catches the host that escaped the base image.
• Use NTS where the upstream supports it. The cryptographic authentication prevents an attacker from forging NTP responses.

Verification

You should now be able to answer:

• What are the three pillars of the prevention discipline?
• Why is the periodic audit as important as the initial configuration?
• What is the right threshold for the Prometheus alert at warning and at critical?
• When is systemd-timesyncd sufficient and when is the full chrony daemon required?
• Why is NTS the production baseline for NTP authentication?

Quiz