Mitigation

The alert fires at 03:14. HostClockSkewCritical on the checkout service host. The metric shows 1.4 seconds. The on-call engineer opens the runbook. The runbook says “force a chrony step”. The engineer types chronyc makestep. The clock jumps forward by 1.4 seconds. In-flight spans on the checkout host have negative durations. The trace visualisation breaks for the next thirty seconds. The engineer pages the team lead. The team lead says “wait, the host was in production traffic — the right answer was to slew, not to step”.

Mitigation is the moment when the wrong action causes more damage than the original skew. This lesson is the right discipline: when to slew, when to step, when to wait.

What it is

Mitigation is the act of restoring sync between the local wall clock and the upstream NTP source. Three paths exist:

1. Slew (gradual adjustment). The kernel applies a frequency adjustment that slowly brings the local clock back into sync with the upstream. The clock never jumps; the monotonic time remains consistent with the wall clock. The cost is time: a 1.4-second offset takes hours to slew back at the kernel’s maximum slew rate.
2. Step (instantaneous adjustment). The kernel sets the local clock to the upstream time in one operation. The clock jumps forward or backward; the monotonic time is unaffected but the wall clock discontinuously changes. In-flight telemetry that was stamped with the old wall clock value is now in the wrong position in the timeline.
3. Wait (do nothing). The chrony daemon continues to measure the offset and the system continues to operate with the wrong time. The right action when the host is in production traffic and the offset is below the critical threshold.

The common shape is the silent drift on a host whose chrony daemon has not been running. The clock is 1.4 seconds off true time. The host is in production traffic. The right mitigation is to start the daemon and let it slew; the wrong mitigation is to step the clock and break in-flight spans.

Why a sysadmin cares

The wrong mitigation is worse than the original problem. Stepping the clock during production traffic creates a discontinuity in the wall clock; in-flight telemetry that was stamped with the old wall clock value now appears to be from the future or the past. The trace visualisation breaks. The log entries land in the wrong time range. The investigator sees a discontinuity that is not in the system.

The right mitigation depends on the operational context:

• Host in production traffic. Slew, do not step. The slew takes time but the wall clock is monotonic from the application’s perspective.
• Host in a maintenance window. Step, do not slew. The slew takes hours; the maintenance window is short.
• Host that has not been disciplined for days. Investigate the cause (firewall, dead NTP source, daemon down) before restoring sync. Restoring sync without fixing the cause is a temporary fix.
• Host that is the upstream NTP source. The upstream source is the reference. The fix is to investigate the upstream’s upstream. Stepping the upstream is the last resort.

The discipline is to ask “what is the cause?” before restoring sync. The mitigation is downstream of the cause.

How it works

The Linux kernel accepts two forms of clock adjustment:

   adjtimex(2) / clock_adjtime(2)
              |
              |  mode = ADJ_OFFSET (gradual adjust)
              |  mode = ADJ_SETOFFSET (instantaneous step)
              v
   Kernel timekeeping
              |
              |  ADJ_OFFSET: apply frequency adjustment
              |  ADJ_SETOFFSET: set the clock to a new value
              v
   System clock

The ADJ_OFFSET mode is what chrony uses by default. The kernel applies a frequency correction (in parts per million) that gradually brings the clock back into sync. The slew rate is bounded by the kernel’s max_adj constant, which is typically 10% (100,000 ppm). At 10% slew, a 1.4-second offset takes roughly 14 seconds to correct. A 100-second offset takes roughly 17 minutes.

The ADJ_SETOFFSET mode is what chronyc makestep triggers. The kernel sets the clock to the upstream time in one operation. The wall clock jumps. The monotonic clock is unaffected. In-flight telemetry that was stamped with the old wall clock value is now in the wrong position.

The chrony daemon uses the slew mode by default. The makestep directive in /etc/chrony/chrony.conf enables automatic stepping on startup, with a threshold (typically 1 second) and a maximum count (typically 3). After the maximum count, chrony slews only.

How to configure it

The mitigation configuration has two parts: the chrony configuration that controls automatic stepping, and the commands the operator runs to restore sync on a running host.

The chrony configuration:

# /etc/chrony/chrony.conf
# Three internal NTP sources.
pool ntp1.internal.example.com iburst maxsources 4
pool ntp2.internal.example.com iburst maxsources 4

# Public fallback.
server pool.ntp.org iburst

# Step the clock immediately if the offset exceeds 1 second,
# for the first 3 updates. After that, slew only.
makestep 1.0 3

# Sync the hardware clock to the system clock every 11 minutes.
rtcsync

# Drift file for cold-start.
driftfile /var/lib/chrony/chrony.drift

The two important directives for mitigation:

• makestep 1.0 3 — on startup, if the offset exceeds 1 second, step the clock immediately. The 3 is the maximum number of steps before the directive expires. After 3 steps, chrony slews only. A host that boots with a fresh image (no drift file) may need 3 steps to converge.
• rtcsync — write the system clock back to the hardware clock every 11 minutes. Without this directive, the RTC drifts from the system clock and the next boot reads the wrong time.

The commands the operator runs on a running host:

# SEVERITY: SERVICE-IMPACT (touches the wall clock)
# Force a step. The clock jumps to the upstream time.
chronyc makestep

# SEVERITY: READ-ONLY
# Wait for the daemon to sync. The command blocks until the
# clock is within 0.001 seconds of the upstream, or until the
# timeout expires. Useful in scripts that need to ensure the
# clock is correct before proceeding.
chronyc waitsync 0.001 60 0

# SEVERITY: SERVICE-IMPACT
# Restart the daemon. Used when the daemon is in a degraded
# state and the offset metric contradicts the chrony view.
systemctl restart chrony

The chronyc waitsync command is the right tool for a script that needs the clock to be correct before proceeding (boot order, post-resync validation). The chronyc makestep command is the right tool when the offset is large and the host is in a maintenance window. The systemctl restart chrony command is the right tool when the daemon is in a degraded state.

How to validate it

The validation reads three surfaces: the offset metric, the chrony user-space view, and the upstream source state.

READ-ONLY: confirm the offset after the mitigation.

chronyc tracking
# Reference ID    : C0A80101 (ntp1.internal.example.com)
# Stratum         : 3
# System time     : 0.000012345 seconds fast of NTP time
# Frequency       : 12.345 ppm (slow)
# Leap status     : Normal

The System time value should be below 0.001 seconds.

READ-ONLY: confirm the metric in Prometheus.

curl -s http://prometheus.monitoring.svc:9090/api/v1/query \
  --data-urlencode 'query=node_timex_offset_seconds{instance="checkout-01.internal"}'

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"node_timex_offset_seconds","instance":"checkout-01.internal"},"value":[1736822645.123,"-0.000001234"]}]}}

A value above 0.1 seconds is a hard incident.

READ-ONLY: confirm the upstream source is reachable.

chronyc sources -v

^* ntp1.internal.example.com   3   6   377    -0.0001  0.0001  0.0001
^+ ntp2.internal.example.com   3   6   377    +0.0002  0.0002  0.0002

The ^* prefix indicates the current best source. The Reach column should be 377.

How it can fail

Five failure modes appear repeatedly in mitigation.

1. Stepping in production traffic. The on-call engineer runs chronyc makestep during production traffic. The clock jumps by 1.4 seconds. In-flight spans have negative durations; the trace visualisation breaks for the next thirty seconds. The fix is to slew, not to step, when the host is in production traffic.
2. Restoring sync without fixing the cause. The on-call engineer runs chronyc makestep and the alert clears. The root cause (firewall, dead NTP source, daemon down) is still there. The clock drifts again. The fix is to investigate the cause before restoring sync.
3. The makestep directive has expired. After 3 steps, the makestep 1.0 3 directive is disabled. A host that boots with a fresh image and no drift file may need more than 3 steps to converge. The fix is to increase the count or to set the threshold higher.
4. The upstream NTP source is unreachable. chronyc makestep steps the clock to the upstream time, but the upstream time is itself wrong. The clock is now wrong in a different way. The fix is to investigate the upstream.
5. The slew rate is too slow for the offset. A 100-second offset takes roughly 17 minutes to slew at the kernel’s maximum slew rate. The host is in a maintenance window that is shorter than the slew time. The fix is to step and accept the discontinuity, or to extend the maintenance window.

How to troubleshoot it

The diagnostic order when a mitigation does not clear the alert:

1. Confirm the metric is now below the threshold. Query node_timex_offset_seconds in Prometheus. The value should match the chrony view.
2. Confirm the daemon is still running. systemctl status chrony. The daemon should be active.
3. Confirm the upstream source is reachable. chronyc activity. An “unreachable” source is a network problem.
4. Investigate the upstream source. The upstream may be wrong; the local clock is now wrong in a different way. Run chronyc tracking on the upstream.
5. Coinvestigate the firewall. UDP port 123 must be open from the host to the upstream. A blocked port is the most common cause of an unreachable source.
6. Coinvestigate the daemon log. journalctl -u chrony -n 50. The log may show why the daemon cannot reach the upstream.

The fix is rarely in the mitigation. The fix is at the upstream source or at the network path.

Security implications

The chronyc makestep command requires root or membership in the chrony access list. The right discipline is to restrict the chrony access list to operators only — a non-operator who can step the clock can forge the timestamps on every span and log line the host emits.

A wrong step during a security investigation can hide an attack. The operator steps the clock to bring it back into sync; the timestamps on the attacker’s actions are now in a different time range. The discipline is to capture the clock state (chronyc tracking output) before any mitigation, and to preserve the capture for forensics.

Performance implications

The mitigation is cheap. The chronyc command is a few milliseconds. The kernel’s slew is bounded by the kernel’s max_adj constant; a 1.4-second offset takes roughly 14 seconds at the maximum slew rate. The step is instantaneous.

The dominant cost is the operational cost of getting the mitigation wrong. A stepped clock during production traffic breaks the trace visualisation for thirty seconds. The team pages on the anomaly. The investigation takes longer than the original incident.

Production guidance

• Slew, do not step, when the host is in production traffic. The slew takes time but the wall clock is monotonic.
• Step, do not slew, when the host is in a maintenance window. The slew may take longer than the window.
• Investigate the cause before restoring sync. A clock that drifts without NTP discipline will drift again.
• Configure makestep 1.0 3 and rtcsync in the chrony configuration. The two together handle the boot-time large-offset and the RTC-drift cases.
• Restrict the chrony access list to operators only. The chronyc makestep command can forge timestamps.

Verification

You should now be able to answer:

• What is the difference between slew and step, and when is each one correct?
• Why is stepping the clock during production traffic worse than the original skew?
• What is the role of the makestep 1.0 3 directive in /etc/chrony/chrony.conf?
• When is the right answer to investigate the upstream NTP source rather than to step the local clock?
• How do you restrict the chronyc makestep command to operators only?

Quiz