Clock Problems in Traces

The on-call engineer opens a trace of a checkout that the customer reported as slow. The trace has eight spans. The API gateway span starts at T. The inventory span starts at T+12ms. The warehouse picklist span starts at T-340ms. The inventory span’s parent is the API gateway; the warehouse picklist is a child of the inventory call. A child cannot start before its parent. The on-call engineer concludes the trace is wrong. They are right. The trace is right too — both timestamps are correct on the wall clocks of the respective hosts. The wall clocks are not consistent.

This is the lesson. Trace timing depends on every host agreeing on what time it is.

What it is

A clock problem in traces is the failure mode where the wall clocks of two hosts have drifted enough that the timestamps they stamp on spans are mutually inconsistent. The trace arrives at Tempo with a child that appears to start before its parent, or with two sibling spans whose durations overlap impossibly, or with an end-to-end latency that is negative.

The root cause is clock skew — the difference between the time reported by one host and the time reported by another. Skew is measured in milliseconds at well-disciplined hosts and seconds or minutes at undisciplined hosts.

Two kinds of time matter:

• Wall-clock time — the human time, read from clock_gettime(CLOCK_REALTIME) on Linux. It is subject to NTP discipline, jumps backwards when the clock is corrected, and can be set by an admin.
• Monotonic time — the steady time, read from clock_gettime(CLOCK_MONOTONIC). It never goes backwards. It is not synchronised across hosts. It cannot be set.

Span durations must use monotonic time. Span timestamps (the absolute start and end of a span) must use wall-clock time. The two are independent measurements, and confusing them is the cause of every clock problem in traces.

Why a sysadmin cares

Trace timing is the entire value proposition of distributed tracing. A trace that documents an impossible ordering of events is worse than no trace at all — it actively misleads the investigation.

Three operational payoffs ride on clock consistency:

1. Latency attribution. Tempo computes the latency histogram from span durations. If a child span is 200 ms long on its host but the parent’s wall clock says it lasted 350 ms, the histogram is wrong. The p99 the team monitors is not the p99 the customer experiences.
2. Causality reconstruction. Tempo renders spans in temporal order. A child that appears before its parent confuses the visualisation. The on-call engineer opens a trace and sees spans in an order that does not match the architecture.
3. Alerting on latency. A trace-derived SLO alert (p99 latency above threshold) reads from the same histogram. Wrong histograms produce wrong alerts. The team pages on a problem that is not there, or fails to page on a problem that is.

The fix is upstream. Tempo cannot recover a wrong timestamp.

How it works — the mental model

Every host stamps its spans with a wall-clock start time. Tempo orders the spans by their timestamps. The visualisation assumes the timestamps are mutually consistent.

Host A (wall clock correct)
  +-- span starts at T=1000.000s (wall)
  +-- span ends at   T=1000.350s (wall)
       |
       | outbound call to Host B
       v
Host B (wall clock 340ms behind Host A)
  +-- span starts at T=999.660s (Host B's wall)
  +-- span ends at   T=1000.010s (Host B's wall)

From Tempo’s perspective, the spans are:

Host B span  T=999.660 .. 1000.010  (340ms long)
Host A span  T=1000.000 .. 1000.350 (350ms long)

The Host B span appears to start 340 ms before the Host A span. The child-before-parent symptom is real even though no host did anything wrong. The wall clocks were just not in agreement.

The fix is NTP. The NTP daemon runs on every host, periodically queries a stratum-1 or stratum-2 server, and adjusts the local clock to match. A well-disciplined host stays within a few milliseconds of true time. An undisciplined host drifts by seconds per hour and is the cause of every clock problem in production.

How to configure it

The host discipline side — chrony, the modern replacement for ntpd:

# /etc/chrony/chrony.conf
# Use the data centre's local NTP servers as the primary source.
server ntp1.dc.internal iburst prefer
server ntp2.dc.internal iburst

# Stratum-3 public sources as a fallback for the data centre.
pool pool.ntp.org iburst maxsources 4

# Tolerate a small offset before correcting to avoid jumps.
makestep 1.0 3
rtcsync

# Allow the data centre's monitoring to query our offset.
allow 10.0.0.0/8

# Drift file for cold-start.
driftfile /var/lib/chrony/chrony.drift

The two important directives:

• makestep 1.0 3 — if the offset is greater than 1 second, step the clock immediately on the first three updates. After that, slew (gradual adjust) only. This avoids large jumps that would invalidate in-flight spans.
• iburst — on startup, send a burst of eight packets to speed up the initial synchronisation. A host that boots with a drifted clock comes into alignment within seconds.

The application side — the OpenTelemetry SDK reads the wall clock from the operating system. There is no SDK-level configuration for clock discipline. The fix is at the host.

The Prometheus self-monitoring side — a useful alert:

# /etc/prometheus/rules/ntp.yaml
groups:
  - name: ntp
    rules:
      - alert: HostClockSkew
        expr: abs(avg_over_time(node_timex_offset_seconds[5m])) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Host {{ $labels.instance }} clock skew exceeds 100 ms'
          description: 'The wall clock on this host has drifted by more than
            100 ms from NTP. Distributed-trace timestamps from this host
            are unreliable. Investigate the chrony daemon.'

A 100 ms threshold is conservative. Most production traces degrade visibly at 500 ms. The alert fires before the user- visible damage.

How to validate it

The validation ladder:

# 1. Is chrony running on every host?
systemctl status chrony
# Active: active (running) since ...
#
# (run this on every host that emits traces; a single
#  undisciplined host is enough to corrupt the fleet-wide picture)

# 2. What is the offset?
chronyc tracking
# Reference ID    : C0A80101 (ntp1.dc.internal)
# Stratum         : 3
# System time     : 0.000012345 seconds fast of NTP time
# Frequency       : 18.234 ppm fast
# ...
# (the "System time" line shows the offset in seconds; a healthy
#  offset is under 10 ms; a degraded offset is over 100 ms)

# 3. Is the offset stable or drifting?
chronyc sourcestats -v
# 210 Number of sources = 2
#   Source   NP  NR  Span  Freq  Offset  StdErr
#   ntp1...  12   8   34m  +1.234  -0.002   0.012
# (Offset should be under 10 ms and stable; "Span" is the time
#  the offset was measured over)

# 4. Are there any unreachable sources?
chronyc activity
# 200 OK ...
# (look for "unreachable" lines; a host with no NTP source has
#  drifted since boot)

# 5. Are spans from this host in temporal order with their parents?
# Use TraceQL to find spans that start before their parent.
tctl trace search --service=checkout --since=1h --limit=10 \
    | jq '.traces[].traceID' \
    | xargs -I{} sh -c 'tctl trace show {} | jq ".spans[] |
        select(.startTimeUnixNano < .parentSpanId)"'
# (the output should be empty; non-empty output is a clock-skew
#  signal)

# 6. Is the host's monotonic clock healthy?
# (the monotonic clock cannot be queried directly; if spans from
#  a host have negative durations, the SDK is using wall time
#  for duration. This is a SDK bug, not a clock bug.)

The first command is the answer. A single host with no chrony daemon is enough to corrupt the fleet-wide trace timeline.

How it can fail

Six recurring failure modes.

1. No NTP daemon installed. The host runs systemd and a workload, but chrony or ntpd was never installed. The clock drifts from the moment of boot. Symptom: spans from this host appear in the wrong order in Tempo; the offset from chronyc tracking returns “no NTP source”.
2. NTP is firewalled. The chrony daemon is installed and running, but port 123 (NTP) or 323 (chrony) is blocked outbound. The daemon cannot reach the time server. Symptom: chronyc activity shows “unreachable”; the offset grows over time.
3. The VM was suspended and resumed. Suspend/resume on a virtualised host pauses the wall clock but not the monotonic clock. The gap between resume and NTP re-sync is a clock skew window. Symptom: a small number of spans in a window of time have timestamps that are minutes off.
4. The container runtime did not sync the clock. The container’s clock is inherited from the host. If the host is wrong, every container on it is wrong. Symptom: the Kubernetes node has a drifted clock; all pods on the node emit wrong timestamps.
5. The chrony daemon is in a tight loop correcting. The makestep directive is set to a small threshold; the clock is stepping every minute. The wall clock jumps backwards. Symptom: in-flight spans have negative durations; the trace visualisation breaks.
6. Two data centres, two time sources. A trace that crosses a data-centre boundary crosses a time-discipline boundary. The east coast data centre is on ntp1.dc-east.internal; the west coast is on ntp1.dc-west.internal. The two sources disagree by 50 ms. Symptom: traces that cross the boundary have child-before-parent spans at the cross-DC hop.

How to troubleshoot it

The diagnostic order:

1. Is chrony running on every host? systemctl status chrony. The simplest answer is the most common one.
2. What is the offset? chronyc tracking. A healthy offset is under 10 ms. A degraded offset is over 100 ms.
3. Is the offset stable? chronyc sourcestats -v. A growing offset is a host that is not being disciplined (firewall, misconfigured source, dead NTP server).
4. Are the sources reachable? chronyc activity. An “unreachable” source is a network problem.
5. Is the offset bounded across the fleet? A Prometheus query against node_timex_offset_seconds shows every host’s offset. The distribution should be tight; a long tail is a sign of partial discipline.
6. Are there traces with negative durations? A Tempo query for span.duration < 0 returns the SDK-level instrumentation bug. The fix is in the SDK code that computes the duration, not in the host.

Security implications

NTP traffic is UDP on port 123 (or 323 for chrony). It is a denial-of-service vector for a misconfigured host (an attacker can flood NTP requests and saturate the daemon) and an amplification vector if the daemon is open to the public internet. Bind the NTP daemon to the internal network only.

A second-order risk is around the accuracy of the wall clock for logging. A wrong clock produces wrong timestamps in logs and metrics, which complicates forensics. The remediation is the same: NTP discipline.

A third-order risk: an attacker who can write to the wall clock can forge the timestamps on every span the host emits. The attacker uses NTP’s symmetric mode or sets the system clock directly. The remediation is to bind NTP to authenticated sources (chrony supports NTS — Network Time Security) and to log changes to the system clock at the kernel level.

Performance implications

The cost of NTP discipline is small. The chrony daemon uses roughly 5 MB of RAM and negligible CPU on a modern host. The network cost is one NTP query every few minutes — a few hundred bytes per hour.

The performance cost of not having NTP discipline is the cost of the misleading trace timeline, which is paid in incident response. A team that cannot trust the trace timeline during an incident works harder, takes longer, and arrives at worse decisions.

Production guidance

• Install chrony on every host that emits traces. systemd alone does not discipline the clock. The fix is a daemon.
• Use internal NTP sources as the primary. Public pools are a fallback. The internal sources are reachable, controlled, and monitored.
• Alert on node_timex_offset_seconds. The threshold is 100 ms for warning, 500 ms for critical. The offset should be tight.
• Configure makestep conservatively. A small threshold with frequent steps invalidates in-flight spans. A threshold of 1 second with at most 3 steps is a reasonable default.
• Test suspend/resume. A virtualised host that suspends must come back into discipline quickly. Verify in staging.
• Use NTS where available. chrony supports Network Time Security. The cryptographic authentication prevents an attacker from forging NTP responses.

Verification

You should now be able to answer:

• What is the difference between wall-clock time and monotonic time?
• Why does a child span ever appear to start before its parent?
• What is the most common production cause of clock skew?
• How do you alert on host clock skew in Prometheus?

Quiz