Cluster IR: building one timeline from nodes whose clocks disagree

Single-host troubleshooting has one clock, so the order of events in the journal is the order they happened. linux-evidence-based-diagnosis builds on that: capture before you disturb, and read the record.

A cluster has one clock per node, and the incident you are investigating is often the reason they stopped agreeing. Merging four journals by timestamp and reading the result as a sequence is the most common way a cluster post-incident review reaches a confidently wrong conclusion — usually “node2 failed first”, when node2 was simply eleven seconds ahead.

Measure the offset before you trust the timestamps

Do this first, on every node, and write the numbers down. It takes thirty seconds and it decides whether the rest of the evidence means anything.

$ chronyc tracking

Reference ID    : C0000206 (ntp1.example.com)
Stratum         : 3
Ref time (UTC)  : Tue Aug 11 06:41:12 2026
System time     : 0.000284 seconds slow of NTP time
Last offset     : -0.000121 seconds
RMS offset      : 0.000318 seconds
Frequency       : 12.043 ppm slow
Skew            : 0.094 ppm
Root delay      : 0.004118 seconds
Root dispersion : 0.001204 seconds
Update interval : 64.2 seconds
Leap status     : Normal

System time is the number that matters: how far this node’s clock is from the time source it is following. A node whose logs are usable reports microseconds or low milliseconds here. A node reporting seconds has a timeline you cannot use directly.

Leap status is the second field to read. Anything other than Normal — Not synchronised, in particular — means the node is free-running and has been drifting for however long it has been disconnected.

$ chronyc tracking | grep -E 'System time|Leap status'

System time     : 11.284937 seconds fast of NTP time
Leap status     : Not synchronised

Collect it from every node in one pass:

# Substitute your own values before running:
NODES="node1 node2 node3"

for n in $NODES; do
  printf '%-8s ' "$n"
  ssh -o BatchMode=yes "$n" \
    "chronyc tracking | awk -F: '/System time|Leap status/ {print \$2}' | tr -d '\n'"
  echo
done

Anchor on an event every node observed

Even with offsets measured, arithmetic on four journals is error-prone. The stronger technique is to find one event that appears in every node’s log and use it as the origin. Everything else is then expressed as an interval from the anchor, on that node’s own clock, and intervals are unaffected by offset.

Good anchors, in rough order of usefulness:

The last row is the most valuable and the most often forgotten. A device outside the cluster that watched the incident happen — a load balancer, a reverse proxy, an external monitor, a client application — gives you a timeline with one clock and no stake in the argument.

# Substitute your own values before running:
RING_ID=1.2f4

# The ring ID is a shared identifier: the same membership event
# on every node, regardless of what each node thinks the time is
for n in node1 node2 node3; do
  echo "=== $n"
  ssh -o BatchMode=yes "$n" \
    "journalctl -u corosync --utc -o short-iso-precise --since '-3 hours' | grep -F '$RING_ID'"
done

Collect in a form that merges

Two flags decide whether the captures you take can be combined later. Both are easy to omit and impossible to add retrospectively.

• --utc removes the time-zone question. Nodes in different regions, or a node whose /etc/localtime was never set during the build, otherwise produce logs that differ by whole hours and look like enormous skew.
• -o short-iso-precise gives sortable, microsecond timestamps. The default short format prints Aug 11 06:41:12 — no year, no zone, second resolution — which sorts wrongly across a month boundary and cannot separate events inside the same second.

$ journalctl -u pacemaker --utc -o short-iso-precise -n 3

2026-08-11T06:02:14.318442+0000 node1 pacemaker-controld[2891]: notice: State transition S_IDLE -> S_POLICY_ENGINE
2026-08-11T06:02:14.902117+0000 node1 pacemaker-schedulerd[2889]: warning: Cluster node node3 will be fenced: peer is no longer part of the cluster
2026-08-11T06:02:15.004930+0000 node1 pacemaker-fenced[2886]: notice: Requesting peer fencing (reboot) targeting node3

The collection script, run per node, into per-node files:

# Substitute your own values before running:
NODES="node1 node2 node3"
SINCE="2026-08-11 05:30:00"
OUT=/var/tmp/inc-4820

mkdir -p "$OUT"
for n in $NODES; do
  ssh -o BatchMode=yes "$n" \
    "journalctl --utc -o short-iso-precise --since '$SINCE' \
       -u corosync -u pacemaker -u pcsd" > "$OUT/$n-cluster.log"
  ssh -o BatchMode=yes "$n" "chronyc tracking" > "$OUT/$n-clock.txt"
  ssh -o BatchMode=yes "$n" "journalctl --list-boots" > "$OUT/$n-boots.txt"
done

Keep them per node. A single merged file loses the one attribute you need most — which node said which line — and merging is something you can always do later, from files you kept separate.

Pacemaker also ships a collector that does the cross-node gathering for you, including the CIB and the fence history:

# Substitute your own values before running:
FROM='2026-08-11 05:30:00'
DEST=/var/tmp/inc-4820-cluster-report

# Produces a tarball gathered from every node in the cluster
sudo pcs cluster report --from "$FROM" "$DEST"

Use it in addition to your own captures rather than instead of them. It runs against the cluster, so a node that is fenced, powered off or partitioned at collection time contributes nothing — and that is usually the node you most wanted.

Writing the timeline down

State what you measured and what you inferred. A timeline that hides its uncertainty is a timeline someone will build a root cause on.

INC-4820 TIMELINE (all times UTC, anchored on the fence of node3)

Clock offsets measured at 06:44Z, before any correction:
  node1  -0.0003s   (synchronised)
  node2  +11.2849s  (Leap status: Not synchronised)
  node3   unknown   (fenced; last chronyc sample unavailable)

Anchor: fence of node3, completed 06:02:19 per node1 (node1 clock
        is trustworthy, so the anchor is taken as true UTC).

T-00:05  node2 journal: monitor timeout on fs-data
         [node2 clock +11.28s; corrected time 06:02:03 +/- 0.5s]
T-00:00  node1: "Cluster node node3 will be fenced" -> fence issued
T+00:04  node1: fence of node3 successful
T+00:11  ext-lb health check marks vip 192.0.2.100 down
         [external device, independent clock, not corrected]
T+01:52  node1: fs-data started on node1

UNRESOLVED: node3 recorded nothing after 06:01:58 on its own clock.
Its offset was never measured, so events on node3 cannot be placed
relative to this timeline more precisely than "before the fence".

The UNRESOLVED block is not an admission of failure. It is the part that stops the review from asserting an ordering the evidence does not support, and it names a concrete gap: node clock offset is not collected before a fence, so a fenced node’s timeline is always unanchorable. That is an action item.

Knowledge check