Log correlation and why it needs synchronised time

Why this matters in production

Time synchronisation costs nothing until an incident, at which point its absence costs the ability to establish what caused what. Causality across hosts is inferred from timestamps, and without agreement there is no causality to infer.

The problem

ceph-osd-01  10:14:22.481  osd.12 marked down
ceph-mon-01  10:14:19.203  osd.12 reported failed by osd.47
ceph-osd-03  10:14:25.891  osd.47 heartbeat_check: no reply from osd.12

Read at face value, osd.47 complained about osd.12 three seconds after the monitor had already acted on its report. Either the clocks disagree or the sequence is impossible, and you cannot tell which — so nothing in the sequence can be trusted.

Consistent timestamps

ceph config get global log_to_file
ceph config get global log_file
ceph config set global log_stderr_prefix ""

journalctl -u ceph-osd@12 --output=short-iso-precise
journalctl --since '2026-08-18 10:00:00' --until '2026-08-18 10:30:00' -u 'ceph-*'

Ceph logs in UTC. Configure your log aggregation to preserve that rather than converting to local time, since a fleet spanning time zones converted inconsistently is as bad as unsynchronised clocks.

Correlating across daemons

# a window across every Ceph unit on a host
journalctl -u 'ceph-*' --since '10:14:00' --until '10:15:00' --output=short-iso-precise

# the cluster log, which aggregates monitor-visible events
ceph log last 200
ceph log last 200 cluster
ceph log last 200 audit

The cluster log is the anchor: it records events with monitor timestamps, which are consistent by construction because the monitors agree on time or warn that they do not.

Preserving evidence

During an incident, capture before intervening:

INCIDENT=/var/tmp/incident-$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p "$INCIDENT"
ceph -s                        > "$INCIDENT/status"
ceph health detail             > "$INCIDENT/health"
ceph osd tree                  > "$INCIDENT/osdtree"
ceph pg dump_stuck             > "$INCIDENT/stuck"
ceph log last 1000             > "$INCIDENT/clusterlog"
for h in $(ceph orch host ls --format json | jq -r '.[].hostname'); do
  ssh "$h" "journalctl -u 'ceph-*' --since '-1 hour' --output=short-iso-precise" \
    > "$INCIDENT/journal-$h" 2>/dev/null
done

The state moves as you work, so the capture is what lets you explain afterwards what was actually happening.

Quiz

Production discipline

Capture cluster status, health detail, and per-host journals at the start of an incident before intervening; the state moves and the capture is the only record of what preceded your actions. Preserve UTC through log aggregation rather than converting per host, since inconsistent conversion recreates the problem synchronisation solves.

Cross-course references

• Kubernetes: correlating events across nodes and the API server has the identical requirement
• Linux: centralised logging with consistent timestamps is standard practice for this reason