Correlating evidence across layers

Advanced⏱ ~18 mincephjournalctl

What you'll learn

Align evidence from different layers by time
Distinguish cause from consequence in a correlation
Build a timeline from mixed sources
Avoid the common correlation errors

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A single event produces different symptoms at each layer, at slightly different times. Aligning them separates the cause from its consequences.

Sources and their timestamps

# the fsid names the systemd units; substitute your own client unit:
FSID=$(ceph fsid)
CLIENT_UNIT=app-api.service

# cluster events
ceph log last 200
ceph crash ls

# OSD daemon logs
journalctl -u "ceph-$FSID@osd.12" --since '2 hours ago'

# kernel, on the OSD host
dmesg -T | tail -50

# client
journalctl -u "$CLIENT_UNIT" --since '2 hours ago'

# metrics
# Prometheus range queries around the window

All of these must agree on the time, which is why clock synchronisation is a prerequisite for any correlation work:

ceph time-sync-status
chronyc tracking

Building a timeline

14:02:11  kernel (ceph-osd-03): I/O error, dev sdf, sector 91238400
14:02:11  osd.12 log: bdev read failed
14:02:14  ceph log: osd.12 marked down by osd.31
14:02:14  ceph log: osd.12 marked down by osd.7
14:02:31  ceph -s: 412 PGs degraded, recovery starting
14:02:35  client: write latency p99 12ms → 47ms
14:03:02  application: request timeouts begin

Read top to bottom: the kernel error precedes everything, and each subsequent line follows from the one above. The cause is at the top and every other line is a consequence.

Distinguishing cause from consequence

Pattern	Reading
A precedes B consistently	A may cause B
A and B start simultaneously	both may follow a common cause
B precedes A	A does not cause B
A precedes B once	insufficient evidence

The most common error is treating the first observed symptom as the cause — usually the client-facing one, because that is what generated the report. The client symptom is almost always last in the timeline.

Correlation errors to avoid

Error	Example
Assuming the reported symptom is the cause	“clients are slow, so the network”
Ignoring clock skew	events appearing out of order across hosts
Correlating with too coarse a resolution	1-minute metrics hide a 10-second cause
Stopping at the first correlation	recovery correlates with latency, but why did recovery start
Confusing a trigger with a condition	the reboot triggered it; the misconfiguration allowed it

The last is worth dwelling on: an event that reveals a latent problem is not the same as the problem.

# check clock agreement before trusting any cross-host timeline
ceph time-sync-status | python3 -c '
import sys,json; d=json.load(sys.stdin)
for k,v in d.get("time_sync",{}).get("monitors",{}).items():
    print(k, v.get("skew"))'

Quiz

Knowledge check · 4 questions

Q1. Why are logs more useful than metrics for establishing causal ordering?

Logs record exact values, while metrics are averages that lose the peak the incident was caused by Metrics come from the mgr, which reports the cluster's aggregate view rather than the daemon that saw the event Logs are retained for months, while metrics roll up to a lower resolution after a few days Logs carry sub-second timestamps, while a 60-second scrape smears cause and effect into the same sample
Q2. The symptom that generated the incident report is usually the cause.

True False
Q3. Correlate evidence for a latency incident.

Clients reported latency spikes at 14:05. Metrics show recovery starting at 14:03 and OSD latency rising at 14:02. The team concludes recovery caused the latency and plans to throttle it.
Q4. What is the difference between a trigger and a condition in an incident timeline?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Build the timeline backwards from the client report toward the OSD and kernel logs; the reported symptom is almost always last and the cause is minutes earlier. Verify clock synchronisation before trusting any cross-host ordering, and use logs for causal ordering and metrics for magnitude.

Cross-course references

Kubernetes: event timelines across pod, node, and control plane need the same alignment
Linux: correlating application logs with dmesg requires the same clock discipline