Correlating slow ops with cluster state

Why this matters in production

The cluster state at the moment of a slow ops report narrows the cause sharply, and reading it takes one command. Working from state to cause is faster than working from evidence to hypothesis.

The decision procedure

flowchart TD
  A[SLOW_OPS reported] --> B{OSDs down?}
  B -->|yes| C[The down OSDs and their cause]
  B -->|no| D{Recovery running?}
  D -->|yes| E{Slow ops concentrated?}
  E -->|yes| F[A fault exposed by recovery]
  E -->|no| G[Recovery load]
  D -->|no| H{Capacity warnings?}
  H -->|yes| I[Nearfull or full]
  H -->|no| J{Scrub running?}
  J -->|yes| K[Scrub contention]
  J -->|no| L{One OSD or many?}
  L -->|one| M[Device or host]
  L -->|many| N[Network or client]

The single command

ceph -s

  cluster:
    health: HEALTH_WARN
            47 slow ops, oldest one blocked for 92 sec
  services:
    osd: 96 osds: 95 up, 96 in            ← one OSD down
  data:
    pgs:  3891 active+clean
          102  active+recovering+degraded  ← recovery running

That output answers three of the branches at once: an OSD is down, recovery is running, and no capacity warning is present.

The correlation table

Reaching the attribution

# a compact evidence gather
{
  echo "== state"; ceph -s
  echo "== health"; ceph health detail | head -30
  echo "== osd perf"; ceph osd perf | sort -k2 -rn | head -5
  echo "== flags"; ceph osd dump | grep flags
  echo "== slow op stages"
  for o in $(ceph health detail | grep -oE 'osd\.[0-9]+' | sort -u | head -3); do
    echo "-- $o"
    ceph daemon "$o" dump_ops_in_flight 2>/dev/null | python3 -c '
import sys,json
from collections import Counter
d=json.load(sys.stdin)
c=Counter(op.get("type_data",{}).get("flag_point") for op in d.get("ops",[]))
[print("   %4d %s" % (v,k)) for k,v in c.most_common(3)]' 2>/dev/null
  done
} > /tmp/slowops-$(date +%s).txt

One script, one file, everything the correlation needs.

Recording it

Slow ops incident, 2026-08-18 14:02
State:      1 OSD down, recovery running, no capacity warnings
Spread:     14 OSDs across 9 hosts
Flag point: waiting for subops from [31] dominant
Attributed: osd.31 — device with 74 pending sectors
Action:     drained, replaced; slow ops cleared
Time to attribution: 12 minutes

Recording the time to attribution is what shows whether the procedure is improving.

Quiz

Production discipline

Run ceph -s before gathering any specific evidence — it eliminates most causes in one command and keeps the investigation from accumulating observations. Record the cluster state, the attribution, and the time taken; the record is what makes the next incident faster.

Cross-course references

• Kubernetes: checking cluster events before pod logs narrows the same way
• Linux: system state before process-level detail is the standard triage order