The slow ops alert and its escalation

Why this matters in production

Slow ops during a recovery on a busy cluster is expected. The same alert with no recovery running and an idle cluster is a fault. The alert cannot distinguish them; the response must.

Context-dependent severity

- alert: CephSlowOpsWithRecovery
  expr: |
    ceph_health_detail{name="SLOW_OPS"} == 1
      and on() (ceph_pg_recovering + ceph_pg_backfilling) > 0
  for: 30m
  labels: { severity: ticket }
  annotations:
    summary: "Slow ops during recovery — expected, but check the duration"

- alert: CephSlowOpsNoRecovery
  expr: |
    ceph_health_detail{name="SLOW_OPS"} == 1
      and on() (ceph_pg_recovering + ceph_pg_backfilling) == 0
  for: 10m
  labels: { severity: page }
  annotations:
    summary: "Slow ops with no recovery running — a fault is likely"

Two rules from one condition, distinguished by context, with different severities and durations.

The investigation sequence

# 1. which daemons, and how long?
ceph health detail | grep -A5 SLOW_OPS

# 2. what stage are the operations stuck at?
for osd in $(ceph health detail | grep -oE 'osd\.[0-9]+' | sort -u | head -3); do
  echo "== $osd"
  ceph daemon "$osd" dump_ops_in_flight 2>/dev/null | python3 -c '
import sys,json
d=json.load(sys.stdin)
for op in d.get("ops",[])[:3]:
    print("  ", op.get("type_data",{}).get("flag_point"))' 2>/dev/null
done

# 3. is another OSD implicated?
ceph health detail | grep -oE 'waiting for subops from [0-9,]+'

# 4. the layer below
ceph osd perf | sort -k2 -rn | head -5

Four steps, each narrowing. The flag_point in step 2 is the highest-value output.

When to escalate

# is the oldest operation age growing?
ceph health detail | grep -o 'blocked for [0-9]*'
sleep 120
ceph health detail | grep -o 'blocked for [0-9]*'

An age growing by the elapsed time with no operations completing means blocked rather than slow, which is a different and more serious condition.

Slow versus blocked in the response

Quiz

Production discipline

Write two SLOW_OPS rules distinguished by whether recovery is running — one condition with opposite meanings cannot share a severity. Sample the oldest blocked age twice before investigating: growth matching elapsed time means blocked, which is a different and more urgent condition.

Cross-course references

• Kubernetes: the same alert during a rollout versus at steady state warrants different urgency
• Linux: distinguishing a slow process from a blocked one changes the entire investigation