The blocked operations sweep

Why this matters in production

Every blocking condition is a cheap state lookup. A sweep covering all of them is faster than reasoning about which is most likely.

The sweep

#!/bin/bash
echo "== cluster state"
ceph -s

echo "== health detail"
ceph health detail | head -40

echo "== osd map flags"
ceph osd dump | grep -E '^flags|full_ratio'

echo "== monitor quorum"
ceph quorum_status --format json 2>/dev/null | python3 -c '
import sys,json; d=json.load(sys.stdin)
print("quorum:", d["quorum_names"])' || echo "QUORUM UNAVAILABLE"

echo "== PGs not active"
ceph pg dump pgs 2>/dev/null | awk '$10 !~ /active/ {print $1, $10}' | head -20

echo "== capacity"
ceph osd df | sort -k17 -rn | head -3

echo "== pool quotas"
for p in $(ceph osd pool ls); do
  q=$(ceph osd pool get-quota "$p" 2>/dev/null | grep -v 'N/A' | grep max)
  [ -n "$q" ] && echo "$p: $q"
done

echo "== blocklist"
ceph osd blocklist ls 2>/dev/null | head -10

Eight checks, under two minutes, covering every structural condition.

Interpreting the results

The last row matters: if the sweep finds nothing, the condition is probably slow rather than blocked, and the two-sample age test should be repeated.

Reaching the attribution

# PGID is one of the PGs the "PGs not active" check listed above:
PGID=3.1f

# for PGs not active, the specific reason
ceph pg "$PGID" query | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("state:", d.get("state"))
print("up:", d.get("up"), "acting:", d.get("acting"))
rs = d.get("recovery_state", [])
if rs:
    print("stage:", rs[0].get("name"))
    print("blocked_by:", rs[0].get("blocked_by"))
    print("comment:", rs[0].get("comment", "")[:120])'

blocked_by and the recovery state comment together usually name the exact cause.

Scripting it

# keep it in the repository next to the runbooks
git add runbooks/ceph/blocked-ops-sweep.sh
chmod +x runbooks/ceph/blocked-ops-sweep.sh

Sweep output should be captured, not read and discarded:
  ./blocked-ops-sweep.sh > /tmp/sweep-$(date +%s).txt 2>&1

Capturing it means the state at the time of the incident is preserved for the post-incident review, when the cluster has recovered and the evidence is gone.

Quiz

Production discipline

Script the blocking sweep with the quorum check first and commit it next to the runbooks — every condition is a cheap state lookup and a fixed order beats reasoning about likelihood. Capture the output to a file; the state at the time of the incident is gone once the cluster recovers.

Cross-course references

• Kubernetes: a fixed triage script beats ad hoc kubectl exploration under pressure
• Linux: capturing system state during an incident preserves evidence that vanishes