Working a blocked-ops report

Why this matters in production

A slow operation completes eventually. A blocked one does not, and the causes are structural rather than performance-related.

Slow versus blocked

The distinguishing observation is whether the age of the oldest operation keeps growing without any completing:

ceph health detail | grep -o 'blocked for [0-9]* sec'
sleep 60
ceph health detail | grep -o 'blocked for [0-9]* sec'

If the number grows by 60 and the count does not fall, operations are blocked rather than slow.

The blocking conditions

# a single sweep
ceph -s
ceph health detail
ceph osd dump | grep flags
ceph quorum_status --format json 2>/dev/null | python3 -c '
import sys,json; d=json.load(sys.stdin); print("quorum:", d["quorum_names"])'

Diagnosing each

# PGs not active
ceph pg dump pgs | awk '$10 !~ /active/ {print $1, $10}' | head

# why a specific PG is stuck
ceph pg 3.1f query | python3 -c '
import sys,json; d=json.load(sys.stdin)
rs=d.get("recovery_state",[])
for s in rs[:2]: print(s.get("name"), "|", s.get("comment","")[:100])'

The recovery_state and blocked_by fields name what the PG is waiting for, which is usually an OSD that is down.

ceph pg 3.1f query | python3 -c '
import sys,json; d=json.load(sys.stdin)
print("blocked_by:", d.get("peer_info") and [p["peer"] for p in d.get("peer_info",[])])
print("acting:", d.get("acting"), "up:", d.get("up"))'

Restoring progress

ADDR=10.20.0.11
ceph osd blocklist ls
ceph osd blocklist rm ${ADDR}

Lowering min_size deserves particular care: it permits I/O with less redundancy than the pool was designed for, and it must be restored.

Quiz

Production discipline

Distinguish blocked from slow by watching whether the operation count falls — a growing age with a static count means a structural condition, and performance tuning cannot help. Run the sweep (ceph -s, health detail, OSD flags, quorum status, quotas) before investigating anything else.

Cross-course references

• Kubernetes: a Pending pod and a slow pod need entirely different investigations
• Linux: a blocked process in D state has a structural cause, not a performance one