Skip to main content
RunBook Academy

CephLXXX · Blocked OperationsBlocked Operations

Blocked and slow: the distinction that changes everything

Intermediate⏱ ~16 minceph

What you'll learn

  • Distinguish blocked from slow definitively
  • Explain why the responses differ
  • Apply the right first checks to each
  • Communicate the distinction to stakeholders

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

Slow operations complete and blocked ones do not. Performance work on a blocked condition achieves nothing, and structural investigation of a slow one wastes the window in which tuning would have helped.

The definitive test

ceph health detail | grep -oE '[0-9]+ (slow ops|ops).*blocked for [0-9]+'
sleep 120
ceph health detail | grep -oE '[0-9]+ (slow ops|ops).*blocked for [0-9]+'
ObservationCondition
Count falls, age boundedslow — operations completing
Count static, age grows by the elapsed timeblocked — nothing completing
Count grows, age boundedoverload
Count grows, age growsworsening; both

Two minutes and two commands settle it.

Why the responses differ

Slow:
  cause is contention, a device, a network path
  the cluster is functioning and constrained
  remedies: tuning, throttling, replacing hardware

Blocked:
  cause is a state that forbids progress
  the cluster is functioning correctly and refusing
  remedies: change the state

Tuning a blocked condition is the common error: throttles are adjusted, concurrency is changed, and nothing moves because nothing was ever going to.

The first checks for each

# slow — performance layers
IFACE=bond0
POOL=rbd-vms
ceph osd perf | sort -k2 -rn | head -5
iostat -x 1 5
ethtool -S ${IFACE} | grep -i err

# blocked — structural conditions
ceph -s
ceph health detail
ceph osd dump | grep flags
ceph osd pool get-quota ${POOL}
ceph quorum_status

The blocked checks are all cheap state lookups, which is why the sweep takes under a minute.

Communicating the distinction

To stakeholdersSlowBlocked
What is happeningrequests are taking longerrequests are not completing
Impactdegraded performanceeffective outage for affected data
Expected resolutionafter tuning or hardware workonce a specific condition is changed
Can it self-resolveoftenrarely

Requests to this pool are not completing rather than running slowly. The cluster is refusing them because a condition prevents progress; we have identified it as [X] and are changing it. Performance tuning would not have helped.

That framing correctly sets expectations in both directions.

Quiz

Knowledge check · 4 questions

  1. Q1. Two samples two minutes apart show the same operation count with the oldest age up by 120 seconds. What does this indicate?

  2. Q2. Twenty minutes of tuning that changes nothing is itself evidence about which condition you are facing.

  3. Q3. Respond to operations not completing.

    Operations against a pool are not completing. The team has spent twenty minutes adjusting mClock profiles and backfill settings with no effect.

  4. Q4. How should a blocked condition be described to stakeholders?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Sample the operation count and oldest age twice before choosing a response path — the age growing by exactly the elapsed time with a static count is the definition of blocked, not a proxy for it. Run the structural sweep rather than tuning; it takes under a minute and tuning cannot help.

Cross-course references

  • Kubernetes: a Pending pod and a slow pod need entirely different investigations
  • Linux: a process in uninterruptible sleep is not a performance problem