Blocked requests and full-cluster conditions

Advanced⏱ ~18 minceph

What you'll learn

Interpret slow ops and blocked request warnings
Identify the OSD and stage responsible
Handle nearfull, backfillfull, and full conditions
Restore service when writes are blocked by capacity

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Blocked requests mean clients are waiting, and the capacity thresholds that most often cause them are a graduated series that ends with the cluster refusing all writes. Recognising which threshold you have crossed determines how much time you have.

Reading the warning

ceph health detail

[WRN] SLOW_OPS: 12 slow ops, oldest one blocked for 94 sec,
      osd.31 has slow ops

The OSD is named. That is usually the whole diagnosis:

ceph daemon osd.31 dump_blocked_ops
ceph daemon osd.31 dump_historic_ops

The stage the operations are stuck at tells you where to look:

Stage	Meaning
`waiting for rw locks`	contention on the same objects
`waiting for subops from [...]`	a peer OSD is slow — investigate those
`waiting for degraded object`	recovery must complete first
`waiting for scrub`	a scrub holds the chunk
no progress past `queued_for_pg`	the OSD is saturated or wedged

The capacity thresholds

ceph osd df | sort -k17 -rn | head
ceph health detail | grep -i full

Threshold	Default	Effect
`nearfull_ratio`	0.85	warning only; latency already degrading
`backfillfull_ratio`	0.90	backfill refuses this OSD as a destination
`full_ratio`	0.95	writes to affected PGs are refused

Crossing full blocks writes to every PG on that OSD, which for a well-distributed pool means effectively all writes.

ceph osd set-nearfull-ratio 0.85
ceph osd set-backfillfull-ratio 0.90
ceph osd set-full-ratio 0.95

Restoring service when full

In order of preference:

# 1. rebalance away from the fullest OSDs — no capacity added, no risk
POOL=rbd-vms
IMAGE=vm-disk-01
ceph osd reweight-by-utilization 110
ceph balancer status
ceph balancer on

# 2. reclaim space
rbd snap ls ${POOL}/${IMAGE}          # old snapshots
rbd trash ls ${POOL}                 # deleted images awaiting purge
rbd trash purge ${POOL}

# 3. temporary threshold increase — buys hours, not a solution
ceph osd set-full-ratio 0.96

# 4. add capacity
ceph orch apply osd --all-available-devices

Raising the full ratio is a stopgap that consumes the safety margin. Each increment leaves less room before the OSD is genuinely out of space, at which point recovery itself cannot proceed.

Quiz

Knowledge check · 4 questions

Q1. One OSD out of 200 crosses full_ratio. What is the effect on client writes?

Only writes landing on that OSD are refused, so roughly 0.5% of client traffic The cluster redistributes that OSD's data automatically, so client writes continue uninterrupted Writes to every PG that OSD participates in are refused — a large fraction of the pool Reads are blocked while writes continue, since only reads touch the full device
Q2. Repeatedly raising full_ratio is an acceptable way to manage a cluster approaching capacity.

True False
Q3. Restore writes on a cluster where an OSD has crossed full.

Production writes have stopped. `ceph health detail` shows osd.117 above full_ratio at 95.4%; the cluster average is 79%. Additional capacity is two weeks away.
Q4. Why is the full threshold set at 0.95 rather than closer to 1.0?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Alert on per-OSD utilisation variance alongside the cluster average; one OSD crossing full stops the workload while the average looks healthy. Keep the balancer enabled and verify it is keeping pace, since distribution is what determines effective capacity rather than the aggregate figure.

Cross-course references

Kubernetes: a single node at disk pressure evicts pods while the cluster shows spare capacity
Linux: a filesystem at 100% halts writers regardless of free space on other mounts