Mapping the blast radius

Advanced⏱ ~18 minceph

What you'll learn

Map affected OSDs to pools and to workloads
Predict what each client type does when writes block
Detect the monitor store growth that a long incident causes
Record the radius before taking any action

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The people asking what is broken will not accept OSD numbers. Turning osd.37 is full into a list of affected services takes two commands, and without it every subsequent decision is made blind.

From OSD to pool to workload

ceph pg ls-by-osd 37 --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)
pgs = d.get("pg_stats", d)
pools = {}
for p in pgs:
    pools[p["pgid"].split(".")[0]] = pools.get(p["pgid"].split(".")[0], 0) + 1
for k, v in sorted(pools.items(), key=lambda x: -x[1]):
    print("pool id %-5s %5d pgs on osd.37" % (k, v))'

ceph osd pool ls detail | grep -E "^pool"
ceph osd pool stats

Every pool with a PG on a full OSD is affected, and only those pools. A
pool restricted to a different device class by its CRUSH rule is not
affected however alarming the cluster health looks.

What each client does when writes block

Client	Behaviour
krbd	IO hangs; the guest filesystem may remount read-only after its own timeout
librbd under QEMU	the guest stalls, then sees errors according to its `werror` policy
CephFS	writes block, reads continue, and the MDS reports slow requests
RGW	PUT returns an error to the client; GET continues to serve
librados application	the write call blocks and returns no error at all

ceph tell osd.37 dump_ops_in_flight | python3 -c '
import sys,json
ops = json.load(sys.stdin).get("ops", [])
print(len(ops), "ops in flight")
for o in ops[:5]:
    print(" %-8.1fs %s" % (o.get("duration",0), o.get("description","")[:70]))'

ceph fs status
ceph health detail | grep -iE 'slow|blocked'

A blocked write is not a failed write. Applications that would have retried an error instead accumulate threads, and the first visible symptom is often a connection pool exhausted somewhere three layers away.

The second incident

ceph health detail | grep -E 'MON_DISK|MON_CLOCK'
ceph config get mon mon_data_avail_warn
ceph config get mon mon_data_avail_crit
du -sh /var/lib/ceph/*/mon.*/store.db

What grows	Why	Consequence
monitor store	osdmaps are retained while PGs are not clean	a mon disk fills and the cluster loses quorum
PG logs on OSDs	trimming is limited while PGs are not clean	OSD space consumed by the incident itself
MDS journal	it cannot flush while metadata writes block	MDS memory and replay time grow

ceph tell mon.a compact

Compaction reclaims space inside the store but does not stop it growing; only clean PGs do that.

Record the radius before acting

ceph -s > /root/incident/$(date +%H%M)-status.txt
ceph osd df tree >> /root/incident/$(date +%H%M)-status.txt
ceph df detail >> /root/incident/$(date +%H%M)-status.txt

Quiz

Knowledge check · 4 questions

Q1. Why does a capacity incident that lasts a day threaten monitor quorum?

Monitors write far more log output while the cluster sits in `HEALTH_ERR`, and that output shares the volume holding the store Each blocked client write is buffered on the monitors until the pool leaves the full state, and nothing bounds how large that buffer grows The monitors take over placement group mapping once OSDs are full, which is a far heavier job than serving the osdmap Monitors retain every osdmap epoch any PG still needs, so trimming stops while PGs are unclean and the store grows until a mon disk fills
Q2. A write blocked by a full pool never returns an error to the application.

True False
Q3. Establish which services are affected by a full OSD.

`osd.37` is full. The cluster hosts RBD for virtualisation, a CephFS filesystem, and an RGW estate, with the RGW index pool on SSDs and everything else on HDD.
Q4. What grows during a long capacity incident that makes the situation worse independently of the original problem?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Convert OSD numbers into service names in the first ten minutes and state plainly which pools are not affected — a device class boundary usually excludes more than people expect. Watch the monitor store alongside the OSDs: a capacity incident that costs you quorum has become a different and much harder incident.

Cross-course references

Kubernetes: a blocked volume mount surfaces as pods stuck in ContainerCreating far from the cause
Linux: an NFS hard mount hangs rather than erroring, which is why the symptom appears elsewhere