Blocked by capacity thresholds

Why this matters in production

Capacity blocking is the most common structural cause, and the three thresholds block different things — so the symptom identifies which threshold has been crossed.

The mapping

ceph osd dump | grep -E 'full_ratio|nearfull_ratio'
ceph osd df | sort -k17 -rn | head -5
ceph health detail | grep -E 'FULL|NEARFULL'

The symptom names the threshold: stalled recovery with working client I/O is backfillfull; blocked writes is full.

Diagnosing

# which OSDs, and how far over?
ceph osd df | awk 'NR>1 && $17+0 > 85 {print $1, $17"%"}'

# which PGs are affected?
ceph pg dump pgs | awk '/toofull/ {print $1, $10}' | head

# is it one OSD or many?
ceph osd df | awk 'NR>1 && $17+0 > 90' | wc -l

# is it a genuine shortage or imbalance?
ceph osd df | awk 'NR>1 {n++; s+=$17; if ($17+0>m) m=$17+0}
  END {printf "max %.1f avg %.1f spread %.1f\n", m, s/n, m-s/n}'

A wide spread means the capacity exists elsewhere and the blocking OSD is an outlier.

Restoring progress in order

# 1. rebalance, if the spread is wide
BUCKET=acme-data
ceph balancer status && ceph balancer on

# 2. delete
rbd snap ls --all --pool rbd-vms
radosgw-admin bucket check --bucket=${BUCKET}

# 3. reduce consumption without moving data
ceph osd pool set scratch size 2

# 4. raise the ratio temporarily
ceph osd set-backfillfull-ratio 0.92

# 5. add capacity
ceph orch apply osd --all-available-devices

Steps 2 and 3 free bytes without needing a destination, which is why they work when nothing else does. Step 4 permits movement into the safety margin and must be restored.

OSD-level versus pool-level

# OSD capacity
ceph osd df | sort -k17 -rn | head -3

# pool quota
ceph osd pool get-quota rbd-vms

OSD full:   affects every pool with PGs on that OSD
Pool quota: affects that pool only, regardless of device capacity

A pool blocked while the cluster has ample space is a quota, and the fix is a quota change rather than any capacity work.

ceph osd pool set-quota rbd-vms max_bytes 80T
ceph osd pool set-quota rbd-vms max_objects 0     # 0 removes the limit

Quiz

Production discipline

Read the symptom to identify the threshold: stalled recovery with working client I/O is backfillfull, blocked writes is full. Include ceph osd pool get-quota in the standard sweep — a pool blocked with devices barely used is a quota and no capacity work will help.

Cross-course references

• Kubernetes: a namespace ResourceQuota blocks with cluster capacity available
• Linux: a filesystem quota and a full device produce the same ENOSPC