Capacity for recovery and backfill

Why this matters in production

A cluster that cannot absorb a host failure discovers this during the failure, when the options are worst. The check takes a minute and can be run at any time.

Verifying host failure absorption

# how much does the largest host hold?
for h in $(ceph osd tree | awk '/host/ {print $NF}'); do
  echo -n "$h "
  ceph osd df tree | awk -v h="$h" '$0 ~ h {found=1} found && /osd\./ {s+=$5} END {print s}'
done

# how much free space is there elsewhere?
ceph df | grep -A2 'RAW STORAGE'

The comparison: can the remaining hosts hold the largest host’s data without any OSD reaching backfillfull?

ceph osd df | sort -k17 -rn | head -5
ceph osd df | awk 'NR>1 {n++; s+=$17} END {print "avg", s/n}'

If the fullest OSD is at 78% and the failure would add roughly 11 percentage points, it would land at 89% — past backfillfull at 0.90.

Checking before adding or removing OSDs

Removing an OSD requires the remaining ones to absorb its data:

ceph osd df | grep '^ *13 '     # what it holds
ceph osd df | sort -k17 -rn | head -3    # where it will go

Adding an OSD is safe from a capacity perspective — it increases capacity — but backfilling into it still requires the sources to complete their part, and a cluster already past backfillfull cannot start.

ceph health detail | grep -i backfillfull

Recognising a capacity-constrained recovery

HEALTH_ERR ...
[WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself)
    pg 3.a2 is active+undersized+degraded+remapped+backfill_toofull

backfill_toofull in a PG state is the signal. The PG has somewhere it needs to go and the destination is too full to accept it.

ceph pg dump pgs | grep toofull | head
ceph health detail | grep -A5 BACKFILL_FULL

Restoring progress

# 1. rebalance first — often enough on its own
ceph balancer status
ceph balancer on

# 2. delete what can be deleted
rbd snap ls --all rbd-vms/vm-disk-1
ceph df detail

# 3. raise the ratio temporarily — deliberate and reversible
ceph osd set-backfillfull-ratio 0.92
# restore it once recovery completes
ceph osd set-backfillfull-ratio 0.90

# 4. add capacity
ceph orch apply osd --all-available-devices

The order matters. Raising the ratio buys room to move data but removes the margin protecting against the full ratio, so it belongs after the options that create real space.

Quiz

Production discipline

Verify host failure absorption monthly — compare the largest host’s data against free space elsewhere and check whether any OSD would pass backfillfull. When recovery stalls, balance and delete before raising the backfillfull ratio; that ratio is the margin protecting client writes, so raising it is temporary and must be restored.

Cross-course references

• Kubernetes: verifying the cluster can reschedule a node’s pods before losing one
• Linux: confirming a RAID array can rebuild within its remaining capacity