Headroom for a host failure

Why this matters in production

A cluster that cannot absorb one host failure is a cluster that stays degraded after one. The calculation is short and is rarely performed.

The requirement

requirement = data on the largest host
available   = Σ over remaining OSDs of (capacity × backfillfull_ratio − used)

# per-host data
ceph osd df tree | awk '
  /host/ {h=$NF}
  /osd\./ {s[h] += $5}
  END {for (k in s) printf "%s %.1f TiB\n", k, s[k]/1024}' | sort -k2 -rn

# available below backfillfull, excluding the largest host
ceph osd df | awk 'NR>1 && $1 ~ /^[0-9]+$/ {
  cap=$5; used=$6; avail += cap*0.90 - used
} END {printf "%.1f TiB below backfillfull\n", avail/1024}'

If the requirement exceeds the available figure, a host failure stalls.

The failure domain matters

With failure_domain=host and size=3, the replacement copies must go to hosts that do not already hold a copy of the PG:

ceph osd crush rule dump | grep -A5 'type'
ceph osd pool get rbd-vms crush_rule

On a cluster with exactly three hosts and size=3, losing one host means there is no valid third host — the PGs stay undersized regardless of free space:

3 hosts, size=3, one host lost
  → 2 hosts remain, each already holds one copy
  → CRUSH cannot place a third
  → PGs are active+undersized indefinitely

This is a topology limit, not a capacity one, and adding disks does not fix it.

Verifying

# what would happen — check the CRUSH rule can still place
ceph osd tree | grep -c '^-.*host'
ceph osd pool get rbd-vms size

# and the capacity
# (the two commands above)

Both checks matter, and they fail in different ways: the topology check failing means recovery is impossible; the capacity check failing means it stalls partway.

Tracking it as the cluster changes

# a check worth scheduling monthly
hosts=$(ceph osd tree | grep -c 'host ')
size=$(ceph osd pool get rbd-vms size -f json | python3 -c 'import sys,json;print(json.load(sys.stdin)["size"])')
[ "$hosts" -gt "$size" ] && echo 'topology OK' || echo 'TOPOLOGY: cannot recover a host failure'

The figure changes every time a host is added, an OSD is added, or usage grows — so a one-time check at deployment is not sufficient.

Quiz

Production discipline

Run both host-failure checks monthly: the topology check (host count exceeds pool size) and the capacity check (largest host’s data against space below backfillfull elsewhere). Compute the requirement from the largest host, not the average — expansions make hosts heterogeneous and the worst case is what matters.

Cross-course references

• Kubernetes: verifying the largest node’s pods can be rescheduled, not the average node’s
• Linux: sizing a spare against the largest array member rather than the typical one