Whether the recovery will fit

Why this matters in production

Recovery from a host loss writes that host’s data onto the survivors, and a cluster with plenty of free space in aggregate can still stall on a single OSD.

The arithmetic

ceph df
ceph osd df tree | tail -5

TOTAL   1.1 PiB  310 TiB  790 TiB   71.82
MIN/MAX VAR: 0.83/1.19  STDDEV: 4.12

A 12-OSD host in a 120-OSD cluster holds roughly a tenth of the data.
Recovery spreads that over the remaining 108, so each gains about a
ninth of its current contents — and the OSD already at VAR 1.19 gains
the same proportion as the one at 0.83.

ceph osd df --format json | python3 -c '
import sys,json
n = [o for o in json.load(sys.stdin)["nodes"] if o.get("device_class")]
n.sort(key=lambda o: -o["utilization"])
for o in n[:5]:
    print("osd.%-4d %6.2f%%  var %.2f" % (o["id"], o["utilization"], o["var"]))'

The three ratios

ceph osd dump | grep -E 'full_ratio|nearfull_ratio'

ceph health detail | grep -E 'OSD_NEARFULL|OSD_BACKFILLFULL|POOL_NEARFULL|PG_BACKFILL_FULL'
ceph pg ls backfill_toofull | head

Reading MAX AVAIL

ceph df detail

POOL          ID  STORED   OBJECTS  USED    %USED  MAX AVAIL
rbd-primary    3   210 TiB   55.31M  630 TiB  87.4    30 TiB

MAX AVAIL is derived from the most constrained OSD the pool's rule can
reach, divided by the replication factor. It falls faster than raw free
space as the distribution skews, which is exactly what a host loss does.

Device classes bound the arithmetic

ceph osd crush class ls
ceph osd df tree --class ssd | tail -3

Recovery only reaches OSDs the pool rule can select. Losing a host of
SSDs is absorbed by the remaining SSDs alone, and spare HDD capacity in
the same cluster contributes nothing to it.

Buying room during an incident

ceph osd set-backfillfull-ratio 0.92
ceph osd set-nearfull-ratio 0.87

ceph osd test-reweight-by-utilization 110
ceph osd reweight-by-utilization 110

Quiz

Production discipline

Size a recovery against the fullest OSD and the pool MAX AVAIL, never against average utilisation — backfill is bound by the most constrained device. Treat a plateau in the degraded percentage as a capacity stall until ceph pg ls backfill_toofull proves otherwise.

Cross-course references

• Kubernetes: a node with free memory in aggregate still evicts on the binding resource
• Linux: filesystem free space is meaningless when one device in the set is full