How much free space to keep

Intermediate⏱ ~17 minceph

What you'll learn

Enumerate what headroom is reserved for
Compute a headroom requirement for a cluster
Justify the figure to stakeholders
Monitor headroom rather than utilisation alone

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Headroom looks like unused capacity that was paid for, which makes it the first thing challenged. The defence is arithmetic: each component of it covers a specific event.

What headroom covers

Component	Covers	Typical size
Host failure absorption	one host’s data re-replicated onto the rest	1/N of usable
Imbalance	the gap between average and fullest OSD	5–15%
Full ratio reserve	the 5% above the full ratio	5%
Growth lead time	the data arriving before an order lands	order lead time × growth rate
Maintenance	a host out for a rebuild	1/N of usable

These do not simply add — host failure and maintenance overlap — but the largest two do need to coexist.

Computing the requirement

For a 10-host cluster growing at 4 TB/week with an 8-week procurement lead time:

host failure absorption   = 1/10 = 10%
imbalance allowance       =         8%
full ratio reserve        =         5%
growth during lead time   = 4 TB × 8 = 32 TB

# usable capacity
ceph df | grep -A2 'POOLS' | awk '/rbd/ {print $NF}'

# current imbalance
ceph osd df | awk 'NR>1 {print $17}' | sort -n | sed -n '1p;$p'

If usable capacity is 140 TB, the fixed components are 23% ≈ 32 TB, and the growth allowance is another 32 TB — so the cluster should trigger an expansion at roughly 76 TB stored, not at 130 TB.

Justifying it

The persuasive framing is what happens without it:

Without the host failure allowance, losing one host means the remaining nine cannot absorb its data. Recovery stalls at backfillfull, the cluster stays degraded, and adding capacity is the only way out — which takes eight weeks.

That is a concrete outcome, unlike “we should keep 25% free”.

Monitoring headroom, not utilisation

# the number that matters
ceph df | grep MAX

# the constraint
ceph osd df | sort -k17 -rn | head -3

An alert on cluster utilisation crossing 70% is less useful than one on the fullest OSD crossing 70%, because that is what binds. Better still is an alert on the projected date of crossing:

# weekly stored growth
ceph df detail | awk '/rbd-vms/ {print $4}'

Quiz

Knowledge check · 4 questions

Q1. Why does host failure absorption require roughly 1/N of usable capacity free on an N-host cluster?

A full copy of the failed host OSD data is staged in the reserve before it is redistributed The failed host's share of data must be re-replicated onto the remaining hosts, which need that much free space The reserve keeps a host's worth of space free so a replacement host can be rebuilt in place without moving data CRUSH holds back 1/N of every OSD as a hidden reserve that only recovery is permitted to write into
Q2. An alert on cluster average utilisation crossing 70% is the most useful capacity alert.

True False
Q3. Justify headroom against a challenge.

A 4-host cluster is at 55% utilisation. Finance questions why a further expansion is being requested when nearly half the capacity is unused.
Q4. Why does adding hosts improve usable capacity by more than the raw space added?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Alert on the fullest OSD’s utilisation and the pool’s MAX AVAIL rather than on cluster average — the average constrains nothing. Compute headroom from its components (host absorption 1/N, imbalance, full ratio reserve, growth over procurement lead time) so the figure can be defended rather than asserted.

Cross-course references

Kubernetes: cluster autoscaler headroom for node failure follows the same 1/N logic
Linux: filesystem reserve exists to keep operations possible near full