Capacity planning for erasure-coded pools

Why this matters in production

A capacity plan that quotes raw capacity divided by the EC ratio is wrong by 30–40%, and the error surfaces when the cluster is nearly full — the worst possible moment. Working the reserves explicitly is what makes the number usable.

The full derivation

Start from raw and subtract each reserve in turn.

Step 1 — raw to EC-usable.

raw = 1,200 TB
profile 8+3 → ratio 1.375
EC-usable = 1200 / 1.375 = 873 TB

Step 2 — fill headroom. Ceph warns at nearfull (0.85) and stops writes at full (0.95), and OSDs are never perfectly balanced, so the fullest OSD hits the threshold before the average does.

target average fill: 0.75
usable after headroom = 873 × 0.75 = 655 TB

Step 3 — failure reserve. After a host fails, its data is redistributed across the survivors. That capacity must already exist.

14 hosts, lose 1 → survivors must absorb 1/14 ≈ 7.1%
usable after failure reserve = 655 × (1 − 0.071) = 608 TB

Step 4 — growth margin. Adding capacity takes procurement time. Leave enough that you trigger the purchase before the reserves are consumed.

6-month growth at observed rate: 90 TB
plan to fill to 608 − 90 = 518 TB before ordering

So 1.2 PB raw plans to roughly 520 TB of committed user data on 8+3 — not the 873 TB the ratio alone suggests.

Reading actual usage

ceph df
ceph df detail
ceph osd df                     # per-OSD, and the variance that matters
ceph osd pool autoscale-status  # RATE column confirms the ratio

ceph df reports STORED (user bytes) and USED (raw bytes) per pool. Their ratio is your real, measured overhead including allocation waste — compare it against the profile ratio to catch a small-object problem early.

The variance trap

ceph osd df | awk '{print $17}' | sort -n | tail -5

If the fullest OSD is 15 points above the average, your effective capacity is set by that OSD, not by the average. Run the balancer and re-measure before finalising the plan:

ceph balancer status
ceph balancer mode upmap
ceph balancer on

Quiz

Production discipline

Publish the capacity plan with every reserve shown as a separate line so reviewers can see and challenge each assumption. Track the measured USED/STORED ratio over time — a rising figure means object sizes are shrinking and the plan needs revisiting before the reserves absorb the difference. Alert on OSD utilisation variance alongside average utilisation.

Cross-course references

• Kubernetes: cluster capacity planning has the identical structure of nominal capacity minus reserves
• Linux: filesystem reserved blocks exist for exactly the same reason as the fill headroom