CephLXIII · Capacity ManagementCapacity Management
Raw capacity and usable capacity
What you'll learn
- Compute usable capacity from raw capacity
- Identify every factor that reduces it
- Read the capacity figures Ceph reports
- Communicate capacity honestly to stakeholders
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
A cluster bought as “600 TB” stores far less, and the gap is not overhead in the vague sense — it is three specific multipliers that can each be computed.
The three reductions
raw capacity
× durability policy (1/3 for size=3, k/(k+m) for EC)
× full ratio (0.95 by default — you cannot use the last 5%)
× usable headroom (what you choose to leave free)
= capacity you can actually plan againstFor a 600 TB raw cluster with size=3 and a 25% operational headroom:
600 TB × (1/3) × 0.95 × 0.75 ≈ 142 TBThe invoice said 600. The plannable figure is 142.
What Ceph reports
ceph df--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 600 TiB 418 TiB 182 TiB 182 TiB 30.33
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
rbd-vms 3 1024 58 TiB 15.2M 174 TiB 29.4 139 TiB| Column | Meaning |
|---|---|
SIZE | raw capacity of the device class |
RAW USED | raw bytes consumed including replication |
STORED | logical bytes the clients wrote |
USED | raw bytes those logical bytes consume |
MAX AVAIL | logical bytes this pool can still accept |
MAX AVAIL is the number that answers “how much more can we store” — it
already accounts for the pool’s durability policy and the full ratio.
Why MAX AVAIL is smaller than you expect
ceph osd df | sort -k17 -rn | head -3MAX AVAIL is computed from the fullest OSD, not the average. A cluster
at 60% overall with one OSD at 88% reports a MAX AVAIL reflecting that
OSD, because writes are distributed by CRUSH and the fullest OSD reaches
the full ratio first.
This is the single most common source of “the cluster says it is full but
ceph df shows plenty of space”.
Communicating honestly
| Statement | Accurate? |
|---|---|
| “We have 600 TB” | raw only; misleading |
| “We have 418 TB available” | raw available; still misleading |
| “We can store 139 TB more in this pool” | correct — MAX AVAIL |
| “We should plan for 100 TB more before expanding” | correct with headroom |
Quiz
Knowledge check · 4 questions
Q1. Why is `MAX AVAIL` computed from the fullest OSD rather than the average?
Q2. On a 600 TB raw cluster with size=3, roughly 200 TB is plannable capacity.
Q3. Answer a capacity question from management.
Management asks how much of the new 600 TB cluster is still free. `ceph df` shows 418 TiB available raw. All pools are size=3 and the cluster is at 30% raw utilisation.
Q4. Why does running the balancer increase MAX AVAIL?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Quote MAX AVAIL rather than raw available when reporting capacity;
it applies the durability policy and the full ratio and derives from the
fullest OSD, which is the constraint that binds. Run the balancer before
concluding a cluster needs expansion — flattening the distribution raises
the same figure.
Cross-course references
- Kubernetes: allocatable node capacity differs from capacity for the same reason
- Linux: filesystem reserved blocks make df available smaller than the device size