Growth composition and the real lead time

Intermediate⏱ ~18 mincephrbd

What you'll learn

Classify growth as drift, step, or structural
Count the full lead-time chain including backfill
Attribute growth to the pool and the cause
Plan against the reserve crossing rather than exhaustion

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The forecast is not the hard part. The hard part is that the order has to be placed months before the number it is based on becomes uncomfortable.

Three kinds of growth, planned differently

Kind	Example	How it is planned
Drift	daily writes from an established workload	extrapolate the measured rate
Step	a NAS migration, a new tenant, a new cluster consumer	add the known volume on its known date
Structural	raising size from 2 to 3, enabling RGW versioning, retaining snapshots	multiplies data already stored

Structural growth is the one that surprises people, because it applies to data that is already there and arrives as fast as the cluster can move it.

The lead-time chain

Stage	Typical
Decision and internal approval	5-20 days
Purchase order and vendor lead time	15-60 days
Delivery, racking, cabling	2-5 days
Burn-in and firmware baseline	2-3 days
Adding the OSDs	under a day
Backfill to a balanced state	2-10 days
Total	30-100 days

The chain ends when the capacity is usable, not when the pallet arrives.

Which pool is growing, and why

ceph df detail
rbd du -p rbd | tail -5
ceph fs status

# snapshots inflate stored bytes with no new client writes
rbd snap ls --all rbd/vm-101-disk-0
ceph osd pool get rbd size

A pool that grows while its application reports steady data is usually retaining something rather than receiving something.

Symptom	Usual cause
RBD pool grows, guests report no change	snapshots retained by a backup job
RGW data pool grows, object count flat	versioning on with no expiry rule
CephFS pool grows after a deletion	snapshots pinning the deleted files
Every pool grows at once	a `size` change, applied to existing data

Attribute the growth before extrapolating it. A retention fault produces a straight line that stops the day someone enables the expiry rule, and planning hardware against it buys capacity for a bug.

The date that matters is the reserve crossing

Exhaustion date   the point at which MAX AVAIL reaches zero
Reserve crossing  the point at which the failure reserve starts being spent

The reserve crossing comes first and it is the real deadline: past it, a host failure can no longer be absorbed even though the cluster still accepts writes and reports HEALTH_OK. Order against the reserve crossing minus the whole chain, not against exhaustion.

Quiz

Knowledge check · 4 questions

Q1. Which stage of the capacity lead-time chain gets longer the later the order is placed?

Vendor lead time, because an order placed later sits further back in the allocation queue behind everyone who ordered ahead of you Racking and cabling, because a fuller rack leaves less room and the work has to be scheduled around live equipment Burn-in, because drives from a later batch have to be tested against the firmware baseline the cluster already runs Backfill to a balanced state, because a fuller cluster has less room to move into and throttles recovery against client I/O
Q2. Procurement lead time is the interval between placing the order and the hardware arriving.

True False
Q3. Set an order date for a cluster approaching its reserve.

A six-host cluster has 28 TiB of growth allowance and grows 180 GiB per day. A tenant migration of 9 TiB is committed for six weeks time. Vendor lead time is currently 45 days.
Q4. Why is the reserve crossing the planning deadline rather than the exhaustion date?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Keep committed step changes as a dated list beside the capacity plan and add them to the projection explicitly — a trend fitted across a step change is most confident when it is most wrong. Count the whole chain from decision to completed backfill when working back to an order date.

Cross-course references

Kubernetes: cluster autoscaler lead time includes image pull and warm-up, not just node boot
Linux: procurement lead time is the smaller half of the interval that actually matters