The triggers that mean another host is due

Intermediate⏱ ~17 minceph

What you'll learn

Recognise the signals that precede a capacity number
Distinguish a drive purchase from a host purchase
Explain what each step in host count unlocks
Record the trigger alongside the decision

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The signal that a small cluster needs another host almost never arrives as a capacity alert. It arrives as work that keeps being postponed.

The triggers, roughly in the order they appear

Signal	What it means	Answer
A kernel update postponed because the cluster was not healthy	maintenance has no spare domain	a host
A drive replacement had to be expedited	nothing absorbed the failure	a host, or spare capacity
Recovery is still running the next morning	the degraded window exceeds tolerance	hosts
Client latency is visible during backfill	survivors carry recovery and clients	hosts
MAX AVAIL below one host’s worth of data	no room to absorb a failure	capacity, then hosts
A new pool refused on PG count	OSD count constrains pool count	OSDs

ceph -s
ceph osd df tree | head -20
ceph df detail
ceph osd pool autoscale-status

Deferred maintenance is the leading indicator

Capacity produces an alert. Risk produces a decision that quietly does not get made — the firmware update that slips a quarter, the reboot deferred because the cluster was backfilling, the drive left in a warning state because pulling it would leave no margin.

Count the maintenance items you have deferred in the last six months
because of cluster state. Two or more is the trigger, and it is the
one that shows up before any number does.

Drives or hosts

Binding constraint	Buy
Bytes, with five or more hosts	drives
Bytes, with three or four hosts	a host
Failure domains	a host
Recovery duration	hosts
IOPS on existing pools, hosts have spare CPU and network	drives
PG budget	OSDs, which usually means drives

What each step unlocks

Step	What becomes possible
3 to 4	a host loss can re-replicate without human intervention
4 to 5	five monitors become worth placing, and EC profiles with m=2 fit
5 to 6	a rack or row tier becomes describable in CRUSH
6 upward	recovery duration and fill ceiling improve on a slope

cat >> /secure/ceph-growth-log.txt <<'EOF'
2026-08-18  trigger:  kernel update deferred 3 weeks, cluster degraded
            state:    3 hosts, 9 OSDs, 58% mean use, fullest OSD VAR 1.16
            decision: order host 4, rack and add by 2026-10-15
EOF

Quiz

Knowledge check · 4 questions

Q1. What is usually the earliest reliable signal that a small cluster needs another host?

Maintenance work being repeatedly postponed because the cluster was not healthy The nearfull warning appearing against the fullest OSD, well ahead of the cluster mean A sustained rise in client IOPS beyond what the existing drives can serve The autoscaler raising pg_num on pools that are still growing steadily
Q2. On a small cluster the decision to add a host is usually driven by a risk posture rather than a capacity number.

True False
Q3. Justify a fourth host to someone looking at 58 per cent utilisation.

Three hosts, nine OSDs, 58 per cent mean use. Two kernel updates have been deferred this quarter, and a drive replacement last month had to be expedited because there was no margin to run without it.
Q4. What single change does the fourth host make that no later host repeats?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Track deferred maintenance as a capacity metric — two items postponed in six months because of cluster state is a stronger signal than any utilisation figure. Record the trigger alongside the decision, so the next person can tell whether the purchase was for bytes or for redundancy.

Cross-course references

Kubernetes: a cluster that cannot be drained node by node is already undersized
Linux: postponed patching is a capacity symptom as often as it is a process one