Three hosts: the utilisation ceiling and the repair window

Intermediate⏱ ~18 minceph

What you'll learn

Distinguish OSD failure from host failure on three hosts
Compute the per-host utilisation ceiling
Plan maintenance that runs at reduced redundancy
State the accepted risks explicitly

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Three hosts is a legitimate production choice, and it changes what the word redundant means in a way that has to be written down.

Two different failures

Failure	Three hosts, size 3, host failure domain
One OSD fails	recovers automatically, within the same host
One host fails	PGs go degraded and stay degraded until it returns
One host plus one OSD elsewhere	PGs below `min_size`; I/O stops for those PGs
Two hosts fail	monitors lose quorum; the cluster stops

ceph -s
ceph osd pool get vms size min_size
ceph osd tree

CRUSH must place one copy per host. With exactly three hosts there is no
fourth candidate, so a host loss cannot be repaired by backfill — only by
repairing the host. The mean time to repair becomes hardware logistics.

The utilisation ceiling

An OSD failure is repaired inside its own host, so the surviving OSDs of
that host must have room for the failed one.

ceph osd df tree
ceph df detail

OSDs per host	Room needed for one failure	Practical per-host ceiling
4	each survivor absorbs a third	about 67%
6	each survivor absorbs a fifth	about 75%
8	each survivor absorbs a seventh	about 78%
12	each survivor absorbs an eleventh	about 82%

# the ratios the ceiling is derived from
ceph config get osd osd_backfillfull_ratio
ceph config get osd osd_failsafe_full_ratio
ceph osd dump | grep -E 'full_ratio'

Cluster-wide utilisation is the wrong number to watch here. One host
running hot while the others are empty still cannot absorb an OSD loss.

Maintenance

ceph osd set noout
ceph orch upgrade start --ceph-version 20.2.1
ceph orch upgrade status

During a host reboot	State
Copies available	2 of 3
`min_size` 2 satisfied	yes, with no margin
A second fault during the window	I/O stops for the affected PGs
Duration	the full reboot plus recovery of writes made meanwhile

Every maintenance window on three hosts is a window with no redundancy
margin. Shorten it and schedule it; do not lower `min_size` to widen it.

What to write down

Accepted: a host failure leaves the cluster degraded until the host is
repaired, with no automatic recovery. Mitigations: spare hardware on
site, per-host utilisation capped, maintenance windows scheduled and
short, min_size held at 2.

Why erasure coding does not help on three hosts

An erasure-coded pool with a host failure domain needs k plus m distinct hosts, so three hosts permits only k=2 m=1. That gives 1.5 times overhead against 3 times for replication, which looks like a saving until the failure tolerance is compared: 2+1 survives exactly one shard loss, the same as size 3 replication, and with only three hosts the lost shard cannot be regenerated anywhere any more than a replica can. What changes is the cost of running degraded — every read of a degraded EC placement group reconstructs the missing shard from the survivors, consuming CPU and adding latency for as long as the host is down, whereas a degraded replicated PG serves reads directly from a surviving copy. On three hosts, replication is the correct choice and erasure coding is a false economy.

Quiz

Knowledge check · 4 questions

Q1. A three-host cluster averages 40% utilisation but one host is at 85%. What happens when an OSD in that host fails?

The placement groups stay degraded, because the replacement copy can only be placed on the remaining OSDs of that same host The data rebalances across all three hosts, since CRUSH recomputes placement for every placement group the OSD held The cluster stops accepting writes, because a size 3 pool cannot serve any placement group while a copy is missing from its acting set The monitors mark the whole host out and recovery completes normally onto the two hosts that remain available
Q2. Erasure coding on three hosts reduces cost without reducing failure tolerance.

True False
Q3. Prepare a three-host cluster for a firmware upgrade.

Three hosts, eight OSDs each, size 3 and min_size 2. Per-host utilisation is 61%, 64%, and 79%. Each host needs a reboot for firmware.
Q4. Why is cluster-wide utilisation the wrong figure to watch on a three-host cluster?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Track per-host utilisation with ceph osd df tree and treat the busiest host as the cluster figure — an OSD failure is repaired inside its own host, so the cluster average is not the constraint. Write down that a host failure leaves the cluster degraded until the hardware is repaired, and keep spare hardware on site because that is the recovery mechanism.

Cross-course references

Kubernetes: a three-node control plane tolerates one failure and cannot reschedule its own quorum
Linux: an average across devices hides the one that is about to fill