How many hosts a Ceph cluster needs

Intermediate⏱ ~17 minceph

What you'll learn

Derive the minimum host count from durability requirements
Explain why the practical minimum exceeds the theoretical one
Size a cluster for growth and self-healing
Evaluate a proposed host count against its requirements

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Host count is usually proposed from capacity — enough disks for the data — and that is the wrong starting point. The number is set by what the cluster must survive, and a capacity-derived count frequently cannot self-heal.

The floors

Requirement	Minimum hosts
Monitor quorum	3
size 3 with `failure_domain=host`	3
size 3 that can self-heal after a host loss	4
EC k+m with `failure_domain=host`	k+m
EC k+m that can self-heal	k+m+1
Comfortable production	5+

The self-healing rows are the ones people miss. With exactly size hosts, losing one leaves nowhere to place the missing replica, so the cluster stays degraded until the host returns rather than repairing itself.

Deriving from requirements

requirement: survive one host loss, self-heal, size 3
  → 3 for placement + 1 for recovery = 4 minimum
  → 5 or 6 for comfort and capacity headroom

requirement: survive one rack loss, size 3, failure_domain=rack
  → 3 racks for placement + 1 for recovery = 4 racks
  → hosts per rack from capacity

The failure domain and the replication factor give the count of domains; capacity gives the hosts per domain.

Capacity per host and the failure reserve

Concentrating capacity on fewer hosts means each host failure removes more of the cluster:

Hosts	Capacity lost per host failure	Reserve needed
4	25%	25%
10	10%	10%
20	5%	5%

The reserve must exist for the cluster to self-heal after a host loss. A four-host cluster therefore has to run at 75% of its nominal capacity at most — a cost that additional hosts reduce directly.

Daemon placement

ceph orch apply mon --placement="5 label:mon"
ceph orch apply mgr --placement="2 label:mgr"

Daemon	Count
MON	3 or 5, always odd
MGR	2
MDS	active ranks plus standbys
RGW	peak load plus one
OSD	one per device

Monitors and managers can share hosts with OSDs on smaller clusters and should have dedicated hosts on larger ones, since monitor latency affects everything.

Evaluating a proposal

Ask three questions:

How many failure domains does the replication scheme need?
Is there one more than that, so recovery has somewhere to go?
Does the capacity reserve cover the loss of one domain?

A proposal failing any of them is under-specified regardless of how much disk it contains.

Quiz

Knowledge check · 4 questions

Q1. A cluster has exactly three hosts and a size-3 pool with failure_domain=host. What happens when one host fails?

The pool serves with two copies and stays degraded — there is no third host to place the missing replica on Every PG in the pool goes inactive, because CRUSH will not serve an acting set that is smaller than the pool's size CRUSH relaxes the failure domain for the duration and places the third replica beside an existing one The pool is marked read-only until a fourth host joins, since writes below size cannot be acknowledged
Q2. A four-host cluster must run at a lower percentage of its nominal capacity than a twenty-host cluster to self-heal after a host loss.

True False
Q3. Evaluate a cluster proposal.

A proposal specifies four hosts with 24 × 20 TB drives each — 1.92 PB raw — for a size-3 pool, with a stated requirement to survive a host failure and continue self-healing. Capacity planning assumes 85% fill.
Q4. Why does the failure domain rather than capacity determine the minimum host count?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Derive host count from the failure domain and replication factor first, then add one for self-healing, and only then check capacity. State the usable capacity with the host-failure reserve already subtracted, since that reserve is what makes the self-healing requirement real.

Cross-course references

Kubernetes: node count driven by availability zones rather than by pod capacity follows the same logic
Linux: RAID array sizing with hot spares embodies the same reserve principle