The six-node reference topology

Advanced⏱ ~18 mincephcephadmcrushtool

What you'll learn

Specify a node from the OSD count outwards
Justify six hosts rather than three or four
Choose a failure domain that two racks can honour
Derive usable capacity that survives a host loss

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Node count and failure domain fix the availability envelope that every later decision inherits. Six nodes in two racks is the most common shape in the field, and it survives less than the people who build it assume.

The reference node

Component	Specification	Why this number
Chassis	2U, 12 x 3.5-inch bays	one host is 1/6 of the cluster, so a host loss is a 17% rebuild
Data	12 x 18 TB SAS HDD	216 TB raw per node
DB/WAL	2 x 1.6 TB NVMe, 6 OSDs each	`block.db` at 4% of an 18 TB device is roughly 720 GB
Boot	2 x 480 GB SATA SSD, mirrored	the OS never shares a device with an OSD
CPU	2 x 16 cores	about 1 core per HDD OSD, plus headroom for network, mon, mgr, cephadm
RAM	128 GiB	4-5 GiB per HDD OSD plus 60 GiB for the OS and co-located daemons
NIC	2 x 25 GbE, LACP	a node backfills at roughly 20 Gbit/s of drive throughput

# let cephadm size OSD memory from what the host actually has
ceph config set osd osd_memory_target_autotune true
ceph config set mgr mgr/cephadm/autotune_memory_target_ratio 0.7

# or pin it per device class
ceph config set osd/class:hdd osd_memory_target 4294967296
ceph config set osd/class:ssd osd_memory_target 8589934592

Where the numbers change: an all-NVMe node wants 2-4 cores and 8-12 GiB per OSD and 100 GbE, and drops to 8-10 OSDs per chassis because CPU, not the bay count, becomes the limit.

Why six hosts and not three

Hosts	Behaviour with size=3 and a host failure domain
3	placement works; a host loss cannot heal — there is nowhere for the third copy
4	one host loss heals; a second concurrent loss does not
5	a host loss heals while another host is in a maintenance window
6	the above, and 4+2 erasure coding can place on hosts
7+	4+2 erasure coding also self-heals a host loss

ceph orch host add node1 10.20.0.11 --labels=mon,osd
ceph osd crush rule create-replicated by-host default host
ceph osd pool set rbd-vm crush_rule by-host

Two racks and three replicas

A rule with `type rack` and size=3 needs three racks. Given two, CRUSH
returns two OSDs and the PGs stay undersized permanently. It does not
fall back to host.

Design	Rack loss	Host loss
size=3, failure domain host	half the OSDs gone; many PGs inactive	heals automatically
size=4, 2 racks x 2 hosts	degraded, still writable at min_size=2	heals automatically
size=3, failure domain rack	never places; do not build this on two racks

ceph osd getcrushmap -o /tmp/cm && crushtool -d /tmp/cm -o /tmp/cm.txt
# rule two-rack-four-copy {
#   type replicated
#   step take default
#   step choose firstn 2 type rack
#   step chooseleaf firstn 2 type host
#   step emit
# }
crushtool -c /tmp/cm.txt -o /tmp/cm.new
crushtool -i /tmp/cm.new --test --rule 2 --num-rep 4 --show-mappings | head
ceph osd setcrushmap -i /tmp/cm.new

Test the compiled map before installing it. --show-mappings prints the OSD set CRUSH would choose for each PG, and short sets are visible there before they are visible in ceph -s.

Monitors

Three monitors at this size; five once the cluster passes roughly fifteen hosts or spans more failure domains. Across two racks, three monitors put two in one rack.

ceph orch apply mon --placement="label:mon"
ceph quorum_status -f json | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("quorum:", d["quorum_names"], " of", [m["name"] for m in d["monmap"]["mons"]])'

Capacity that survives a host loss

Constraint	Calculation	Result
Raw	6 x 216 TB	1296 TB (1179 TiB)
Replicated at size=3	1296 / 3	432 TB
Target fill 70%	432 x 0.70	302 TB
One host down, at nearfull 0.85	(5 x 216 / 3) x 0.85	306 TB

Both constraints land near 300 TB, which is the number to publish. Take another 1-2% off for BlueStore metadata.

Why CRUSH returns too few OSDs instead of falling back to a smaller failure domain

A CRUSH rule is a deterministic descent through the bucket tree. A step of chooseleaf firstn 3 type rack performs three independent descents from the root, each required to land in a distinct rack bucket. With only two racks the third descent retries, collides with an already-chosen bucket every time, exhausts its retry budget and emits nothing. CRUSH has no fallback because the mapping must be a pure function of the map and the PG identifier — a fallback would make placement depend on cluster state and therefore change under you. The operational consequence is that the pool reports undersized PGs from the moment it is created and no amount of waiting or repairing fixes it; the only remedy is a map change, and a map change moves data.

Quiz

Knowledge check · 4 questions

Q1. A six-host cluster in two racks uses a CRUSH rule with rack as the failure domain and size=3. What happens?

Replicas are spread two in one rack and one in the other, and placement succeeds CRUSH places two replicas and Ceph reduces the pool size to 2 to match CRUSH returns only two OSDs per PG and the PGs stay undersized permanently The pool creation is rejected by the monitors before any PG is created
Q2. With two racks and size=3, a rack failure domain is safe because CRUSH falls back to host placement for the third replica.

True False
Q3. Size a six-node cluster and publish a usable capacity figure.

Six nodes, each with 12 x 18 TB HDDs, size=3 replication, host failure domain. The finance team wants a single number for the storage they can sell internally.
Q4. Why does the reference design use six hosts rather than four?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Publish the capacity figure that survives a host loss, not the one that assumes all six hosts are up. Test any CRUSH rule with crushtool --test --show-mappings before installing it — a rule that cannot satisfy its own replica count fails silently at map load and only becomes visible as undersized PGs afterwards.

Cross-course references

Kubernetes: a three-node control plane in two racks has the same quorum problem
Linux: quorum devices exist because an even split of votes has no correct winner