Designing a dedicated storage cluster

Advanced⏱ ~18 mincephcephadm

What you'll learn

Design host roles for a dedicated cluster
Place monitors, managers, and OSDs appropriately
Identify the decisions that are difficult to reverse
Size the cluster for its intended workload

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Several architectural decisions — failure domain, device class layout, daemon placement — are set at deployment and expensive to revisit. Getting them right is cheaper than correcting them.

Host roles

Role	Runs	Count
Monitor	MON, MGR	3 or 5, odd
Storage	OSD	as many as capacity requires
Gateway	RGW, MDS, NFS	as the services require
Admin	client tooling, `_admin` label	1–2

ceph orch host add ceph-mon-01 10.0.2.11 --labels mon,mgr,_admin
ceph orch host add ceph-osd-01 10.0.2.21 --labels osd
ceph orch apply mon --placement="label:mon"
ceph orch apply mgr --placement="label:mgr"
ceph orch apply osd --all-available-devices --placement="label:osd"

Label-based placement means adding a host with the right label deploys the right daemons, which is what makes growth mechanical.

Monitor placement

3 monitors: tolerates 1 failure
5 monitors: tolerates 2 failures

Consideration	Guidance
Count	3 for most clusters, 5 above ~50 OSDs or across racks
Placement	one per failure domain where possible
Co-location with OSDs	acceptable on small clusters, avoid at scale
Storage	fast local device; the store is write-heavy
Never	an even number

ceph orch apply mon --placement="ceph-mon-01,ceph-mon-02,ceph-mon-03"
ceph mon dump

Monitors in the same rack means a rack failure loses quorum regardless of how many there are.

Device class layout

ceph osd crush class ls
ceph osd crush rule create-replicated nvme-rule default host nvme
ceph osd crush rule create-replicated hdd-rule default host hdd

Pool	Class	Reason
VM disks	NVMe or SSD	latency
CephFS metadata	NVMe	every operation touches it
RGW bucket index	NVMe	same
Bulk data, backups	HDD	capacity

Deciding this at deployment means the hardware is bought correctly; deciding later means moving data.

The decisions that are hard to reverse

Decision	Why it is hard to change
Failure domain	changing it moves most of the pool
EC profile	fixed at pool creation; needs a new pool and migration
Monitor count and placement	changeable, but disruptive
Network topology	physical
Device class mix	requires buying different hardware
`size` on a large pool	a full copy of data movement

# verify the failure domain can be satisfied before committing
ceph osd tree | grep -cE '^-.*rack'

Sizing

capacity:    required usable × durability multiplier / (full ratio × (1 - headroom))
IOPS:        required IOPS × size / per-device IOPS
throughput:  required throughput, with the cluster network sized above it
hosts:       at least size + 1 for the failure domain, more for absorption efficiency

Quiz

Knowledge check · 4 questions

Q1. Three monitors are placed in one rack. What failure tolerance does the cluster have?

Two monitor failures, since the five-monitor rule of thumb applies once monitors share a rack One rack failure, because the surviving monitors rebuild quorum from the monmap when the rack returns One monitor failure and zero rack failures — a rack event takes all three and loses quorum The same as three monitors spread across racks, since quorum counts daemons rather than locations
Q2. Moving a pool to a different erasure coding profile means creating a second pool and migrating every object into it.

True False
Q3. Design a dedicated cluster for a Proxmox deployment.

A new dedicated Ceph cluster will serve a Proxmox environment. It will hold VM disks, CephFS for ISOs and backups, and will span three racks.
Q4. Why is label-based daemon placement preferable to explicit host lists?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Place monitors one per failure domain rather than counting them — three in one rack tolerate zero rack failures regardless of the number. Decide the device class layout before buying hardware; correcting it later means moving data and possibly buying different drives.

Cross-course references

Kubernetes: control plane node placement across zones follows the same reasoning
Linux: any quorum system depends on independent failure domains, not member count