CephL · Cluster DeploymentCluster Deployment
Placing monitors
What you'll learn
- Choose a monitor count and justify it
- Distribute monitors across failure domains
- Size monitor hosts appropriately
- Evaluate a monitor placement for quorum survival
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
Losing monitor quorum stops the cluster — no map updates, no authentication, no new client connections. Monitor placement is therefore the most consequential availability decision in a cluster, and it is frequently made by whichever hosts were convenient.
Count and quorum
| Monitors | Quorum | Failures tolerated |
|---|---|---|
| 1 | 1 | 0 |
| 3 | 2 | 1 |
| 5 | 3 | 2 |
| 7 | 4 | 3 |
Always odd. Four monitors need three for quorum, tolerating one failure — the same as three, at more cost and more Paxos overhead.
Three for most clusters. Five for large clusters, multi-rack deployments, or where two simultaneous monitor failures are plausible. Seven is rare and adds latency to every consensus operation.
Distributing across failure domains
ceph orch host label add ceph-mon-01 mon
ceph orch host label add ceph-mon-02 mon
ceph orch host label add ceph-mon-03 mon
ceph orch apply mon --placement="3 label:mon"
ceph mon dump
ceph quorum_status | jq -r '.quorum_names'The placement rule: no failure domain may hold enough monitors to break quorum by failing.
| Monitors | Racks | Per rack | Rack loss survivable? |
|---|---|---|---|
| 3 | 1 | 3 | no |
| 3 | 2 | 2 + 1 | no — losing the 2-rack breaks quorum |
| 3 | 3 | 1 each | yes |
| 5 | 3 | 2 + 2 + 1 | yes |
| 5 | 2 | 3 + 2 | no |
Three monitors across two racks is the arrangement that looks distributed and is not.
Sizing monitor hosts
| Resource | Requirement |
|---|---|
| Disk | fast — the monitor store is latency-critical; SSD or NVMe |
| Capacity | tens of GB, more on large clusters |
| Memory | a few GB, more with many OSDs |
| CPU | modest |
| Network | low latency matters more than bandwidth |
Monitor store latency affects every map update and every authentication. Placing the store on spinning disk is a common and damaging economy.
FSID=$(ceph fsid)
ceph config get mon mon_data
df -h "/var/lib/ceph/$FSID"/mon.*/Co-location
On smaller clusters monitors share hosts with OSDs, which is supported and works. The risk is resource contention: an OSD host under heavy recovery can starve the monitor, and monitor slowness affects everything.
On larger clusters, dedicated monitor hosts are worth the machines.
Quiz
Knowledge check · 4 questions
Q1. Three monitors are placed across two racks, two in one and one in the other. What happens if the rack with two fails?
Q2. Four monitors tolerate more failures than three.
Q3. Review a monitor placement for a multi-rack cluster.
A cluster spans four racks. Five monitors are proposed, placed three in the primary rack and two in a second, with the other two racks holding only OSDs. The rationale given is that the primary rack has the best hardware.
Q4. Why does monitor store latency matter more than its capacity?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Check monitor placement against the rule that no failure domain may hold enough monitors to break quorum by failing — the three-across-two-racks arrangement looks distributed and provides nothing. Put the monitor store on flash regardless of how small it is; the latency is felt in every cluster event.
Cross-course references
- Kubernetes: etcd member distribution across zones follows the identical quorum arithmetic
- Linux: any Paxos or Raft cluster has the same placement constraint