Changing the monitor count

Why this matters in production

The monitor count determines failure tolerance, and changing it touches the component whose failure stops the cluster entirely.

When more monitors help

ceph mon dump
ceph quorum_status --format json | python3 -c '
import sys,json; d=json.load(sys.stdin)
print("quorum:", d["quorum_names"], "of", [m["name"] for m in d["monmap"]["mons"]])'

The most common mistake is adding monitors to improve availability when the actual problem is that all of them share a failure domain.

When they do not

5 monitors in one rack: tolerates 2 monitor failures, 0 rack failures
3 monitors in three racks: tolerates 1 monitor failure, 1 rack failure

The second configuration is more available despite having fewer monitors.

Adding a monitor

# always to an odd total
ceph orch apply mon --placement="ceph-01,ceph-02,ceph-03,ceph-04,ceph-05"

# verify each joins quorum before proceeding
watch -n 5 'ceph quorum_status --format json | python3 -c "
import sys,json; print(json.load(sys.stdin)[\"quorum_names\"])"'

Adding monitors one at a time, verifying quorum between each, means a problem affects one addition rather than several.

Removing a monitor

ceph orch daemon rm mon.ceph-05
ceph mon dump
ceph quorum_status

Removing takes the cluster through a period with an even count.
Verify quorum after each removal before the next.

# if a monitor is unreachable and must be removed from the map
ceph mon remove ceph-05

Verifying throughout

ceph -s
ceph mon stat
ceph quorum_status
ceph mon dump | grep -E 'epoch|mon\.'

After any change:
  the expected monitors are in the monmap
  all of them are in quorum
  the cluster is HEALTH_OK
  no elections are occurring

ceph log last 50 | grep -i election

Quiz

Production discipline

Change monitor placement before considering monitor count — five in one rack tolerate zero rack failures while three across three racks tolerate one. Verify quorum after each individual addition or removal rather than after the whole change, and never settle on an even count.

Cross-course references

• Kubernetes: etcd member placement across zones matters more than member count
• Linux: any quorum system depends on independent failure domains