Monitoring the monitors

Why this matters in production

Without quorum the cluster does nothing — no I/O, no recovery, no administration. It is the one component whose failure is total, and the warning signs are visible well in advance.

The metrics

ceph_mon_quorum_status              # 1 if this monitor is in quorum
ceph_mon_num_sessions
ceph_mon_metadata                   # labels carry version, hostname
ceph_mon_election_call              # elections called
ceph_mon_num_elections
ceph_health_detail{name="MON_DOWN"}
ceph_health_detail{name="MON_CLOCK_SKEW"}
ceph_health_detail{name="MON_DISK_LOW"}
ceph_health_detail{name="MON_DISK_CRIT"}

# how many monitors are in quorum
count(ceph_mon_quorum_status == 1)

# how many exist
count(ceph_mon_quorum_status)

Alerting before quorum is lost

The important alert is not “quorum lost” — by then the cluster is down. It is “one monitor away from losing quorum”:

- alert: CephMonQuorumAtRisk
  expr: |
    count(ceph_mon_quorum_status == 1)
      <= floor(count(ceph_mon_quorum_status) / 2) + 1
  for: 5m
  labels: { severity: critical }
  annotations:
    summary: "Quorum has no margin — one more monitor loss stops the cluster"

- alert: CephMonDown
  expr: ceph_mon_quorum_status == 0
  for: 5m
  labels: { severity: page }
  annotations:
    summary: "Monitor {{ $labels.ceph_daemon }} is out of quorum"

With five monitors, quorum needs three. At four in quorum there is one spare; at three there is none — and that is the state worth paging on.

Clock skew

ceph time-sync-status
ceph config get mon mon_clock_drift_allowed

- alert: CephMonClockSkew
  expr: ceph_health_detail{name="MON_CLOCK_SKEW"} == 1
  for: 10m
  labels: { severity: ticket }

Clock skew between monitors breaks the leases that Paxos depends on. It degrades into election churn before it causes an outright failure, which makes it a genuine early warning.

The monitor store

ceph daemon mon.$(hostname -s) mon_status | python3 -c '
import sys,json; d=json.load(sys.stdin)
print("state:", d["state"], "quorum:", d["quorum"])'

du -sh /var/lib/ceph/*/mon.*/store.db

- alert: CephMonDiskLow
  expr: ceph_health_detail{name="MON_DISK_LOW"} == 1
  for: 15m
  labels: { severity: ticket }

A monitor store that grows without bound — usually because the cluster has been unhealthy for a long time and the store cannot trim — eventually fills the filesystem and takes the monitor down.

Election churn

rate(ceph_mon_num_elections[15m]) > 0

A healthy cluster holds essentially no elections. Repeated elections indicate network instability, clock skew, or an overloaded monitor, and each one briefly pauses cluster operations.

Quiz

Production discipline

Alert when the number of monitors in quorum equals the minimum required, not when quorum is lost — the first is a warning and the second is an outage notification. Treat MON_CLOCK_SKEW as a genuine early warning: it produces election churn that appears to clients as unexplained latency spikes.

Cross-course references

• Kubernetes: etcd quorum monitoring follows exactly this margin-based approach
• Linux: any consensus system needs alerting before the margin is exhausted