Inspecting monitor quorum

Advanced⏱ ~16 minceph

What you'll learn

Read quorum_status output
Identify monitors outside quorum
Diagnose from a single monitor when quorum is lost
Interpret election epochs

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

When quorum is uncertain, cluster-level commands may hang — so the ability to query a single monitor directly is what makes the situation diagnosable at all.

The cluster-level view

ceph quorum_status --format json | jq '{
  election_epoch,
  quorum_names,
  quorum_leader_name,
  monmap_epoch: .monmap.epoch
}'

{
  "election_epoch": 412,
  "quorum_names": ["ceph-mon-01", "ceph-mon-02", "ceph-mon-03"],
  "quorum_leader_name": "ceph-mon-01",
  "monmap_epoch": 7
}

Field	Meaning
`election_epoch`	increments on every election; rising rapidly means instability
`quorum_names`	monitors currently in quorum
`quorum_leader_name`	the current leader
`monmap.mons`	every monitor the cluster knows about

Comparing quorum_names against the full monitor list identifies which are outside.

ceph quorum_status --format json | \
  jq -r '(.monmap.mons[].name) as $all | $all' | sort > /tmp/all
ceph quorum_status --format json | jq -r '.quorum_names[]' | sort > /tmp/in
comm -23 /tmp/all /tmp/in

When quorum is lost

Cluster commands hang, so query a monitor’s admin socket directly:

ceph daemon mon.ceph-mon-01 mon_status

{
  "name": "ceph-mon-01",
  "rank": 0,
  "state": "electing",
  "election_epoch": 4187,
  "quorum": [],
  "outside_quorum": [],
  "monmap": { "epoch": 7, "mons": [...] }
}

`state`	Meaning
`leader`	in quorum, leading
`peon`	in quorum, following
`electing`	election in progress
`probing`	looking for peers
`synchronizing`	catching up its store from a peer

A monitor stuck in probing cannot reach its peers — a network or addressing problem. One stuck in electing with a rapidly rising epoch is the clock skew signature.

Run it on each monitor: what each believes, compared, is the diagnosis.

Election epoch

ceph quorum_status --format json | jq -r '.election_epoch'
sleep 30
ceph quorum_status --format json | jq -r '.election_epoch'

Stable is healthy. Rising by tens in thirty seconds means the monitors are electing continuously and the cluster is making no progress on state.

Query monitors individually when cluster commands hang

# Substitute the name of the monitor you are on:
MON_ID=ceph-mon-01

ceph daemon "mon.$MON_ID" mon_status

This reads the local monitor’s admin socket and does not require quorum, so it works when ceph -s does not. Running it on each monitor and comparing what each believes is how a quorum problem is diagnosed rather than guessed at.

Quiz

Knowledge check · 4 questions

Q1. `ceph -s` hangs and you suspect a quorum problem. What command still works?

ceph health detail, which the monitors answer from their own local store without a Paxos read ceph osd tree, since the CRUSH map is cached by the client and does not need the monitors `ceph daemon mon.<id> mon_status`, which reads the local admin socket and needs no quorum ceph pg stat, since PG statistics are served by the manager rather than through quorum
Q2. A monitor in the `probing` state is catching up its store from a peer.

True False
Q3. Diagnose a cluster where commands hang.

`ceph -s` hangs indefinitely. Five monitors are configured. SSH to each host works and the ceph-mon processes are running.
Q4. Why does a monitor that has been down for a long time rejoin slowly?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Use ceph daemon mon.<id> mon_status when cluster commands hang; it is the only view available without quorum and the state each monitor reports names the problem. Compare all monitors rather than one — the diagnosis is in the disagreement.

Cross-course references

Kubernetes: etcdctl endpoint status per member serves the identical diagnostic purpose
Linux: querying a cluster member directly when the cluster API is unavailable is general practice