The quorum model in operation

Intermediate⏱ ~15 minceph

What you'll learn

Read quorum_status and interpret every field
Distinguish configured tolerance from available tolerance
Monitor quorum health as a first-class signal
Decide whether a planned action is safe against current quorum

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Quorum is the difference between a cluster that works and one that does not, and it is the check most often skipped before maintenance. Reading it takes one command; skipping it has taken down clusters.

Reading quorum_status

ceph quorum_status --format json-pretty

{
  "election_epoch": 42,
  "quorum": [0, 1, 2],
  "quorum_names": ["ceph-01", "ceph-02", "ceph-03"],
  "quorum_leader_name": "ceph-01",
  "monmap": {
    "epoch": 3,
    "num_mons": 3,
    "mons": [ ... ]
  }
}

Field	Read it as
`quorum`	ranks currently in quorum
`quorum_names`	which monitors those are
`quorum_leader_name`	who is committing proposals
`election_epoch`	increments on every election — watch the rate
`monmap.num_mons`	how many exist, versus how many are in quorum

The comparison that matters is len(quorum) against num_mons. Equal means full health. Fewer means reduced tolerance, and how much less is the number to compute before doing anything.

Configured versus available tolerance

configured tolerance = num_mons - ceil((num_mons + 1) / 2)
available tolerance  = len(quorum) - ceil((num_mons + 1) / 2)

For five monitors: configured tolerance is 2. If one is already out of quorum, available tolerance is 1 — and a plan that assumes 2 is a plan that halts the cluster.

Make the quorum check a mandatory pre-step, not a habit

Every runbook that touches a monitor host — reboots, kernel upgrades, network changes, hardware work — should begin by reading quorum_status and computing available tolerance.

A short script makes it enforceable:

#!/bin/bash
q=$(ceph quorum_status --format json)
n=$(echo "$q" | jq '.monmap.num_mons')
i=$(echo "$q" | jq '.quorum | length')
need=$(( n / 2 + 1 ))
echo "monitors: $i/$n in quorum, need $need, spare $(( i - need ))"
[ $(( i - need )) -ge 1 ] || { echo "REFUSING: no spare tolerance"; exit 1; }

The value is not the arithmetic, which is trivial. It is that the check happens every time rather than when someone remembers.

Monitoring quorum

Three signals worth alerting on:

Monitors in quorum below num_mons. Warning immediately; this is reduced tolerance.
Monitors in quorum at exactly the majority. Critical — any further loss halts the cluster.
Election epoch rate. More than a few per day means an active fault.

The first two come straight from quorum_status. The third needs a counter over the logs, and it is the one that catches problems before they become outages.

Why a monitor can be up but not in quorum

ceph mon stat may show a monitor as present while quorum_status excludes it. The states are different and the distinction matters.

A monitor that is running but out of quorum is typically synchronising: after being down for a while, it must catch up on the map history it missed before it can participate. During synchronisation it is alive, consuming resources, and contributing nothing to tolerance.

Synchronisation time scales with how far behind it is and with monitor disk speed. On a cluster with a fast-moving osdmap and a slow monitor disk, a monitor that has been down for hours can take a long time to rejoin — which is another reason monitor stores belong on NVMe.

The operational point is that “the monitor is running again” is not the same as “tolerance is restored”. Confirm with quorum_status before proceeding with anything else.

Deciding whether an action is safe

Before any action affecting a monitor:

Read quorum_status.
Compute available tolerance.
If it is zero, do not proceed — restore a monitor first.
If it is one and the action affects one monitor, proceed but do not overlap with anything else.
Confirm quorum is restored before the next step.

That sequence, applied literally, prevents the most common way clusters lose quorum: two maintenance actions overlapping because nobody checked what was already absent.

Quiz

Knowledge check · 4 questions

Q1. A five-monitor cluster has four monitors in quorum. What is the available tolerance?

Two, since five monitors are configured to tolerate two failures Zero, because any monitor being absent removes all tolerance One, because quorum needs three and only four are currently in Three, because four monitors in quorum exceed the majority of three
Q2. A monitor can be up and consuming resources while contributing nothing at all to the cluster's fault tolerance.

True False
Q3. Design a pre-maintenance gate for any work touching a monitor host, and justify each element.

Team of six operators running a 300-OSD cluster with five monitors across three racks. Monitor-affecting work happens roughly weekly: kernel patching, hardware replacement, network changes. Two near-misses in the past year involved overlapping work while a monitor was already down. Nobody currently checks quorum before starting.
Q4. Give the formula for available tolerance and explain why it differs from configured tolerance.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Make reading quorum_status and computing available tolerance an executable gate at the start of every monitor-affecting runbook, not a remembered habit — the two most common quorum losses come from planning against configured tolerance and from overlapping work. Confirm quorum is fully restored between steps rather than treating a running process as restored tolerance. And alert on quorum below num_mons, on quorum at exactly the majority, and on election rate.

Cross-course references

Ceph: Part VIII (Monitors) for what quorum protects.
Ceph: Part XCV (Maintenance Flags) for the rest of the pre-maintenance checks.
Ceph: Part CXVII (Lost Monitor Quorum) for when the gate was not there.