Paxos at the depth a Ceph admin needs

Advanced⏱ ~15 minceph

What you'll learn

Explain what consensus guarantees and why it requires a majority
Describe the leader and lease model Ceph monitors use
Connect Paxos behaviour to observable monitor symptoms
Recognise when monitor issues are consensus problems versus resource problems

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

You will not implement Paxos. You will, however, read monitor logs full of paxos, lease, and election messages during incidents, and knowing what they mean turns an opaque log into a diagnosis.

What consensus guarantees

Consensus means: all participants agree on the same sequence of values, and once a value is committed it is never changed.

For Ceph that means every monitor agrees on the same sequence of map epochs. Epoch 41,207 is the same map on every monitor, forever. There is no possibility of two monitors having different content for the same epoch.

The price is that committing requires a majority, so a minority can make no progress at all.

The leader and lease model

Ceph’s monitors elect a leader, and the leader holds a lease:

election      monitors agree on a leader (lowest rank wins ties)
lease grant   leader grants a lease to each peon
lease renew   renewed every mon_lease_renew_interval
lease expiry  mon_lease (5s default) after last renewal

While holding a valid lease, a peon may serve reads from its local map without consulting the leader — that is what makes map serving cheap. When a lease expires, the peon stops serving and a new election begins.

stateDiagram-v2
  [*] --> Electing
  Electing --> Leader: won
  Electing --> Peon: lost
  Leader --> Electing: lease not renewed by majority
  Peon --> Electing: lease expired

Reading monitor logs

cephadm logs --name mon.ceph-01 | grep -E 'paxos|lease|election'

Useful lines and what they mean:

Log fragment	Meaning
`calling new monitor election`	this monitor lost its lease or saw a peer change
`win_election` / `lose_election`	outcome, with the resulting quorum set
`lease_timeout`	did not hear from the leader in time
`paxos.*: is_readable ... false`	cannot serve reads; no valid lease
`slow request` on a monitor	commit taking longer than expected

What Paxos costs

Every map change is a round trip to a majority plus a durable write on each. So:

Commit latency is at least one network round trip plus the slowest majority disk commit.
More monitors means more participants per commit and slower commits — one of the reasons five is not automatically better than three.
A slow monitor in the majority slows every commit, even though the cluster has quorum.

Why a slow monitor is harder to spot than a dead one

A dead monitor drops out of quorum, raises a health warning, and the remaining majority proceeds at full speed.

A slow monitor stays in quorum and participates in every commit. If it is one of the two needed for a majority of three, every proposal waits for it. The cluster reports quorum with all three monitors — entirely healthy by the summary — while map changes take an order of magnitude longer than they should.

The consequences ripple outward: failure detection lags, recovery starts late, client reconnections are slow. None of these point at a monitor.

The check that finds it is per-monitor commit latency:

for m in a b c; do
  echo "mon.$m"; ceph daemon mon.$m perf dump | grep -A3 'commit_latency'
done

Comparing monitors against each other is what makes the outlier obvious, which is the same principle as ceph osd perf.

The practical summary

Consensus explains three observable things: why a minority stops, why map changes have a floor on latency, and why monitor disk speed matters more than monitor disk size. Everything else about Paxos is implementation detail an operator can safely leave to the code.

Quiz

Knowledge check · 4 questions

Q1. Monitors are repeatedly calling elections. CPU is idle on all three and RocksDB commit latency is low. What is the likely cause?

A network problem causing lease renewals to miss their deadline A corrupted monitor store requiring a rebuild Too many monitors, making elections take longer than the lease interval Clock skew between the monitors exceeding the allowed drift
Q2. A monitor that is slow but still in quorum can significantly delay cluster-wide map changes while health reports normal.

True False
Q3. Map changes on a five-monitor cluster take several seconds each. All five are in quorum and health is OK. Find the cause.

Five monitors across two rooms, three in room A and two in room B. Cluster is 300 OSDs. Recently the team added the fourth and fifth monitors to improve resilience. Since then, ceph status is slower, OSD down events take longer to register, and recovery start is delayed. All five monitors report in quorum with no warnings.
Q4. Explain the leader and lease model and what happens when a lease expires.

Passing score: 75%. Answers are checked in this browser.

Production discipline

When monitors misbehave, separate consensus problems from resource problems before acting: idle monitors with repeated elections means the network, busy monitors with high commit latency means the disk, and restarting monitors helps neither while generating more elections. Compare per-monitor RocksDB commit latency across monitors, since a slow monitor inside quorum degrades every map change while health reports normal. And treat monitor count as a cost as well as a benefit — every added member is another participant in every commit.

Cross-course references

Ceph: Part IX (Monitor Quorum) for the availability consequences.
Ceph: Part V (Distributed Systems Foundations) for quorum reasoning.
Ceph: Part LII (Time Synchronisation) for clock skew, the other election cause.