Monitor election and the leader lease

Why this matters in production

Elections are how monitors recover from disruption, and their frequency is a precise signal about cluster health that no dashboard summarises. A cluster electing constantly is a cluster in trouble even when ceph status says HEALTH_OK between elections.

How a leader is chosen

Each monitor has a rank, assigned by position in the monmap and ordered by address. Elections favour the lowest rank among those participating:

1. A monitor detects a problem (lease expiry, peer change, startup)
2. It proposes an election, incrementing the election epoch
3. Monitors respond; each defers to any lower-ranked participant
4. The lowest-ranked participant wins and becomes leader
5. The leader establishes quorum and grants leases

Because rank derives from address ordering, the leader is usually the same monitor. That is expected — leadership does not rotate for fairness.

ceph quorum_status --format json-pretty | jq '.quorum_leader_name, .quorum_names'
ceph mon stat

The lease

mon_lease                 5.0s   lease duration
mon_lease_renew_interval_factor  0.6   renew at 60% of lease
mon_lease_ack_timeout_factor     2.0   leader waits this multiple for acks
mon_election_timeout      5.0s   how long an election may take

The leader renews leases every three seconds by default. A peon that does not hear from the leader within five seconds declares the lease expired, stops serving reads, and calls an election.

These defaults assume a datacentre network. On a stretched cluster with meaningful inter-site latency they are worth reviewing — but only with measurements, since raising them delays recovery from genuine leader failure.

Election frequency as a signal

cephadm logs --name mon.ceph-01 --since 24h | grep -c 'calling new monitor election'

This count belongs in monitoring. It is cheap to collect and it detects a class of problem — intermittent network degradation between monitors — that nothing else surfaces until it becomes an outage.

Reading the outcome

After any election, confirm the result rather than assuming it:

ceph quorum_status --format json-pretty

Check that the expected monitors are present, that the leader is sensible, and that the election epoch has stopped incrementing.

Quiz

Production discipline

Add the election count to monitoring — a few per day is normal, dozens means an active fault, and nothing else in Ceph surfaces this class of problem before it becomes an outage. When a cluster is stuck electing, read logs and test the network and clocks rather than restarting monitors, since each restart adds an election and removes evidence. Treat MON_CLOCK_SKEW as a cause rather than a cosmetic warning, and verify time synchronisation on monitor hosts rather than assuming the platform provides it.

Cross-course references

• Ceph: Part LII (Time Synchronisation) for getting clocks right.
• Ceph: Part IX (Monitor Quorum) for the availability model.
• Ceph: Part CXVII (Lost Monitor Quorum) for the worst case.