Monitor elections and clock skew

Why this matters in production

A cluster with skewed monitors elects repeatedly, and during each election it accepts no map updates. Enough of that and the cluster is effectively unavailable while every daemon reports itself healthy — a genuinely confusing state.

The mechanism

The leader grants peers time-limited leases and renews them before expiry. A peer whose clock runs fast reaches the expiry earlier than the leader intended, concludes the leader is gone, and calls an election.

ceph -s
# health: HEALTH_WARN
#         clock skew detected on mon.ceph-mon-02

ceph health detail
# [WRN] MON_CLOCK_SKEW: clock skew detected on mon.ceph-mon-02
#     mon.ceph-mon-02 clock skew 0.184s > max 0.05s

The reported skew and the maximum are both in the message.

Election behaviour

ceph -w | grep -i election
ceph mon stat
journalctl -u ceph-mon@ceph-mon-02 | grep -iE 'election|lease|skew'

Frequent elections with no daemon failures and no network problems is the skew signature. During each election the monitors accept no updates, so OSD state changes queue and clients cannot obtain new maps.

Recovering

# 1. identify the offending monitor
ceph health detail | grep -A2 MON_CLOCK_SKEW

# 2. compare clocks directly
for m in ceph-mon-01 ceph-mon-02 ceph-mon-03; do
  printf '%-16s %s\n' "$m" "$(ssh "$m" date -u +%s.%N)"
done

# 3. fix the time source on the affected host
ssh ceph-mon-02 'chronyc sources -v; chronyc tracking'
ssh ceph-mon-02 'systemctl restart chronyd; chronyc makestep'

# 4. confirm convergence
ceph time-sync-status
ceph -s

chronyc makestep forces an immediate step rather than a slow rate correction, which is what you want during an incident.

If a monitor’s clock is badly wrong and cannot be corrected quickly, removing it from quorum is better than leaving it destabilising the cluster:

ceph mon rm ceph-mon-02
# fix the host, then
ceph orch daemon add mon ceph-mon-02

The tolerance setting

ceph config get mon mon_clock_drift_allowed        # 0.05
ceph config get mon mon_clock_drift_warn_backoff   # 5

Raising the tolerance is occasionally proposed as a workaround. It masks the warning without addressing the consensus risk, and the lease mechanism still depends on the clocks agreeing — so the elections continue while the warning that explained them disappears.

Quiz

Production discipline

Treat clock skew warnings on monitors as critical rather than as warnings; the consensus risk they describe stalls cluster state management. Remove a monitor that cannot be corrected promptly rather than leaving it destabilising quorum — removal is reversible and the destabilisation is not free.

Cross-course references

• Kubernetes: etcd leader elections triggered by clock issues have the same stalling effect
• Linux: any lease-based cluster manager shares this dependency