Losing one monitor — a stable state, not an emergency

Why this matters in production

This is the most common monitor incident and the one most often mishandled — usually by doing more than the situation requires.

What is actually happening

With three monitors and one lost:

• Quorum is two, and two remain. Everything works.
• Map commits proceed normally, possibly a little faster.
• Clients are unaffected and do not need restarting.
• ceph status reports HEALTH_WARN with MON_DOWN.
• Available tolerance is now zero.

That last point is the entire content of the incident. Nothing is broken; the safety margin is gone.

The correct sequence

1. Confirm the state. ceph quorum_status — two in quorum, one absent.
2. Determine why. Host down, daemon crashed, disk full, network.
3. Restore rather than replace, if possible. A monitor whose host simply rebooted rejoins by itself and syncs.
4. If the host is gone, add a replacement on a new host before removing the old entry, where the environment allows.
5. Confirm quorum returns to three.

ceph quorum_status
cephadm logs --name mon.ceph-02
ceph orch daemon restart mon.ceph-02
ceph orch apply mon --placement="ceph-01 ceph-03 ceph-04"
ceph quorum_status

What else to avoid

• Do not defer unrelated maintenance indefinitely, but do not perform monitor-affecting maintenance at all until tolerance is restored. Available tolerance is zero.
• Do not restart the surviving monitors. There is no scenario in which that helps, and each restart risks the quorum you still have.
• Do not lower mon_lease or other timing settings in response. Nothing here is a timing problem.

Replacing a monitor

# add the replacement first
ceph orch host add ceph-04 10.0.1.14
ceph orch apply mon --placement="ceph-01 ceph-03 ceph-04"
ceph quorum_status                      # confirm 3 in quorum

# then remove the dead one
ceph mon rm ceph-02
ceph orch host rm ceph-02 --force
ceph quorum_status

Adding first means the cluster never drops below three configured monitors with two in quorum, so tolerance is restored before anything is removed.

When it stays down

If a monitor will not rejoin, the fastest reliable path is often to destroy and recreate it rather than to debug its store:

ceph orch daemon rm mon.ceph-02
ceph mon rm ceph-02
ceph orch apply mon --placement="ceph-01 ceph-03 ceph-02"

The new monitor syncs from the survivors. Since monitors hold no unique data — every one is a replica of the same state — recreating one costs nothing but sync time.

Quiz

Production discipline

Treat a single monitor loss as a state with zero remaining tolerance rather than as an emergency: everything works, and nothing monitor-affecting may overlap until it is fixed. Restore before you remove — add a replacement, confirm it reaches quorum, and only then take the dead entry out, so the cluster never passes through a two-monitor state. And recreate rather than debug a monitor that will not rejoin, since monitors hold no unique data and syncing from the survivors is the cheaper path.

Cross-course references

• Ceph: Part IX lesson on losing two monitors for the next step down.
• Ceph: Part CX (Monitor Recovery) for deeper repair procedures.
• Ceph: Part VIII (Monitors) for monitor sizing and placement.