Monitor recovery order — what to try, in what sequence

Why this matters in production

Monitor recovery has a natural order from safe to dangerous, and the common failure is jumping several steps because the situation feels urgent. Working the order takes longer to describe than to perform.

The sequence

flowchart TD
  A[Monitor not in quorum] --> B{Host reachable?}
  B -->|No| C[Fix host: power, network, boot]
  B -->|Yes| D{Daemon running?}
  D -->|No| E[Check disk space, then start daemon]
  D -->|Yes| F{Synchronising?}
  F -->|Yes| G[Wait; check disk and network throughput]
  F -->|No| H{Quorum held by others?}
  H -->|Yes| I[Destroy and recreate this monitor]
  H -->|No| J[Investigate before any destructive step]

Step 1 — host and daemon

# Cluster FSID from `ceph fsid`; substitute your own:
FSID=3e0b2c14-9f3a-4d21-8a77-1c9f0e2b5d64

ceph orch ps --daemon-type mon
ssh ceph-02 "systemctl status ceph-$FSID@mon.ceph-02.service"
ssh ceph-02 'df -h /var/lib/ceph'
cephadm logs --name mon.ceph-02 --since 1h

A full filesystem is the single most common reason a monitor is down and stays down. Check it before anything else.

Step 2 — restart

ceph orch daemon restart mon.ceph-02
ceph quorum_status

Safe when others hold quorum. Give it time to synchronise before concluding it failed — check state rather than waiting blindly:

ceph daemon mon.ceph-02 mon_status | jq .state

Step 3 — recreate

If the daemon will not start or will not sync, and quorum is held by others, recreate it. Monitors hold no unique data, so this is cheap:

ceph orch daemon rm mon.ceph-02 --force
ceph mon rm ceph-02
ceph orch apply mon --placement="ceph-01 ceph-03 ceph-02"
ceph quorum_status

Step 4 — monmap injection

Only when quorum is lost and remaining stores are unrecoverable:

ceph orch daemon stop mon.ceph-01
cephadm shell --name mon.ceph-01 -- \
  ceph-mon -i ceph-01 --extract-monmap /tmp/monmap
cephadm shell --name mon.ceph-01 -- \
  monmaptool /tmp/monmap --rm ceph-02 --rm ceph-03 --print
cephadm shell --name mon.ceph-01 -- \
  ceph-mon -i ceph-01 --inject-monmap /tmp/monmap
ceph orch daemon start mon.ceph-01

The --print step before injecting is not optional. Confirm the resulting monmap contains exactly what you intend.

Step 5 — rebuild from OSDs

The last resort, when every monitor store is gone:

# on each OSD host, for each OSD
# OSD_ID is the numeric id from `ceph osd ls`; substitute your own:
OSD_ID=12

ceph-objectstore-tool --data-path "/var/lib/ceph/osd/ceph-$OSD_ID" \
  --op update-mon-db --mon-store-path /tmp/mon-store
# consolidate, then
ceph-monstore-tool /tmp/mon-store rebuild

This reconstructs maps from OSD-held copies. cephx keys must be recreated separately, which means re-keying every client and daemon.

After recovery

Restore the full monitor topology, verify placement across failure domains, confirm quorum, and take a fresh store backup. Then work out why the monitors were lost together — that is usually the more important finding.

Quiz

Production discipline

Work the sequence in order: host, daemon, disk space, restart, recreate — every one of those is validated by the surviving quorum, so a mistake costs nothing. Cross into monmap injection or store rebuild only when quorum is already lost and the other stores are confirmed unrecoverable, and take a store backup first even when it feels like there is no time, because that copy is the only way back. Afterwards, restore the full monitor topology and then find out why two monitors were lost together.

Cross-course references

• Ceph: Part CX (Monitor Recovery) for the deep procedures.
• Ceph: Part VIII (Monitors) for the monitor store contents.
• Ceph: Part CXVII (Lost Monitor Quorum) for the incident context.