Losing two monitors — the cluster stops

Why this matters in production

This is a complete outage with, in almost every case, zero data loss. Keeping those two facts separate is what prevents a bad situation becoming an unrecoverable one, because the destructive recovery options are far more dangerous than the outage itself.

What stops

With one monitor of three surviving:

• No map can be committed. No OSD can be marked down or out.
• The surviving monitor cannot serve reads once its lease expires.
• ceph commands hang or time out.
• OSDs continue running but stop serving, because they cannot validate their map.
• Clients block.

What does not happen

• No data is lost. Objects are on OSD devices, untouched.
• No recovery starts. Without map commits, nothing is marked out.
• No corruption occurs. The cluster stopped rather than proceeding without agreement, which is the entire point of quorum.

The extreme measure, and its cost

If two monitor stores are genuinely destroyed and only one survives, the monmap on the survivor can be edited to make it a single-monitor cluster, which restores quorum:

# on the surviving monitor host, with the daemon stopped
ceph-mon -i ceph-01 --extract-monmap /tmp/monmap
monmaptool /tmp/monmap --rm ceph-02 --rm ceph-03
ceph-mon -i ceph-01 --inject-monmap /tmp/monmap
# start the monitor, then add replacements

This works and it is dangerous, because it makes one monitor authoritative. If that monitor’s store is stale or damaged, the cluster comes back with a wrong view and there is no second copy to check it against.

Use it only when:

• the other two stores are confirmed unrecoverable, and
• the surviving monitor was in quorum immediately before the incident, and
• you have a backup of the surviving store taken before the edit.

The sequence

1. Confirm the scope: how many monitors, which are reachable.
2. Confirm data safety: OSD hosts are up, devices present. Do not touch OSDs.
3. Work the ordinary causes in order — power, daemons, disk space, network, clocks.
4. Restore any one absent monitor. Quorum returns; the cluster resumes.
5. Only if two stores are unrecoverable, consider the monmap edit, with a backup first.
6. After recovery, restore the monitor count and review why two were lost together.

Quiz

Production discipline

Separate the two facts immediately: this is a total outage and the data is almost certainly intact. Work the ordinary causes in order — power, daemons, disk space, network, clocks — because restoring any one absent monitor returns quorum with no Ceph-side change, and most of these incidents are a network path or a full filesystem. Reserve the monmap edit for confirmed unrecoverable stores, take a backup before it, and treat it as a correctness risk rather than a repair.

Cross-course references

• Ceph: Part CXVII (Lost Monitor Quorum) for the full incident runbook.
• Ceph: Part CX (Monitor Recovery) for store-level repair.
• Ceph: Part IV (Failure Domains) for why two racks failed together.