Recover monitor quorum after multi-MON loss

This runbook covers Recover monitor quorum after multi-MON loss on a Ceph Tentacle cluster.

When to use

• Active production scenario matching the impact (data-loss-risk)
• The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

• Multiple MONs are lost
• Quorum is broken
• The cluster cannot commit new maps

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph -s reports fewer than half the MONs are in quorum
# ceph quorum_status lists the surviving MONs
# The lost MONs are identified
# Time and DNS are healthy

Procedure

Step 1

Restore time and DNS on the affected hosts

Step 2

Add the affected hosts back to the cluster: ceph orch host add

Step 3

Restore the MON store from backup

Step 4

Deploy the lost MONs: ceph orch daemon add mon <host>

Step 5

Verify: ceph -s reports quorum restored

Verification

After completing the procedure, verify the result:

# All MONs are in quorum
# ceph -s reports HEALTH_OK
# Recovery can proceed

Rollback

If the procedure does not produce the expected verification, roll back:

• If the MON store is corrupt: restore from backup
• If the new MON cannot join: verify time, DNS, network, SSH
• If the cluster is unrecoverable: escalate

Escalation

If you cannot roll back or the rollback fails, escalate:

• Storage team
• Disaster recovery team