Skip to main content
RunBook Academy

← All runbooks in Ceph

high riskcluster affecting~60 min

Recover a lost MON

1 · Prerequisites

Confirm every item is in place before any state change.

  • A MON is unreachable
  • The cluster has quorum with the remaining MONs
  • The host is recoverable or the MON can be redeployed

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · ceph -s shows the lost MON
  • · ceph quorum_status lists the surviving MONs
  • · The host is reachable
  • · Time and DNS are synchronised

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1ceph mon remove <id> if the lost MON cannot be recovered
  2. 2Provision the new host
  3. 3Add the host: ceph orch host add <hostname>
  4. 4Deploy a new MON: ceph orch daemon add mon <hostname>
  5. 5Verify the new MON joins quorum: ceph mon stat
  6. 6Verify the cluster maps commit

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓ceph -s reports 3 (or 5) MONs in quorum
  • ✓ceph quorum_status lists all MONs
  • ✓A new map commits
  • ✓Cluster is HEALTH_OK

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the new MON does not start: verify podman, time, DNS, SSH keys, monitor port
  • ↶If the MON joins but loses quorum again: check time, network, monitor store
  • ↶If the monitor store is corrupt: see ceph-rb-monitor-recovery

6 · Escalation

When the runbook isn't enough, contact:

  • · Storage team if the recovery stalls
  • · Disaster recovery team if the MON store is corrupt

Recover a lost MON

This runbook covers Recover a lost MON on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (cluster-affecting)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • A MON is unreachable
  • The cluster has quorum with the remaining MONs
  • The host is recoverable or the MON can be redeployed

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph -s shows the lost MON
# ceph quorum_status lists the surviving MONs
# The host is reachable
# Time and DNS are synchronised

Procedure

Step 1

ceph mon remove <id> if the lost MON cannot be recovered

Step 2

Provision the new host

Step 3

Add the host: ceph orch host add <hostname>

Step 4

Deploy a new MON: ceph orch daemon add mon <hostname>

Step 5

Verify the new MON joins quorum: ceph mon stat

Step 6

Verify the cluster maps commit

Verification

After completing the procedure, verify the result:

# ceph -s reports 3 (or 5) MONs in quorum
# ceph quorum_status lists all MONs
# A new map commits
# Cluster is HEALTH_OK

Rollback

If the procedure does not produce the expected verification, roll back:

  • If the new MON does not start: verify podman, time, DNS, SSH keys, monitor port
  • If the MON joins but loses quorum again: check time, network, monitor store
  • If the monitor store is corrupt: see ceph-rb-monitor-recovery

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Storage team if the recovery stalls
  • Disaster recovery team if the MON store is corrupt

References

  1. Ceph documentation