Skip to main content
RunBook Academy

← All runbooks in Ceph

high riskcluster affecting~20 min

Investigate monitor quorum

1 · Prerequisites

Confirm every item is in place before any state change.

  • ceph -s reports fewer than expected MONs in quorum
  • ceph quorum_status lists the MON set and the leader

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · ceph -s shows the MONs and their state
  • · ceph quorum_status shows the ep rank
  • · ceph mon stat shows the leader election epoch
  • · Time on each MON is synchronised to within 100ms
  • · DNS resolves every MON hostname
  • · The monitor port is reachable between every MON pair

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Identify the lost MON(s)
  2. 2Determine the cause: host down, network partition, clock skew
  3. 3If host down: see ceph-rb-recover-failed-control-plane
  4. 4If network: verify the network and switch
  5. 5If clock skew: verify NTP on the affected host

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓All MONs are in quorum
  • ✓ceph quorum_status shows the expected counts
  • ✓Time is synchronised
  • ✓Network is healthy

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the failure is irrecoverable: escalate to the disaster-recovery runbook
  • ↶If quorum cannot be restored: the cluster cannot commit map changes
  • ↶If monitor store is corrupt: restore from the latest backup

6 · Escalation

When the runbook isn't enough, contact:

  • · Storage team
  • · Vendor support
  • · Disaster recovery team

Investigate monitor quorum

This runbook covers Investigate monitor quorum on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (cluster-affecting)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • ceph -s reports fewer than expected MONs in quorum
  • ceph quorum_status lists the MON set and the leader

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph -s shows the MONs and their state
# ceph quorum_status shows the ep rank
# ceph mon stat shows the leader election epoch
# Time on each MON is synchronised to within 100ms
# DNS resolves every MON hostname
# The monitor port is reachable between every MON pair

Procedure

Step 1

Identify the lost MON(s)

Step 2

Determine the cause: host down, network partition, clock skew

Step 3

If host down: see ceph-rb-recover-failed-control-plane

Step 4

If network: verify the network and switch

Step 5

If clock skew: verify NTP on the affected host

Verification

After completing the procedure, verify the result:

# All MONs are in quorum
# ceph quorum_status shows the expected counts
# Time is synchronised
# Network is healthy

Rollback

If the procedure does not produce the expected verification, roll back:

  • If the failure is irrecoverable: escalate to the disaster-recovery runbook
  • If quorum cannot be restored: the cluster cannot commit map changes
  • If monitor store is corrupt: restore from the latest backup

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Storage team
  • Vendor support
  • Disaster recovery team

References

  1. Ceph documentation