Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

advancedceph-mon-quorum~30 min

MON quorum lost - two of three MONs are down

Reported symptoms

  • ●ceph -s shows mon: 1 daemons, quorum ceph01
  • ●PG recovery stalls; no new map commits
  • ●Cluster reports HEALTH_WARN with 1 mons down

Evidence

  • · ceph quorum_status shows 1 MON remaining
  • · SSH to ceph02 and ceph03 fails
  • · Both lost MONs share a host failure
Diagnosis and resolutionclick to reveal

Root cause

A host holding two MONs has failed. Without the third MON, the cluster cannot commit map changes.

Remediation

Recover the host first; restore MON stores from backup onto replacement hosts; verify quorum is 2 of 3; verify PGs converge.

Verification

ceph -s shows 3 daemons, quorum ceph01,ceph02,ceph03; PGs return to active+clean; recovery completes.

Prevention

Spread MONs across hosts; spread MONs across racks; rehearse MON recovery quarterly; back up the MON store.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • ceph -s shows mon: 1 daemons, quorum ceph01
  • PG recovery stalls; no new map commits
  • Cluster reports HEALTH_WARN with 1 mons down

Evidence

  • ceph quorum_status shows 1 MON remaining
  • SSH to ceph02 and ceph03 fails
  • Both lost MONs share a host failure

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.