Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

advancedceph-net-partition~22 min

Network partition - asymmetric MON failure

Reported symptoms

  • ●A MON is unreachable from the cluster
  • ●The other MONs continue serving (quorum preserved)
  • ●Recovery continues; slow ops if the partition affects an OSD

Evidence

  • · ceph quorum_status shows the partitioned MON
  • · ceph -s shows the cluster MON set
  • · ping from the partitioned MON to the others fails in one direction
Diagnosis and resolutionclick to reveal

Root cause

A partition split the cluster between the partitioned MON and the rest.

Remediation

Restore the network. On restore, the partitioned MON rejoins quorum; verify the map converges.

Verification

ceph quorum_status shows all MONs in quorum; no MON is partitioned; the cluster is HEALTH_OK.

Prevention

Redundant paths (LACP, MLAG); monitor the partitioned MON health; rehearse asymmetric failures.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • A MON is unreachable from the cluster
  • The other MONs continue serving (quorum preserved)
  • Recovery continues; slow ops if the partition affects an OSD

Evidence

  • ceph quorum_status shows the partitioned MON
  • ceph -s shows the cluster MON set
  • ping from the partitioned MON to the others fails in one direction

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.