Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

intermediateceph-mds-failover~25 min

MDS failure - CephFS becomes unavailable

Reported symptoms

  • ●CephFS mount returns stale file handle
  • ●ceph mds stat reports the active MDS is missing
  • ●CephFS file I/O errors for more than 30 seconds

Evidence

  • · ceph mds metadata shows the failover in progress
  • · ceph orch ps shows the standby MDS taking over
  • · ceph mds stat shows the rank with a new active MDS
Diagnosis and resolutionclick to reveal

Root cause

The active MDS daemon crashed. The standby MDS is taking over; the failure window is visible to clients.

Remediation

Wait for the standby to take over (typically < 30s); if failover stalls, ceph orch daemon restart mds.<id>; verify CephFS mounts resume operation.

Verification

ceph mds stat shows a healthy active MDS; CephFS mounts respond to ls and stat; clients reconnect; latency returns to baseline.

Prevention

Run standby MDSes on different hosts; spread MDS across racks; rehearse MDS failover quarterly.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • CephFS mount returns stale file handle
  • ceph mds stat reports the active MDS is missing
  • CephFS file I/O errors for more than 30 seconds

Evidence

  • ceph mds metadata shows the failover in progress
  • ceph orch ps shows the standby MDS taking over
  • ceph mds stat shows the rank with a new active MDS

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.