Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

advancedceph-mds-replay~22 min

MDS journal replay takes a long time

Reported symptoms

  • ●MDS comes back up but takes minutes to serve
  • ●Clients see slow file I/O during replay
  • ●CephFS latency is elevated for an extended period

Evidence

  • · ceph mds metadata shows a large journal file
  • · ceph mds status reports replay for an extended time
  • · ceph daemonperf mds.<id> shows replay activity
Diagnosis and resolutionclick to reveal

Root cause

The MDS journal has grown large. Replay takes longer for larger journals; tune mds_log_max_segments.

Remediation

Wait for replay to complete; tune the journal size; document the recovery time.

Verification

ceph mds stat reports the MDS is active; replay is complete; CephFS latency returns to baseline.

Prevention

Tune mds_log_max_segments; allocate enough RAM for the MDS; monitor the journal size.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • MDS comes back up but takes minutes to serve
  • Clients see slow file I/O during replay
  • CephFS latency is elevated for an extended period

Evidence

  • ceph mds metadata shows a large journal file
  • ceph mds status reports replay for an extended time
  • ceph daemonperf mds.&lt;id&gt; shows replay activity

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.