Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

advancedceph-slow-ops~25 min

Slow ops during recovery - I/O is the bottleneck

Reported symptoms

  • ●ceph health detail reports slow ops
  • ●Client I/O latency p99 jumps above 50ms
  • ●VMs behind RBD show slow disk performance

Evidence

  • · ceph -s reports a recovery or backfill in progress
  • · ceph daemonperf <osd> shows a particular OSD with high latency
  • · ceph osd perf shows OSDs at saturation
Diagnosis and resolutionclick to reveal

Root cause

A backfill or recovery is running; the chosen recovery rate saturates disk and network; client I/O waits for recovery to finish.

Remediation

Throttle recovery: ceph config set osd osd_recovery_max_active 1 and osd_recovery_sleep 1; verify client latency returns to baseline.

Verification

ceph health detail shows no slow ops; client I/O p99 returns to baseline; recovery continues at the throttled rate.

Prevention

Tune recovery settings in advance based on the cluster I/O headroom; alert on slow ops before they reach customers.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • ceph health detail reports slow ops
  • Client I/O latency p99 jumps above 50ms
  • VMs behind RBD show slow disk performance

Evidence

  • ceph -s reports a recovery or backfill in progress
  • ceph daemonperf &lt;osd&gt; shows a particular OSD with high latency
  • ceph osd perf shows OSDs at saturation

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.