Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

intermediateceph-backfill~25 min

Backfill is stuck - no progress for an hour

Reported symptoms

  • ●ceph -w shows recovery rate 0
  • ●ceph progress shows no progress
  • ●Affected PGs are active+backfill-to-... for more than 1 hour

Evidence

  • · ceph daemonperf <target-osd> shows low CPU and zero I/O on the target
  • · journalctl -u ceph-osd@<target> shows no log activity
  • · ceph health detail shows the PG state unchanged for an hour
Diagnosis and resolutionclick to reveal

Root cause

A target OSD in the backfill is unresponsive, or the backfill rate is throttled to near zero.

Remediation

Identify the unresponsive target; ceph orch daemon restart osd.<id>; verify backfill resumes; if not, mark the OSD out and replace it.

Verification

ceph progress resumes moving; ceph -w shows recovery activity; backfill completes; PGs return to active+clean.

Prevention

Monitor backfill rate; tune osd_max_backfills; identify unhealthy OSDs before backfill stall.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • ceph -w shows recovery rate 0
  • ceph progress shows no progress
  • Affected PGs are active+backfill-to-… for more than 1 hour

Evidence

  • ceph daemonperf <target-osd> shows low CPU and zero I/O on the target
  • journalctl -u ceph-osd@<target> shows no log activity
  • ceph health detail shows the PG state unchanged for an hour

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.