Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

advancedceph-backfillfull~28 min

Backfillfull - backfill is blocked

Reported symptoms

  • ●ceph -s reports POOL_BACKFILLFULL
  • ●ceph health detail shows backfill is blocked
  • ●Recovery continues but backfill stalls

Evidence

  • · ceph osd df identifies the backfillfull OSDs
  • · ceph progress reports zero backfill rate
  • · ceph pg stat shows PGs stuck in active+backfill-to-...
Diagnosis and resolutionclick to reveal

Root cause

A pool has reached osd_backfill_full_ratio (default 0.90). Backfill cannot proceed.

Remediation

Add capacity immediately; identify which pool is the cause; delete data; rebalance; do not raise the threshold.

Verification

Pool utilisation falls below 0.90; backfill resumes; new OSDs start receiving PGs.

Prevention

Plan capacity (Part LXVI); alert at 75% / 85% so you never reach the backfillfull threshold.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • ceph -s reports POOL_BACKFILLFULL
  • ceph health detail shows backfill is blocked
  • Recovery continues but backfill stalls

Evidence

  • ceph osd df identifies the backfillfull OSDs
  • ceph progress reports zero backfill rate
  • ceph pg stat shows PGs stuck in active+backfill-to-…

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.