Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

intermediateceph-pg-degraded~18 min

PG is active+degraded after an OSD failure

Reported symptoms

  • ●ceph -s shows HEALTH_WARN with degraded data
  • ●A specific PG is active+degraded+remapped
  • ●ceph pg dump_stuck lists the stuck PGs

Evidence

  • · ceph pg <pgid> query shows the missing OSD in the acting set
  • · ceph osd tree shows the OSD as down + in
  • · smartctl -a /dev/sdX reports device errors
Diagnosis and resolutionclick to reveal

Root cause

An OSD on the missing replica host has hardware failure. The PG is degraded until recovery reproduces the replica.

Remediation

Replace the failed disk; wait for the new OSD to come up; verify recovery completes (ceph progress); verify the PG returns to active+clean.

Verification

ceph pg <pgid> query shows the acting set contains the full set of replicas; the PG is active+clean; the cluster is HEALTH_OK.

Prevention

RAID for additional disk-failure tolerance; SMART monitoring with alerting; periodic OSD replacement drills.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • ceph -s shows HEALTH_WARN with degraded data
  • A specific PG is active+degraded+remapped
  • ceph pg dump_stuck lists the stuck PGs

Evidence

  • ceph pg &lt;pgid&gt; query shows the missing OSD in the acting set
  • ceph osd tree shows the OSD as down + in
  • smartctl -a /dev/sdX reports device errors

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.