Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

advancedceph-pg-degraded~25 min

PG is active+undersized - fewer replicas than size

Reported symptoms

  • ●A PG is active+undersized
  • ●The acting set has fewer replicas than size
  • ●ceph -s reports no degraded PGs

Evidence

  • · ceph pg <pgid> query shows the acting set is n < size
  • · ceph osd tree shows a missing OSD in the CRUSH hierarchy
  • · No backfill activity is reported
Diagnosis and resolutionclick to reveal

Root cause

The CRUSH rule cannot place size replicas in distinct failure domains. The cluster is missing OSDs in the right bucket.

Remediation

Add OSDs in the failure domain; reduce the size of the pool; correct the CRUSH rule.

Verification

ceph osd tree shows the right OSDs in the right buckets; CRUSH rule can place size replicas; PGs return to active+clean.

Prevention

Verify CRUSH placement before adding data; size pools to available OSDs; rehearse CRUSH changes.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • A PG is active+undersized
  • The acting set has fewer replicas than size
  • ceph -s reports no degraded PGs

Evidence

  • ceph pg &lt;pgid&gt; query shows the acting set is n < size
  • ceph osd tree shows a missing OSD in the CRUSH hierarchy
  • No backfill activity is reported

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.