Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

advancedceph-pg-peering~25 min

PG stuck peering - waits indefinitely on an OSD

Reported symptoms

  • ●A PG shows peering for more than 30 minutes
  • ●ceph pg dump_stuck lists the same PGs
  • ●ceph -s shows recovery 0/0 objects degraded

Evidence

  • · ceph pg <pgid> query shows acting set with the stuck OSD
  • · ceph daemonperf <osd> shows the OSD is not responding
  • · ceph osd tree shows the OSD as down but stuck in in
Diagnosis and resolutionclick to reveal

Root cause

The OSD daemon is unresponsive. The PG cannot complete peering without that OSD.

Remediation

Identify the stuck OSD; ceph orch daemon restart osd.<id>; if that fails, mark the OSD out; verify the PG returns to active+clean.

Verification

ceph pg dump_stuck no longer lists the PG; cluster is HEALTH_OK; affected PGs are active+clean.

Prevention

Monitor OSD daemon responsiveness; tune osd_heartbeat_grace; never leave the cluster with a permanently unresponsive OSD.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • A PG shows peering for more than 30 minutes
  • ceph pg dump_stuck lists the same PGs
  • ceph -s shows recovery 0/0 objects degraded

Evidence

  • ceph pg &lt;pgid&gt; query shows acting set with the stuck OSD
  • ceph daemonperf &lt;osd&gt; shows the OSD is not responding
  • ceph osd tree shows the OSD as down but stuck in in

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.