Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

intermediateceph-disk-health~22 min

SMART pre-fail on an OSD disk - replacement needed soon

Reported symptoms

  • ●SMART reports Reallocated_Sector_Ct > 0
  • ●The OSD becomes flaky under load
  • ●I/O wait on the host increases

Evidence

  • · smartctl -a /dev/sdX shows the failing SMART attributes
  • · ceph osd tree shows the OSD is up + in
  • · ceph daemonperf <osd> shows a deteriorating latency p99
Diagnosis and resolutionclick to reveal

Root cause

The disk has bad sectors and is about to fail. The OSD will go down + in soon.

Remediation

Plan a maintenance window; ceph osd out <id>; wait for backfill; replace the disk (Part LVII); verify recovery.

Verification

ceph -s reports HEALTH_OK; the replaced disk is on the host; the affected PGs are active+clean.

Prevention

SMART monitoring with alerting; offsite replacement disk; rehearse the procedure quarterly.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • SMART reports Reallocated_Sector_Ct > 0
  • The OSD becomes flaky under load
  • I/O wait on the host increases

Evidence

  • smartctl -a /dev/sdX shows the failing SMART attributes
  • ceph osd tree shows the OSD is up + in
  • ceph daemonperf &lt;osd&gt; shows a deteriorating latency p99

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.