Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

intermediateceph-osd~20 min

OSD is down - degraded cluster, recovery in progress

Reported symptoms

  • ●ceph -s shows HEALTH_WARN with osds down
  • ●PGs in active+clean+remapped state
  • ●One OSD host reports I/O errors
  • ●No alert email yet (within SLA window)

Evidence

  • · ceph osd tree shows osd.5 down with host ceph01
  • · ceph osd df tree shows osd.5 weight 0 but capacity still reported
  • · journalctl -u ceph-osd@5 shows OSD crashed 4 hours ago
  • · dmesg shows I/O errors against /dev/sdc
Diagnosis and resolutionclick to reveal

Root cause

The block device backing osd.5 has developed unrecoverable I/O errors. The OSD process crashed; the cluster tolerated the loss through replication but is reconstructing the missing replica.

Remediation

ceph osd out osd.5; wait for backfill; ceph osd purge osd.5 --yes-i-really-mean-it; physically replace /dev/sdc; ceph-volume lvm create --data /dev/sdc; verify ceph osd tree shows osd.5 with full weight and up; watch ceph -w until PGs are active+clean.

Verification

ceph -s shows HEALTH_OK; ceph osd tree shows osd.5 with full weight; PGs are active+clean; ceph df shows the expected usable capacity.

Prevention

SMART monitoring for OSD disks; alert on HEALTH_WARN not only HEALTH_ERR; rehearse the OSD replacement procedure in a lab; offsite replacement disk in stock.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • ceph -s shows HEALTH_WARN with osds down
  • PGs in active+clean+remapped state
  • One OSD host reports I/O errors
  • No alert email yet (within SLA window)

Evidence

  • ceph osd tree shows osd.5 down with host ceph01
  • ceph osd df tree shows osd.5 weight 0 but capacity still reported
  • journalctl -u ceph-osd@5 shows OSD crashed 4 hours ago
  • dmesg shows I/O errors against /dev/sdc

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.