Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

advancedceph-osd~25 min

OSD fails repeatedly after restart - journal / DB device failure

Reported symptoms

  • ●ceph orch ps shows osd.N repeatedly in error state
  • ●Recovered OSD stays up briefly then crashes
  • ●New OSD creation fails with bluestore error opening path

Evidence

  • · journalctl -u ceph-osd@N shows read error on the device
  • · ceph-volume lvm list shows metadata corruption
  • · ceph-bluestore-tool fsck --path /var/lib/ceph/osd/ceph-N reports checksum errors
Diagnosis and resolutionclick to reveal

Root cause

The BlueStore DB device has corruption. The OSD cannot recover from the corruption on restart; replacing the data device alone is insufficient.

Remediation

Replace the DB device; ceph-volume lvm zap --osd-uuid <uuid>; ceph orch daemon add osd <host>; verify the OSD comes up and rejoins.

Verification

ceph orch ps shows the OSD running with no errors; cluster returns to HEALTH_OK; affected PGs re-replicate.

Prevention

Use SSDs for the DB device; monitor DB device health separately; never run DB on the same device as data.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • ceph orch ps shows osd.N repeatedly in error state
  • Recovered OSD stays up briefly then crashes
  • New OSD creation fails with bluestore error opening path

Evidence

  • journalctl -u ceph-osd@N shows read error on the device
  • ceph-volume lvm list shows metadata corruption
  • ceph-bluestore-tool fsck —path /var/lib/ceph/osd/ceph-N reports checksum errors

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.