Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

intermediateceph-osd~30 min

NVMe-oF target failure

Reported symptoms

  • ●A NVMe-oF target appears down in the cluster
  • ●OSDs hosted on the NVMe-oF target are unreachable

Evidence

  • · ceph orch ps shows the affected OSDs down
  • · smartctl / nvme smart-log against the target fails
  • · The NVMe-oF initiator is responsive on other hosts
Diagnosis and resolutionclick to reveal

Root cause

The NVMe-oF target host has failed. OSDs on that target are down; the cluster recovers via replication.

Remediation

Mark OSDs out; wait for backfill; identify the target failure; recover the target host; replace if necessary; recreate OSDs.

Verification

OSDs are up + in; backfill completes; cluster is HEALTH_OK.

Prevention

Do not rely on NVMe-oF for primary OSD storage in production; have redundant paths; rehearse NVMe-oF target failure.

Architecture

The student examines the broken architecture diagram and identifies the responsible component.

Symptoms

  • A NVMe-oF target appears down in the cluster
  • OSDs hosted on the NVMe-oF target are unreachable

Evidence

  • ceph orch ps shows the affected OSDs down
  • smartctl / nvme smart-log against the target fails
  • The NVMe-oF initiator is responsive on other hosts

Student investigation

The student follows the methodology and gathers evidence.

Progressive hints

  • Hint 1: read ceph -s.
  • Hint 2: read ceph health detail.
  • Hint 3: trace the failing subsystem.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification commands and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.