Skip to main content
RunBook Academy

← All runbooks in Ceph

high riskdata loss risk~90 min

Replace a failed OSD end to end

1 · Prerequisites

Confirm every item is in place before any state change.

  • The replacement disk is on the host
  • A maintenance window is scheduled
  • Capacity headroom is verified

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · The failed OSD is identifiable in ceph osd tree
  • · ceph -s reports the cluster is HEALTH_WARN with the OSD as the source
  • · journalctl -u ceph-osd@<id> shows the daemon crash
  • · smartctl -a /dev/<device> confirms the disk failure

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Mark the OSD out
  2. 2Wait for backfill: ceph -s reports zero recovery
  3. 3Stop the daemon: ceph orch daemon stop osd.<id>
  4. 4Purge the OSD: ceph osd purge osd.<id> --yes-i-really-mean-it
  5. 5Physically replace the disk
  6. 6Recreate the OSD: ceph orch daemon add osd <host>
  7. 7Verify the new OSD is up + in
  8. 8Monitor backfill completion

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓ceph -s is HEALTH_OK
  • ✓The replaced disk appears in the host inventory
  • ✓Affected PGs are active+clean
  • ✓CRUSH weight is correct

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the new OSD fails to create: investigate the device
  • ↶If recovery stalls: tune recovery settings
  • ↶If the new OSD immediately fails: suspect the host

6 · Escalation

When the runbook isn't enough, contact:

  • · Hardware vendor
  • · Storage team if multiple OSDs are failing

Replace a failed OSD end to end

This runbook covers Replace a failed OSD end to end on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (data-loss-risk)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • The replacement disk is on the host
  • A maintenance window is scheduled
  • Capacity headroom is verified

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# The failed OSD is identifiable in ceph osd tree
# ceph -s reports the cluster is HEALTH_WARN with the OSD as the source
# journalctl -u ceph-osd@`<id>` shows the daemon crash
# smartctl -a /dev/<device> confirms the disk failure

Procedure

Step 1

Mark the OSD out

Step 2

Wait for backfill: ceph -s reports zero recovery

Step 3

Stop the daemon: ceph orch daemon stop osd.<id>

Step 4

Purge the OSD: ceph osd purge osd.<id> —yes-i-really-mean-it

Step 5

Physically replace the disk

Step 6

Recreate the OSD: ceph orch daemon add osd <host>

Step 7

Verify the new OSD is up + in

Step 8

Monitor backfill completion

Verification

After completing the procedure, verify the result:

# ceph -s is HEALTH_OK
# The replaced disk appears in the host inventory
# Affected PGs are active+clean
# CRUSH weight is correct

Rollback

If the procedure does not produce the expected verification, roll back:

  • If the new OSD fails to create: investigate the device
  • If recovery stalls: tune recovery settings
  • If the new OSD immediately fails: suspect the host

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Hardware vendor
  • Storage team if multiple OSDs are failing

References

  1. Ceph documentation