Skip to main content
RunBook Academy

← All checklists in Ceph

As neededceph-osd-replacement

OSD Replacement Checklist

8 items ·6 critical ·2 warn ·0 info

Purpose

Replacing an OSD is routine and is the operation most often botched under time pressure. The two classic mistakes are purging the OSD before backfill has finished, which removes a copy the cluster was still relying on, and replacing the disk without ever asking why it died, which schedules the same replacement again next month. This checklist is the guard rail around both.

Order of work

This list is sequential, not thematic, and the frontmatter is in execution order. identify fixes which OSD and which physical device - do it from the tree and the device path together, because pulling the wrong disk from a degraded cluster is how a single failure becomes data loss. out-wait, stop-purge, disk-replace and recreate are the replacement proper, and the wait inside out-wait is a hard gate rather than a courtesy. verify and monitor close the loop on placement and recovery. rca is last and is not optional: without it, the drive model that just failed is still in the purchase order.

Before you start

Have the OSD id, the host, the device path, the drive serial, and confirmation that no other OSD is currently out.

Verification

Done when the new OSD is up and in, every affected PG is active+clean, and the cluster is HEALTH_OK - all three, in that order. Mark an item N/A rather than leaving it unchecked when it does not apply to this cluster.

Escalation

If a second OSD fails while the first is being replaced, stop and escalate. Two concurrent failures in one failure domain is an incident, not maintenance.

Critical6 items

  1. ceph osd tree
  2. ceph progress
  3. ceph osd purge
  4. ceph osd tree
  5. ceph -s

Warning2 items