Replace a failed Ceph OSD

This runbook covers replacing a single failed OSD.

When to use

• A specific OSD is reporting down or is missing
• SMART errors indicate imminent disk failure
• Capacity needs expansion by adding a new disk to an existing host

Prerequisites

• Replacement disk is installed in the host
• Disk size ≥ the failed disk
• Disk type matches (NVMe to NVMe, SATA to SATA)
• Cluster has capacity headroom (≥ 30 % free in the relevant pool)

Pre-checks

ceph -s
ceph osd tree | grep -E 'down|out'
ceph osd df | head

Identify the failed OSD ID and host.

Procedure

Step 1: Mark the OSD out

# Substitute the id you identified above:
OSD_ID=7

ceph osd out "$OSD_ID"

Wait for backfill/recovery to complete (verify with ceph -s; recovery: 0 active).

Step 2: Stop and destroy the OSD

# Substitute the id you identified above:
OSD_ID=7

pveceph stop --service "osd.$OSD_ID"
ceph osd purge "$OSD_ID" --yes-i-really-mean-it

Step 3: Physically replace the disk

Replace the failed disk with the new one. For hot-swap bays, this is hot. For SATA direct-attach, schedule a brief downtime.

Step 4: Create the new OSD

# Substitute the device the replacement disk came up as (check `lsblk`):
NEW_DISK=/dev/sdd

ceph-volume lvm zap "$NEW_DISK" --destroy
pveceph osd create "$NEW_DISK"

If using a separate DB/WAL device:

# Substitute your own values before running:
NEW_DISK=/dev/sdd
DB_DEV=/dev/nvme0n1
WAL_DEV=/dev/nvme1n1

pveceph osd create "$NEW_DISK" -db_dev "$DB_DEV" -wal_dev "$WAL_DEV"

Step 5: Verify

# Substitute the id Ceph assigned to the new OSD:
NEW_OSD_ID=7

ceph -s
# Expect: HEALTH_OK, all OSDs up
ceph osd tree | grep "$NEW_OSD_ID"
# Expect: up, in
ceph pg stat
# Expect: active+clean for all PGs

Verification checklist

• Cluster is HEALTH_OK
• New OSD appears in ceph osd tree as up in
• PG states are all active+clean
• Recovery has completed
• No new health warnings

Rollback

If the new OSD fails to create:

# Substitute the device the replacement disk came up as:
NEW_DISK=/dev/sdd

# Check for existing partition tables
ceph-volume lvm zap "$NEW_DISK" --destroy
# Try again
pveceph osd create "$NEW_DISK"

If recovery stalls (cluster overcommitted):

ceph config set osd osd_recovery_max_active 1
ceph config set osd osd_recovery_sleep 5

Escalation

• Storage team: if multiple OSDs are degraded simultaneously
• Vendor support: if hardware diagnostics point to controller or backplane issues
• Capacity team: if the replacement adds capacity and triggers a rebalance