← All break/fix scenarios in Proxmox VE
The disk backing osd.5 (here /dev/sdc on pve-02) has developed I/O errors. The OSD process crashed, but the cluster didn't remove the OSD from the CRUSH map because the disk is still physically present. PGs that were on osd.5 are remapped to other OSDs but data is degraded.
1. Pull the disk: `ceph osd purge osd.5 --yes-i-really-mean-it` (use `purge` if the OSD is permanently down and you want to skip the recovery wait; use `out` then `down` then `purge` for the safer path). 2. Replace the disk physically. 3. Recreate the OSD on the new disk: `ceph-volume lvm create --data /dev/sdc`. 4. Verify CRUSH weight: `ceph osd tree` should show osd.5 with the full weight and `up`. 5. Watch recovery: `ceph -w` (live console) shows PGs transitioning from active+clean+remapped to active+clean.
- `ceph -s` reports health: HEALTH_OK - `ceph osd tree` shows osd.5 up with full weight - All PGs are `active+clean` (no remapped or degraded) - `ceph df` shows the expected usable capacity
- Add SMART monitoring for disks backing OSDs - Configure ZFS / mdadm scrubbing for non-Ceph redundancy layers - Alert on `HEALTH_WARN` rather than only `HEALTH_ERR` - Test the OSD replacement procedure in a lab before you need it
This is the bread-and-butter Ceph incident. The cluster is degraded but still serving data. Your goal is to remove the bad OSD cleanly, replace the disk, and bring it back without overloading the survivors during recovery.