← All break/fix scenarios in Proxmox VE
Ceph OSD is down and the cluster is degraded
Reported symptoms
- ●ceph -s reports HEALTH_WARN with "1 osd down"
- ●PGs in active+degraded state
- ●VMs on the Ceph pool experience higher latency
- ●OSD host has a SMART warning in dmesg
Evidence
- · ceph osd tree shows the OSD with status "down"
- · ceph osd metadata <osd-id> shows the host and disk
- · smartctl -a /dev/sdX shows reallocated_sector_count > 0
- · dmesg shows I/O errors for the underlying device
Diagnosis and resolutionclick to reveal
Root cause
The OSD's underlying disk has failed or is failing. SMART errors indicate media degradation. The OSD is marked down by the cluster, and the affected PGs are degraded until the OSD returns or is replaced.
Remediation
1. Mark the OSD as "out" of the cluster to start recovery: `ceph osd out osd.<id>` 2. Stop the OSD service: `systemctl stop ceph-osd@<id>` 3. Replace the disk (physically or virtually). 4. If using ceph-volume LVM: `ceph-volume lvm zap /dev/sdX` `ceph-volume lvm create --osd-id <id> /dev/sdX` 5. Start the OSD: `systemctl start ceph-osd@<id>` 6. Mark it back in: `ceph osd in osd.<id>` 7. Watch the recovery progress: `ceph -w`
Verification
- `ceph -s` returns to HEALTH_OK - All PGs are active+clean - The replacement OSD is "up" and "in" - VM I/O latency returns to baseline - SMART monitoring shows the new disk healthy
Prevention
- Enable SMART monitoring on every OSD disk - Use ceph-mgr to send SMART warnings to the alerting system - Schedule periodic scrub and deep-scrub to detect media issues early - Test the OSD replacement procedure in a lab before it happens in production