← All break/fix scenarios in Proxmox VE
ZFS pool degraded after a disk failure
Reported symptoms
- ●zpool status reports state DEGRADED with one or more UNAVAIL devices
- ●VMs on the pool freeze or report I/O errors
- ●dmesg shows SCSI error or device timeout messages
- ●SMART reports reallocated_sector_count increasing
Evidence
- · zpool status -v shows the specific device as UNAVAIL
- · smartctl -H /dev/sdX returns FAILED
- · lsblk does not show the affected disk or shows it as errored
- · Pool has spare available (if configured): spare shown as ACTIVE
Diagnosis and resolutionclick to reveal
Root cause
A physical disk in the ZFS pool has failed. If a hot spare was configured, it activates automatically. If not, the pool is in a degraded state and at risk of data loss if another disk fails.
Remediation
1. Identify the failed disk: `zpool status -v` (look for UNAVAIL) `smartctl -a /dev/sdX` (look for SMART failures) 2. If a spare is already active, the pool is resilvering: `zpool status datapool` (watch resilver progress) 3. Once resilver completes (or after replacing the failed disk): - Power off the host (if the disk is hot-swap, skip this) - Physically replace the disk - Bring the new disk online 4. Replace the failed disk: `zpool replace datapool <old-device> /dev/sdX` 5. If using a disk by ID (recommended): `zpool replace datapool /dev/disk/by-id/<old> /dev/disk/by-id/<new>` 6. Watch the resilver: `zpool status -v datapool` (every few minutes)
Verification
- `zpool status` returns ONLINE state - No resilver in progress - All VMs on the pool are responsive - SMART on the new disk is healthy
Prevention
- Always configure at least one hot spare per pool - Use disk-by-id paths, not sdX names - Enable SMART monitoring via smartd - Test the replacement procedure in a lab