Skip to main content
RunBook Academy

← All runbooks in Proxmox VE

medium riskservice affecting~60 min

Replace a failed disk in a ZFS mirror

1 · Prerequisites

Confirm every item is in place before any state change.

  • A ZFS mirror or RAIDZ pool with one failed disk
  • Replacement disk of equal or greater capacity available
  • Hot spare bay or ability to hot-swap

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · zpool status shows the disk as FAULTED or REMOVED
  • · The replacement disk is visible: lsblk
  • · No other disks in the same vdev are showing errors
  • · Recent scrub results are clean

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Identify the failed disk by serial number: lsblk -o NAME,SERIAL,TRAN,MODEL
  2. 2On the host running the affected pool, replace the disk: zpool replace <pool> <old-disk> <new-disk>
  3. 3Watch resilver progress: zpool status <pool> (look for scan: resilver in progress)
  4. 4After resilver completes, run a scrub: zpool scrub <pool>
  5. 5Verify no errors: zpool status <pool> should show no errors and no degraded state

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓zpool status reports ONLINE for the new disk and no degraded vdevs
  • ✓zpool scrub completes with 0 errors
  • ✓Read/write smoke test on the affected dataset: dd if=/dev/zero of=/pool/testfile bs=1M count=100
  • ✓No errors in dmesg during the resilver

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If resilver fails midway, do NOT reuse the partially-resilvered disk
  • ↶Replace with another fresh disk and restart the replace operation
  • ↶If the pool is unrecoverable, restore from PBS backups

6 · Escalation

When the runbook isn't enough, contact:

  • · Storage admin if the second disk in the mirror also starts failing
  • · PBS admin if pool is unrecoverable and restore is needed

Replace a failed disk in a ZFS mirror

ZFS rebuilds (resilver) data onto the new disk in the background. This is usually faster than a scrub because only the data that’s actually on the pool needs to be rewritten, not every block.

Things to remember

  • Never reuse a disk that was previously in a degraded pool as the new mirror member
  • The replacement disk should be the same size or larger
  • Resilver performance impacts the whole pool — schedule during low I/O periods if possible
  • After any disk replacement, run a scrub to catch silent corruption

References

  1. ZFS — Replacing a Disk