ZFS scrub, resilver, and disk replacement

Why this matters in production

Bit-rot happens. Disks fail. Without regular scrubs and a clear disk-replacement procedure, a single bit-flip on the wrong block destroys data that no one notices until a restore from backup fails.

Scrub vs resilver

A scrub is read-mostly (it reads everything, repairs on the fly). A resilver is write-mostly (it copies specific data to the replacement disk).

Running a scrub

zpool scrub tank

zpool status -v tank

zpool list -o name,last_scrubbed tank

cat > /etc/cron.d/zfs-scrub <<EOF
# Scrub the tank pool on the first Sunday of each month at 02:00
0 2 1-7 * 0   root   [ "$(date +%u)" -eq 7 ] && zpool scrub tank
EOF

What the output means

  pool: tank
 state: ONLINE
  scan: scrub repaired 0 in 2h34m with 0 errors on Sun Aug  3 02:00:12 2026
config:
        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sda     ONLINE       0     0     0
            sdb     ONLINE       0     0     0

• 0 errors — every block matched its checksum.
• repaired 0 — no silent corruption was found and repaired.
• The scan took 2h34m.

If errors appear:

  scan: scrub repaired 128K in 2h34m with 32 errors on Sun Aug  3 02:00:12 2026

32 blocks had checksum errors; ZFS repaired them from redundancy. This is normal — drives develop bad blocks over time. Investigate if the count grows rapidly.

If the pool cannot repair (e.g., two mirror disks both report bad blocks for the same file):

  scan: scrub found 4 unrepairable errors

The 4 files are now corrupt. Recovery is from backup.

What to do when a scrub reports errors, in order

The output tells you three different things and they call for three different responses. Reading them as one number — “the scrub found errors” — is what turns a disk replacement into a data-loss incident.

POOL=tank
zpool status -v "$POOL"
zpool events -v "$POOL" | tail -40

repaired N with 0 errors. ZFS found checksum mismatches and fixed them from redundancy. This is the system working. Disks develop bad blocks; that is why scrubs exist. Note the number and compare it to last month’s.

A non-zero CKSUM count against one device. That device is returning corrupted data. The pool is fine and the disk is not — it is producing errors that redundancy is currently absorbing. Replace it before the redundancy is needed for something else. A rising CKSUM count on one device with zeros on its siblings is a hardware verdict, not a ZFS problem.

Permanent errors, listed by file. Redundancy could not repair these, which means every copy was bad. zpool status -v names them. Restore those specific files or the affected guest from backup; the rest of the pool is unaffected and does not need rebuilding.

Scrubs can be paused, and on a busy pool they should be

POOL=tank

zpool scrub -p "$POOL"    # pause, keeping progress
zpool scrub "$POOL"       # resume
zpool scrub -s "$POOL"    # stop and discard progress

zpool status "$POOL"

This matters because the usual reaction to “the scrub is making production slow” is zpool scrub -s, which throws the work away. Pause it through the busy period and resume afterwards, and the scrub still completes — just later.

Disk replacement

When a disk fails (zpool status shows state: DEGRADED):

1. Physically replace the disk (or attach via hot-swap bay).

2. Tell ZFS about the new disk:

   zpool replace tank /dev/disk/by-id/old-disk /dev/disk/by-id/new-disk

3. Wait for the resilver to complete (zpool status).

4. Verify the pool is ONLINE with all devices present.

zpool replace tank /dev/disk/by-id/old-disk /dev/disk/by-id/new-disk

Hot spares

A hot spare is a disk ZFS automatically uses when another fails:

zpool add tank spare /dev/disk/by-id/spare-disk

After a fault, the spare activates and resilver begins automatically. The operator can physically replace the failed disk later, then “detach” the failed disk from the spare.

Detaching a disk

Once a resilver is complete and the pool is ONLINE, the failed disk can be removed:

zpool detach tank /dev/disk/by-id/old-disk

Production considerations

Common mistakes

• Ignoring CKSUM column errors.
• Running scrubs without monitoring the impact.
• Replacing a disk without verifying the by-id name.
• Forgetting that resilver time is bounded by vdev size, not data size.

A break/fix exercise

Key takeaways

• Scrub: read everything, verify checksums, repair from redundancy.
• Resilver: rebuild a vdev after disk replacement.
• Read the three signals separately: repaired N is the system working, a CKSUM count on one device is a hardware verdict, and permanent errors named by zpool status -v are files to restore.
• CKSUM errors on every device point at a shared component — HBA, cable, backplane, power or non-ECC RAM — not at every disk failing.
• Use zpool scrub -p to pause a scrub through a busy period. -s stops it and discards the progress.
• Schedule scrubs; use hot spares; monitor CKSUM column.

Knowledge check