Storage monitoring: S.M.A.R.T. and ZFS scrub health

The cheapest way to recover from a disk failure is to replace the disk before it fails. S.M.A.R.T. attributes and ZFS scrub status give you weeks of warning — if you’re reading the numbers. This lesson shows how to read them, alert on them, and wire them into monitoring.

Reading S.M.A.R.T.

smartctl is the standard tool:

apt install -y smartmontools
smartctl -a /dev/sda

The output has dozens of attributes. The ones that matter:

For SSDs, also watch:

Running a short self-test

smartctl -t short /dev/sda runs a 2-minute self-test. long runs ~2 hours. After the test:

smartctl -l selftest /dev/sda
# Or
smartctl -a /dev/sda | grep -A1 'Self-test execution status'

A “Completed without error” is what you want. “Completed: read failure” means the disk has bad sectors — replace it.

Setting up smartd

smartd runs continuously and alerts you on attribute changes. The PVE installer doesn’t enable it by default.

# /etc/smartd.conf — replace DEVICESCAN with explicit device paths
# -a: enable most attributes
# -o on: enable offline testing
# -S on: enable autosave
# -n never: don't skip tests even if power is on battery
# -m admin@example.com: email on alert
# -M exec /usr/local/bin/smartd-hook.sh: run this script on alert
DEVICESCAN -a -o on -S on -n never,q -m admin@example.com -M test

For a multi-disk server:

# One line per disk, with serial number for unambiguous identification
/dev/disk/by-id/ata-Samsung_SSD_870_EVO_1TB_S6PXNG0R200001 -a -o on -S on -n never \
  -m admin@example.com -M exec /usr/local/bin/smartd-hook.sh
/dev/disk/by-id/ata-Samsung_SSD_870_EVO_1TB_S6PXNG0R200002 -a -o on -S on -n never \
  -m admin@example.com -M exec /usr/local/bin/smartd-hook.sh

The hook script can call your alerting system (Slack webhook, PagerDuty, etc.). A minimal example:

#!/bin/bash
# /usr/local/bin/smartd-hook.sh
while read -r line; do
  curl -X POST -H 'Content-Type: application/json' \
    -d "{\"text\": \"S.M.A.R.T. alert on $(hostname): $line\"}" \
    https://hooks.slack.com/services/YOUR/WEBHOOK/URL
done

chmod +x /usr/local/bin/smartd-hook.sh
systemctl enable --now smartd
systemctl restart smartd

Test the configuration with smartd -q onecheck — it sends a test alert without waiting for an actual S.M.A.R.T. event.

ZFS scrub health

S.M.A.R.T. tells you about the disk. ZFS tells you about the data. A scrub verifies every block’s checksum against its parent and rebuilds from redundancy when corruption is found.

# Manual scrub
zpool scrub tank

# Status
zpool status tank
# Look for:
#   state: ONLINE
#   scan: scrub repaired 0 in XhXm with 0 errors on Mon Jan 1 12:00:00 2024
#   errors: No known data errors

# Scrub history
zpool history tank | grep scrub

If a scrub finds and repairs errors, the count is the number of silent corruptions ZFS caught. If you see non-zero numbers, replace the disk that produced the bad blocks — ZFS will tell you which one.

Schedule weekly scrubs

The zfs-scrub-weekly@pool.timer template ships with PVE:

systemctl enable --now zfs-scrub-weekly@tank.timer
systemctl list-timers zfs-scrub-weekly*

For pools that hold backups (large, infrequent writes), monthly is fine:

# (heredoc replaced)
echo "[Unit]" >> /etc/systemd/system/zfs-scrub-monthly@pbs.timer
echo "Description=Monthly ZFS scrub on %i" >> /etc/systemd/system/zfs-scrub-monthly@pbs.timer
echo "" >> /etc/systemd/system/zfs-scrub-monthly@pbs.timer
echo "[Timer]" >> /etc/systemd/system/zfs-scrub-monthly@pbs.timer
echo "OnCalendar=*-*-01 03:00:00" >> /etc/systemd/system/zfs-scrub-monthly@pbs.timer
echo "RandomizedDelaySec=2h" >> /etc/systemd/system/zfs-scrub-monthly@pbs.timer
echo "Persistent=true" >> /etc/systemd/system/zfs-scrub-monthly@pbs.timer
echo "" >> /etc/systemd/system/zfs-scrub-monthly@pbs.timer
echo "[Install]" >> /etc/systemd/system/zfs-scrub-monthly@pbs.timer
echo "WantedBy=timers.target" >> /etc/systemd/system/zfs-scrub-monthly@pbs.timer
# (heredoc replaced)
echo "[Unit]" >> /etc/systemd/system/zfs-scrub-monthly@pbs.service
echo "Description=Monthly ZFS scrub on %i" >> /etc/systemd/system/zfs-scrub-monthly@pbs.service
echo "" >> /etc/systemd/system/zfs-scrub-monthly@pbs.service
echo "[Service]" >> /etc/systemd/system/zfs-scrub-monthly@pbs.service
echo "Type=oneshot" >> /etc/systemd/system/zfs-scrub-monthly@pbs.service
echo "ExecStart=/sbin/zpool scrub %i" >> /etc/systemd/system/zfs-scrub-monthly@pbs.service
systemctl enable --now zfs-scrub-monthly@pbs.timer

Alert on scrub errors

The alert script, /usr/local/bin/zfs-scrub-alert.sh:

#!/bin/bash
# Substitute your own webhook URL before running:
WEBHOOK_URL=https://hooks.slack.com/services/REPLACE/ME/NOW

pool="$1"
result=$(zpool status -x "$pool")
if [ $? -ne 0 ]; then
  curl -X POST -H 'Content-Type: application/json' \
    -d "{\"text\": \"ZFS pool $pool degraded on $(hostname): $result\"}" \
    "$WEBHOOK_URL"
fi

Make it executable with chmod +x /usr/local/bin/zfs-scrub-alert.sh, then add the unit that calls it, /etc/systemd/system/zpool-scrub-alert@.service:

[Unit]
Description=ZFS scrub alert for %i

[Service]
Type=oneshot
ExecStart=/usr/local/bin/zfs-scrub-alert.sh %i

Wire it up via a path unit on the ZFS event log, /etc/systemd/path/zfs-scrub-finished.path:

[Unit]
Description=Watch for ZFS scrub completion

[Path]
PathChanged=/proc/spl/kstat/zfs/$1/io

[Install]
WantedBy=multi-user.target

(The exact path unit depends on ZFS version; the systemd approach is fragile. Simpler: run the alert script from a cron job every hour that checks zpool status -x for any pool.)

/etc/cron.hourly/zfs-status-check:

#!/bin/bash
for pool in $(zpool list -H -o name); do
  result=$(zpool status -x "$pool" 2>&1)
  if [ $? -ne 0 ]; then
    /usr/local/bin/zfs-scrub-alert.sh "$pool"
  fi
done

Make it executable with chmod +x /etc/cron.hourly/zfs-status-check.

Other storage-health signals

A few more things to monitor:

• Pool capacity — ZFS performance collapses above 80% full, especially on copy-on-write filesystems where free space is needed for new blocks. Alert at 70%.
• Fragmentation — zpool list -v shows fragmentation %. Above 30% is worth investigating; above 50% consider rebalancing.
• Resilver time — a healthy resilver takes hours; a degraded resilver (rebuilding onto a failing second disk) takes days. Watch resilver progress and abort + replace if it stalls.
• I/O errors in dmesg — dmesg --since '1 day ago' | grep -i 'i/o error\|medium error' is the canonical “something is wrong” check. Wire it into cron.

Putting it together: a storage-health dashboard

For Grafana + Prometheus, the standard exporters are:

• smartctl_exporter — exposes S.M.A.R.T. attributes as Prometheus metrics.
• zfs_exporter — exposes pool state, fragmentation, scrub status.
• node_exporter — disk-space and SMART summary.

A minimal Grafana dashboard:

• Per-pool: state (online/degraded), capacity %, last scrub age, last scrub errors
• Per-disk: S.M.A.R.T. attribute 5 (reallocated sectors), 187 (uncorrectable), 199 (CRC), 197 (pending sectors), 231 (SSD life left)
• Per-host: I/O wait, dmesg errors in the last hour

Set alerts on:

• Pool state != ONLINE
• Capacity > 80%
• Last scrub > 14 days ago
• Reallocated sectors > 0
• Pending sectors > 0
• SSD life < 10%

These alerts fire weeks before a real failure. The response time is hours, not minutes — plenty of time to order a replacement drive and schedule a maintenance window.

Production considerations

• Test alerts regularly. A monitoring system that never fires is worse than no monitoring, because the team trusts it.
• Set different alert thresholds for different disk classes. Consumer SSDs wear out faster than enterprise. Don’t alert on consumer SSD wear at 50%.
• Track S.M.A.R.T. trends, not absolute values. A disk that goes from 0 to 5 reallocated sectors in a week is failing. A disk that has had 5 reallocated sectors for two years is fine.
• Backup before replacing. When you replace a disk in a degraded pool, the chance of a second disk failing during resilver is small but non-zero. Have a backup before you start.

Common mistakes

• Trusting S.M.A.R.T. alone. A disk can go from healthy to failed in 24 hours with no S.M.A.R.T. warning. S.M.A.R.T. is a probability signal, not a guarantee.
• Not scheduling scrubs. Without scrubs, silent corruption isn’t detected until ZFS tries to read the corrupted block — which can be long after the second copy is also bad.
• Ignoring pending sectors. Attribute 197 (pending sectors) is the most reliable early-warning attribute. Replace the disk as soon as it goes non-zero.
• Not testing smartd alert delivery. Configure smartd, walk away, and a year later discover that the alerts went to /dev/null because you typo’d the webhook URL.

Key takeaways

• Watch attribute 197 (pending sectors), 5 (reallocated), 187 (uncorrectable) on every disk.
• Schedule ZFS scrubs weekly on production pools.
• Save baselines; alert on trends, not absolutes.
• Test your alerting — a system that never fires is worse than no system.

Knowledge check