SMART and NVMe health - disk failure prediction · Linux

SMART (Self-Monitoring, Analysis, and Reporting Technology) predicts HDD failures. NVMe health predicts SSD failures. Both are early warning systems.

SMART for HDDs

# Install
sudo apt install smartmontools

# FIRST: find out what smartctl can actually address on this host
sudo smartctl --scan-open

# Run a test (short, ~2 minutes)
sudo smartctl -t short /dev/sda

# Show results
sudo smartctl -a /dev/sda

Drives behind a RAID controller

smartctl -a /dev/sda only works when /dev/sda is a drive. On a server with a MegaRAID, PERC or HPE Smart Array controller — the hardware the next lesson, RAID controller monitoring, is about — the OS sees one virtual disk. The member drives are behind the controller and SMART cannot reach them without being told how. This is the single most common way SMART monitoring ends up silently blind on a fleet: it returns clean output, because it is describing a virtual disk that has no SMART data of its own.

--scan-open is the discovery step. It reports the -d type needed for each addressable drive:

sudo smartctl --scan-open
# /dev/bus/0 -d megaraid,0 # /dev/bus/0 [megaraid_disk_00], SCSI device
# /dev/bus/0 -d megaraid,1 # /dev/bus/0 [megaraid_disk_01], SCSI device

# Broadcom MegaRAID / Dell PERC - one call per member drive
sudo smartctl -H -A -d megaraid,0 /dev/bus/0

# HPE Smart Array
sudo smartctl -H -A -d cciss,0 /dev/sg0

# Enumerate the drive count from the controller tool first
sudo storcli /c0 show

Key attributes:

Reallocated_Sector_Ct: sectors remapped due to bad blocks. > 0 = drive is dying.
Reallocated_Event_Count: number of remap events.
Current_Pending_Sector: sectors waiting to be remapped.

0 = drive is failing.
Offline_Uncorrectable: sectors that cannot be read.

0 = drive has bad blocks.

If any of these are non-zero, replace the drive.

NVMe health for SSDs

# Install
sudo apt install nvme-cli

# Show health
sudo nvme smart-log /dev/nvme0n1

Key fields:

critical_warning: a bitmask; any bit set = problem. Bit 0 is “available spare below threshold”, bit 1 “temperature past a threshold”, bit 2 “reliability degraded”, bit 3 “media is read-only”, bit 4 “volatile memory backup failed”. This is the field to alert on, because the drive has already done the comparison for you.
temperature: the human-readable output prints Celsius first, with the raw Kelvin value in parentheses (temperature : 34 °C (307 K)); the JSON output gives Kelvin only. Do not compare it against a number you invented — the drive publishes its own limits, which vary by model: sudo nvme id-ctrl /dev/nvme0n1 | grep -i 'wctemp\|cctemp' gives the warning and critical composite thresholds.
available_spare and available_spare_threshold: spare capacity remaining, and the point at which this drive considers it exhausted. The threshold is a per-model value the drive publishes; it is not universally 10%. Compare the two against each other, or just watch critical_warning bit 0, which is set exactly when available_spare < available_spare_threshold.
percentage_used: drive life used. > 90% = near end of life. It is a vendor estimate and is allowed to exceed 100.
data_units_read / data_units_written: drive activity.
power_on_hours: total hours.

End-of-life prediction: monitor percentage_used. When it reaches 100, replace the drive.

Alert on SMART and NVMe

A grep over human-readable output prints lines; it does not detect a condition. It exits 0 whenever the words are present — which is always — so wired into a cron job it either alerts every night or never. Both tools already expose a machine-readable verdict; use it.

# SMART: the exit status is a bitmask, and it is what smartctl
# exists for. -H asks for the health verdict, -A for attributes.
smartctl -H -A /dev/sda >/dev/null
rc=$?
(( rc & 8 ))  && echo "CRITICAL: /dev/sda SMART health FAILED"
(( rc & 32 )) && echo "WARNING: /dev/sda attribute was below threshold at some point"
(( rc & 64 )) && echo "WARNING: /dev/sda error log contains records"

# NVMe: ask for JSON and compare values, including the drive's
# own spare threshold rather than a number you picked.
nvme smart-log /dev/nvme0n1 -o json | jq -e '
  .critical_warning == 0 and .avail_spare > .spare_thresh and .percent_used < 90
' >/dev/null || echo "CRITICAL: NVMe health check failed on /dev/nvme0n1"

Set up monitoring with smartd, the daemon that ships in the same smartmontools package you already installed. It runs the self-tests on a schedule, watches temperature deltas, and mails on a state change — none of which a nightly cron one-liner does:

# /etc/smartd.conf
DEVICESCAN -a -o on -S on -n standby,q \
           -s (S/../.././02|L/../../6/03) \
           -W 4,45,55 -m root -M exec /usr/share/smartmontools/smartd-runner

sudo systemctl enable --now smartd

-s (S/../.././02|L/../../6/03) schedules a short self-test daily at 02:00 and a long one on Saturdays at 03:00. -W 4,45,55 warns on a 4°C jump and on absolute temperatures of 45°C and 55°C. For NVMe, run the jq check above from a systemd timer and route its output to the same alerting path as everything else.