Device health — SMART, NVMe logs, and what actually predicts failure

Why this matters in production

Devices rarely fail without warning; they fail without anyone looking. The attributes that predict failure are well established, they are readable in seconds, and Ceph can collect them for you. The gap is almost always operational rather than technical.

The attributes that matter on spinning disks

smartctl -a /dev/sdb

Predictive, in descending order of usefulness:

• Reallocated_Sector_Ct — sectors the drive has remapped after failing to read or write them. Non-zero is a signal; rising over weeks is a replacement.
• Current_Pending_Sector — sectors that failed a read and are awaiting reallocation. This is the strongest single predictor. Non-zero means the drive has data it could not read.
• Offline_Uncorrectable — sectors that could not be recovered at all.
• Reported_Uncorrect — errors surfaced to the host.

Informational rather than predictive: Power_On_Hours, Temperature_Celsius (unless extreme), Load_Cycle_Count, Raw_Read_Error_Rate — the last of which is vendor-encoded and routinely misread as alarming when it is normal.

The attributes that matter on NVMe

nvme smart-log /dev/nvme0n1

• critical_warning — a bitfield; anything non-zero deserves immediate attention.
• percentage_used — consumed endurance, as covered in the endurance lesson.
• media_errors — uncorrectable errors. Non-zero is significant.
• num_err_log_entries — read the log with nvme error-log when this rises.
• temperature and the thermal throttling counters — a device that throttles produces latency spikes that look like Ceph problems.

Letting Ceph do the collecting

The device health module is built in and collects SMART data on a schedule, storing it with the device rather than the host:

DEVID=devid
ceph device monitoring on
ceph device ls
ceph device get-health-metrics ${DEVID}
ceph device ls-by-daemon osd.12
ceph config set global device_failure_prediction_mode local

Two things make this better than a script. It keeps history, so “rising over weeks” becomes answerable. And it tracks devices by their identity rather than by /dev/sdX, which is essential because kernel device names are not stable across reboots — the single most common reason a hand-rolled SMART script reports on the wrong drive.

When prediction is enabled and a device is predicted to fail, Ceph raises a health warning naming the device and the OSD.

The decision rule

Replace, do not monitor, when any of these is true:

• Current pending sectors are non-zero.
• Reallocated sectors are rising month over month.
• NVMe critical_warning is non-zero or media_errors is rising.
• The device is a persistent outlier in ceph osd perf.
• percentage_used is above 90.

Everything else is a watch item. The list is short deliberately: a long watch list becomes a spreadsheet nobody reads, and the four attributes above catch most of what matters.

Quiz

Production discipline

Enable ceph device monitoring on so history is collected against stable device identities rather than kernel names. Alert on the four predictive attributes only — pending sectors, rising reallocated sectors, NVMe media errors or critical warnings, and consumed endurance — and resist adding informational attributes that turn the alert into noise. Treat a persistent ceph osd perf outlier as equivalent evidence even when health is OK, and prefer proactive replacement in a chosen window over the same backfill during an incident.

Cross-course references

• Ceph: Part LVII (Replacing Failed OSDs) for the replacement procedure itself.
• Ceph: Part LXII (Inconsistent PGs) for what unreadable sectors produce at scrub time.
• Linux: Part LXIX (Hardware) for smartctl and the host-side view.