Reading device health data

Why this matters in production

SMART output is long and most of it is noise. A handful of attributes predict failure and the rest are informational, and knowing which is which is what makes the data usable.

The attributes that matter — HDD

smartctl -a /dev/sdX

smartctl -A /dev/sdX | awk '$1 ~ /^(5|187|188|197|198|199)$/ {print $1, $2, $10}'

Current_Pending_Sector is the most directly actionable: it is a block the drive cannot read now, so the read error it will produce is already determined.

The attributes that matter — NVMe

nvme smart-log /dev/nvme0n1

nvme smart-log /dev/nvme0n1 | grep -iE 'critical_warning|media_errors|percentage_used|available_spare'

Informational rather than actionable

Alerting on attribute 1 produces constant false positives on Seagate drives, whose raw value encodes several fields.

Ceph’s own collection

DEVICE_ID=12
ceph config set mgr mgr/devicehealth/enable_monitoring true
ceph device ls
ceph device get-health-metrics ${DEVICE_ID}
ceph device predict-life-expectancy ${DEVICE_ID}
ceph health detail | grep -i DEVICE_HEALTH

This collects SMART daily across the cluster and stores it, which is more useful than running smartctl by hand because it gives a trend rather than a point reading.

DEVICE_ID=12
ceph device get-health-metrics ${DEVICE_ID} | python3 -c '
import sys,json; d=json.load(sys.stdin)
for ts, rec in sorted(d.items()):
    a = rec.get("ata_smart_attributes",{}).get("table",[])
    p = [x["raw"]["value"] for x in a if x["id"]==197]
    print(ts, "pending:", p)'

Quiz

Production discipline

Record a per-device SMART baseline at deployment and alert on any increase rather than on absolute thresholds; drives differ enough that fixed thresholds are both noisy and insensitive. Recognise UDMA_CRC_Error_Count as a link fault — replacing the drive leaves the cable or HBA in place.

Cross-course references

• Kubernetes: node problem detector distinguishes signal attributes the same way
• Linux: smartd configuration faces the identical signal-versus-noise problem