Automating device health collection

Why this matters in production

Health data that requires someone to run a command is health data that is read after a failure rather than before it.

Ceph’s device health module

ceph mgr module enable devicehealth
ceph config set mgr mgr/devicehealth/enable_monitoring true
ceph config set mgr mgr/devicehealth/scrape_frequency 86400
ceph config set mgr mgr/devicehealth/warn_threshold 604800
ceph config set mgr mgr/devicehealth/self_heal true

self_heal is worth considering carefully: it drains an OSD automatically, which is right on a cluster with headroom and wrong on one without.

HOST=stor-04
DEVICE_ID=12
ceph device ls
ceph device ls-by-host ${HOST}
ceph device get-health-metrics ${DEVICE_ID}

Exporting to monitoring

ceph mgr module enable prometheus
curl -s localhost:9283/metrics | grep -i 'device_health\|smart'

Ceph exports a device health metric; for full SMART attribute coverage a node exporter with the smartmon collector is the usual addition:

# textfile collector approach
cat > /usr/local/bin/smart-export.sh <<'EOF'
#!/bin/bash
for d in /dev/sd?; do
  smartctl -A "$d" | awk -v dev="${d##*/}" '
    $1 ~ /^(5|187|188|197|198|199)$/ {
      printf "smart_attr{device="%s",id="%s",name="%s"} %s
", dev, $1, $2, $10
    }'
done
EOF
chmod +x /usr/local/bin/smart-export.sh

*/10 * * * * /usr/local/bin/smart-export.sh > /var/lib/node_exporter/smart.prom.tmp &&              mv /var/lib/node_exporter/smart.prom.tmp /var/lib/node_exporter/smart.prom

The atomic move matters: a partially written file is scraped as truncated metrics.

Alerting on what matters

- alert: DiskPendingSectors
  expr: smart_attr{id="197"} > 0
  for: 10m
  labels: { severity: ticket }
  annotations:
    summary: "{{ $labels.device }} has pending sectors — schedule replacement"

- alert: DiskReallocatedIncreasing
  expr: increase(smart_attr{id="5"}[24h]) > 0
  for: 1h
  labels: { severity: ticket }
  annotations:
    summary: "{{ $labels.device }} reallocated sectors increasing"

- alert: CephDeviceHealthPredictedFailure
  expr: ceph_health_detail{name="DEVICE_HEALTH"} == 1
  for: 1h
  labels: { severity: ticket }

Verifying collection works

# is data actually arriving?
DEVICE_ID=12
ceph device ls | head
ceph device get-health-metrics ${DEVICE_ID} | head -20

# how recent?
ceph device get-health-metrics ${DEVICE_ID} | python3 -c '
import sys,json; d=json.load(sys.stdin)
print("most recent sample:", max(d.keys()) if d else "NONE")'

An empty result is common and usually means the container lacks the privileges to read SMART, or the HBA presents the devices in a way smartctl needs a -d flag for:

smartctl -a -d megaraid,0 /dev/sda
smartctl -a -d sat /dev/sdb

Quiz

Production discipline

Verify device health data is actually arriving per hardware generation rather than trusting that collection is enabled — RAID controllers and container privileges both produce silent empty records. Alert on devices with no recent sample so coverage gaps are visible rather than assumed.

Cross-course references

• Kubernetes: verifying an exporter produces data, not just that it is deployed
• Linux: smartd configuration behind RAID controllers has the identical pitfall