Lab: Hardware sensor monitoring and predictive replacement

This lab sets up monitoring for SMART, NVMe, IPMI, and ECC, and produces a predictive replacement plan.

Objective

By the end of this lab, you can:

• Enumerate every physical drive a host has, including the ones hidden behind a RAID controller.
• Read SMART, NVMe, IPMI and ECC health as machine-checkable values rather than as text a human skims.
• Turn those values into a dated replacement schedule.
• Explain why a green dashboard on a RAID host can be reporting on a device that does not exist.

Architecture

Four independent sensor sources on one host, each with its own access path:

Nothing here shares a data path. A host can have perfect SMART coverage and no ECC visibility at all, which is why the lab checks each one separately.

Requirements

# Tooling
sudo apt install smartmontools nvme-cli ipmitool rasdaemon jq

# Know what you are working with before you start
lsblk -d -o NAME,SIZE,ROTA,MODEL
sudo smartctl --scan-open

• Root or sudo. Every command here needs it.
• Physical hardware, or a VM that passes disks through. A fully virtualised disk has no SMART data; smartctl will report Unavailable - device lacks SMART capability and there is nothing to fix. NVMe and IPMI tasks likewise need real hardware or an emulated BMC.
• If the host has a RAID controller, the vendor CLI (storcli, ssacli, perccli) so you can confirm the physical drive count independently.
• BMC credentials for Task 3, if you are doing it. Have somewhere to put them that is not your shell history.

If a source is genuinely unavailable on your host, record that as a finding in the plan rather than skipping it silently - an unmonitored source is exactly what this lab exists to surface.

Scenario

A drive failed last quarter on a host that was, according to the monitoring, healthy. The postmortem found that the host has a RAID controller, that the monitoring was polling the virtual disk it presents, and that the virtual disk had been reporting PASSED throughout - because a virtual disk always does. The physical drive behind it had been logging reallocated sectors for two months.

You have been asked to build monitoring that could not fail that way again, and to produce a replacement schedule from it.

Tasks

Task 1: SMART monitoring

# Install
sudo apt install smartmontools

# Discover what smartctl can actually address. Do this BEFORE
# globbing /dev/sd?, because on a host with a RAID controller
# that glob finds virtual disks, not drives.
sudo smartctl --scan-open

# Bare or HBA-attached drives
for disk in /dev/sd?; do
    sudo smartctl -H -A "$disk"
done

If --scan-open reported a -d type, the drives are behind a controller and every command needs the passthrough:

# Broadcom MegaRAID / Dell PERC: how many physical drives?
sudo storcli /c0 show

# One call per member drive
for n in 0 1 2 3; do
    sudo smartctl -H -A -d "megaraid,$n" /dev/bus/0
done

# HPE Smart Array
sudo smartctl -H -A -d cciss,0 /dev/sg0

Record for each disk:

• Model, serial.
• Reallocated_Sector_Ct, Current_Pending_Sector.
• Temperature.
• The exit status of smartctl -H -A, which is the bitmask you will actually alert on: bit 3 (8) = health FAILED, bit 5 (32) = an attribute has been below threshold, bit 6 (64) = the error log has records.

Task 2: NVMe monitoring

# Install
sudo apt install nvme-cli

# Check NVMe disks
sudo nvme smart-log /dev/nvme0n1

Record for each NVMe:

• Model, serial.
• critical_warning, temperature, percentage_used.

Task 3: IPMI / BMC monitoring

# One-time: root-only credential file. -P would put the BMC
# password in /proc/<pid>/cmdline, readable by every local
# user, and in this shell's history.
sudo install -d -m 0700 /etc/ipmi
sudo install -m 0600 /dev/null /etc/ipmi/bmc.pw
read -rs -p 'BMC password: ' BMC_PW; echo
printf '%s' "$BMC_PW" | sudo tee /etc/ipmi/bmc.pw >/dev/null
unset BMC_PW

# Read sensors
sudo ipmitool -I lanplus -H <bmc-ip> -U admin \
    -f /etc/ipmi/bmc.pw sensor

# Power status
sudo ipmitool -I lanplus -H <bmc-ip> -U admin \
    -f /etc/ipmi/bmc.pw chassis power status

Use a read-only BMC account for health polling if the BMC supports one. A monitoring job needs sensor data, not power control.

Task 4: ECC monitoring

sudo apt install rasdaemon
sudo ras-mc-ctl --error-count

Task 5: Predictive replacement plan

HARDWARE PREDICTIVE REPLACEMENT PLAN
===================================
Host: <host>
Date: 2026-08-09

Disks:
- /dev/sda: 0 reallocated, 38°C. No action.
- /dev/sdb: 5 reallocated, 42°C. Schedule replacement.
- /dev/nvme0n1: 30% used, 38°C. 3 years life. No action.

Memory:
- 0 ECC errors this month. No action.

BMC:
- IPMI 2.0, RAKP. Healthy. No action.

Fans:
- All within normal RPM. No action.

Replacement schedule:
- 2026-09-01: replace /dev/sdb (5 reallocated sectors)
- 2027-01-01: replace /dev/nvme0n1 if percentage_used > 80

Next review: 2026-11-09

Record the BMC account name, never the password.

Validation

Each of these must produce the stated result before the lab is done.

# 1. SMART reaches every PHYSICAL drive, not just the virtual disk.
sudo smartctl --scan-open | wc -l
# Must equal the physical drive count from `storcli /c0 show` (or
# `ssacli ctrl all show config`, or `lsblk -d` on a host with no
# RAID controller). If it is smaller, your monitoring is blind:
# go back to Task 1 and add the -d passthrough.

# 2. The health verdict is readable as an exit status, not as text.
sudo smartctl -H -A /dev/sda >/dev/null; echo "exit=$?"
# Must print a number. 0 means healthy; any bit set is a finding
# you must be able to name.

# 3. The NVMe check is a comparison, not a grep.
sudo nvme smart-log /dev/nvme0n1 -o json | jq -e '
  .critical_warning == 0 and .avail_spare > .spare_thresh
' >/dev/null && echo "NVMe OK" || echo "NVMe FAILED the check"
# Must print one of the two lines. If jq errors, the field names
# differ on your nvme-cli version - read the JSON and adjust.

# 4. ECC counters are being collected.
sudo ras-mc-ctl --error-count
# Must print a table, not "No DIMM found". An empty table means
# rasdaemon is not running or the platform exposes no EDAC.

The plan in Task 5 is the deliverable, and it is only valid if step 1 passed. A replacement schedule built from a virtual disk’s SMART data is a schedule for a device that cannot fail the way you are predicting.

Expected outcome

The deliverable is a plan in which every physical device is named. A plan with three entries on a host with eight drives is the failure this lab exists to prevent, even though every check in it passed.

Troubleshooting

Cleanup

Task 3 wrote a credential to disk. Remove it if this host is not going to keep polling the BMC:

sudo rm -f /etc/ipmi/bmc.pw
sudo rmdir --ignore-fail-on-non-empty /etc/ipmi

Then confirm nothing leaked into your history:

grep -n -- '-P ' ~/.bash_history | grep ipmitool
# Must print nothing

If it does print something, rotate the password on the BMC. A password that has been written to a history file is a password that has been disclosed.

Leave the monitoring itself in place - that was the point of the lab. If this was a scratch host, stop and disable what you started:

sudo systemctl disable --now smartd rasdaemon

What you learned

• Enumeration comes before monitoring. smartctl --scan-open is the step that decides whether the rest of your monitoring is looking at real devices, and a RAID controller silently makes the naive version wrong.
• Health checks belong in exit statuses and field comparisons, not in greps against human-readable text. PASSED in the output of a virtual disk means nothing at all.
• The four sensor sources are independent. Coverage of one tells you nothing about the others, so each needs its own check and its own alert.
• A replacement plan is only as good as its device list. Dates and thresholds are the easy part; naming every device is the part that fails.