Hardware monitoring sensors - temperature, voltage, fan

Hardware sensors report temperature, voltage, fan speed, and more. A failing sensor or out-of-range reading is the canary for an impending hardware failure. This lesson covers how to read and alert on them.

Install lm-sensors

sudo apt install lm-sensors
sudo sensors-detect    # probe for sensors - see the warning below

sensors-detect walks through a series of prompts, each one asking permission to probe a class of hardware.

Read sensors

sensors

Output:

coretemp-isa-0000
Adapter: ISA adapter
Core 0:       +45.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:       +47.0°C  (high = +80.0°C, crit = +100.0°C)

nvme-pci-0100
Adapter: PCI adapter
Composite:    +38.0°C  (low  =  -5.0°C, high = +70.0°C, crit = +80.0°C)

power_supply-BAT0
Adapter: ISA adapter
in0:          +12.00 V
fan1:         1500 RPM

Reading:

• Temperature: in °C. Compare to high and crit.
• Voltage: in V. Compare to nominal.
• Fan speed: in RPM. Compare to nominal.

A sensor close to crit is overheating. Replace cooling or check airflow.

Alert on sensors

You do not need a script for this. node_exporter’s hwmon collector is enabled by default and reads /sys/class/hwmon/ directly — the same source sensors reads:

node_hwmon_temp_celsius
node_hwmon_temp_crit_celsius
node_hwmon_fan_rpm

Alert against the chip’s own critical threshold rather than a number you picked, because what is hot differs per sensor and per platform:

- alert: SensorNearCritical
  expr: node_hwmon_temp_celsius > (node_hwmon_temp_crit_celsius * 0.85)
  for: 10m

- alert: FanStopped
  expr: node_hwmon_fan_rpm == 0
  for: 2m

Confirm the collector is actually producing data before building dashboards on it — inside a container or a VM there may be no hwmon devices at all:

curl -s localhost:9100/metrics | grep '^node_hwmon_temp_celsius'

Alert thresholds — read them from the chip, do not pick them:

• Temperature: warn above the chip’s own high value, page above crit. Both are exported alongside the reading as node_hwmon_temp_max_celsius and node_hwmon_temp_crit_celsius, and sensors -j prints them as JSON. A hard-coded “70°C is hot” is wrong in both directions: the sample output above shows a CPU package whose vendor limit is 80/100°C, so 70 pages you for a healthy core, while an NVMe drive at crit = +80.0°C throttles long before an alert at 85 would fire.
• Fan at 0 RPM on a chip that reports a nominal RPM: critical, regardless of temperature. Temperature is a lagging indicator; the fan stopping is the event.
• Voltage: deviation beyond 5% of nominal (warning), 10% (critical) — voltage rails, unlike temperature, genuinely do have a platform-independent tolerance.

IPMI and BMC

For servers with IPMI/BMC:

# Install ipmitool
sudo apt install ipmitool

# One-time: keep the BMC password out of argv. -P puts it in
# /proc/<pid>/cmdline for every local user to read, and in
# shell history. -f reads it from a root-only file instead.
sudo install -d -m 0700 /etc/ipmi
sudo install -m 0600 /dev/null /etc/ipmi/bmc.pw
read -rs -p 'BMC password: ' BMC_PW; echo
printf '%s' "$BMC_PW" | sudo tee /etc/ipmi/bmc.pw >/dev/null
unset BMC_PW

# Read sensors via IPMI
ipmitool -I lanplus -H <bmc-ip> -U <user> -f /etc/ipmi/bmc.pw sensor

# Read FRU (field-replaceable unit) info
ipmitool -I lanplus -H <bmc-ip> -U <user> -f /etc/ipmi/bmc.pw fru

ipmitool also reads the password from the environment when given -E, which suits an exporter more than a file path:

IPMI_PASSWORD="$(sudo cat /etc/ipmi/bmc.pw)" \
  ipmitool -I lanplus -H "$BMC" -U monitor -E sensor

This matters more here than it looks. Sensor polling is the one IPMI call you run on a schedule, so a -P form is the one most likely to be sitting in a cron job or a monitoring exporter, exposed in ps output every scrape interval. Use -I lanplus (IPMI v2.0 with RMCP+ and RAKP), not -I lan, which is IPMI v1.5 and puts the session on the wire without encryption.

Two more rules for BMC credentials:

• Least privilege. Create a dedicated BMC user for monitoring at User or Operator privilege. Sensor and FRU reads need nothing more. Administrator grants power control, console and virtual-media boot - out of band and below the OS - on every server that shares the credential.
• Prefer an exporter. ipmi_exporter holds one authenticated session and exposes the readings as metrics, rather than spawning ipmitool with credentials on every scrape.

IPMI gives access to:

• Temperature sensors (CPU, disk, ambient).
• Fan speed.
• Voltage.
• Power consumption.
• Hardware inventory (FRU).

For cloud VMs, IPMI is replaced by the cloud provider’s hardware monitoring (AWS CloudWatch, Azure Monitor, etc.).

Common failure modes

For predictive replacement, monitor sensor trends over months. A gradual temperature rise suggests a slow failure (fan, thermal paste) before a sudden outage.

Knowledge check