Thermal and ECC monitoring - the memory and CPU health

Thermal and ECC errors are early signs of hardware failure. Monitoring them turns “the host crashed” from a sudden event into a managed replacement.

Thermal monitoring

CPU and system temperature:

• CPU > 70°C: warning. Throttling may occur.
• CPU > 85°C: critical. Performance impacted.
• System > 50°C: warning. Cooling is marginal.
• System > 60°C: critical. Disk failures likely.

sensors

Track temperature trends over months. A gradual rise suggests dying fans or dried thermal paste.

ECC memory monitoring

ECC memory detects and corrects single-bit errors. The correctable error count is the early warning:

# Install rasdaemon (Reliability, Availability, Serviceability daemon)
sudo apt install rasdaemon

# REQUIRED: nothing is recorded until the daemon is running.
# ras-mc-ctl reads rasdaemon's database, not the hardware.
sudo systemctl enable --now rasdaemon
systemctl is-active rasdaemon        # must print 'active'

# Confirm EDAC is loaded and the DIMMs are enumerated
sudo ras-mc-ctl --status

# Per-DIMM counters, separated into corrected and uncorrected
sudo ras-mc-ctl --error-count

# Detail, including the DIMM label and the failing address
sudo ras-mc-ctl --errors

Correctable errors (CE)

The memory controller detected a single-bit error and fixed it. The machine is unharmed and the workload never noticed.

The useful signal here is where the errors are, not how many there are. A raw monthly count is a weak predictor: single events scattered across many different addresses are genuinely transient, while a handful of errors that keep landing on the same address, rank or bank is a DIMM that is starting to fail — that is the pattern that precedes an uncorrectable error.

• Scattered single CEs at different addresses: log it, move on.
• Any CE recurring at the same address, rank or bank: schedule DIMM replacement, whatever the total count is.
• A rising CE rate on one DIMM label over successive weeks: replace at the next maintenance window.

# The address and rank are what you correlate on
sudo ras-mc-ctl --errors | grep -i 'memory error'

Track the trend per DIMM label. “Twelve errors this month” is not actionable; “twelve errors, all on DIMM_A1 at the same rank” is.

Uncorrectable errors (UE)

A UE is a categorically different event, not a bigger CE. The error was detected and not corrected. The kernel either kills the process that touched the memory with SIGBUS or panics the machine outright; on some platforms the page or the DIMM is offlined.

• One UE is enough. Replace the DIMM. There is no trend to wait for and no threshold to cross.
• Correlate the event with the DIMM label before you pull hardware — the kernel log gives you the address, ras-mc-ctl maps it to a slot.

sudo journalctl -k --grep='Hardware Error|EDAC|mce'
sudo ras-mc-ctl --errors

MCE and kernel errors

MCE (Machine Check Exception) is the CPU’s way of reporting hardware errors:

sudo journalctl -k --grep='Hardware Error|EDAC|mce'
sudo ras-mc-ctl --summary

MCE events indicate hardware errors:

• Memory errors (ECC).
• CPU errors.
• Bus errors.
• Cache errors.

A non-zero MCE count means the hardware is reporting errors. Investigate immediately.

Predictive replacement

A failing component often shows warning signs weeks before it dies:

• Failing fan: noise, RPM fluctuations, temperature rise.
• Failing disk: SMART warnings, I/O errors.
• Failing DIMM: correctable ECC errors recurring at one address or rank, MCE events. A single uncorrectable error is not a warning sign — it is the failure.
• Failing CPU: MCE events, thermal warnings.

For each:

• Monitor weekly.
• Replace before failure.
• Document the replacement.

Knowledge check