Hardware inventory and the SEL - turning an alert into a part number

The previous lessons in this part produce signals: a SMART attribute is non-zero, a DIMM is accumulating correctable errors, a virtual disk is degraded. None of them are actionable on their own. Somebody has to walk to a rack, open a chassis, and pull one specific part — and getting that wrong means an outage taken to replace a healthy component while the failing one keeps running.

This lesson is the translation layer: from a signal to a slot, a bay, and a serial number.

Memory: from an EDAC label to a physical slot

ras-mc-ctl --errors reports a DIMM label such as DIMM_A1. That label comes from the memory controller’s view, and it is usually — but not always — the silkscreen label on the board. dmidecode gives you the firmware’s view, which is the one the vendor’s service manual uses.

$ sudo dmidecode -t memory

Handle 0x1100, DMI type 17, 92 bytes
Memory Device
      Array Handle: 0x1000
      Total Width: 72 bits
      Data Width: 64 bits
      Size: 32 GB
      Form Factor: DIMM
      Locator: DIMM_A1
      Bank Locator: NODE 1
      Type: DDR4
      Speed: 3200 MT/s
      Manufacturer: ACME
      Serial Number: 1A2B3C4D
      Part Number: ACM32G4RD3200
      Rank: 2
      Configured Memory Speed: 2933 MT/s

Handle 0x1101, DMI type 17, 92 bytes
Memory Device
      Size: No Module Installed
      Locator: DIMM_A2
      Bank Locator: NODE 1

Two things that output settles which nothing else does.

Whether the slot you are about to open is even populated. A failure report naming DIMM_B2 on a host where dmidecode says No Module Installed for that slot means the label mapping is off, not that a phantom DIMM is failing.

What to order. Part number and configured speed together. Ordering a module that matches the part number but not the population rules produces a host that either refuses to POST or downclocks every other DIMM in the channel.

The rest of the platform identity:

sudo dmidecode -t system      # manufacturer, product name, serial, UUID
sudo dmidecode -t bios        # BIOS vendor, version, release date
sudo dmidecode -t processor   # socket, model, core count
sudo dmidecode --list-types   # the keywords this build accepts

dmidecode -t system is where the chassis service tag lives. That string is what the vendor’s support portal wants and what your asset record should key on.

Storage: from a device node to a drive bay

/dev/sdc is not a location. It is a name the kernel assigned in whatever order the controllers enumerated, and it can change across a reboot.

# Serial and model per block device - the RMA fields
lsblk -o NAME,MODEL,SERIAL,SIZE,ROTA,TYPE

# The stable path, which encodes the controller and slot
ls -l /dev/disk/by-path/

# NVMe carries its own identity
sudo nvme list

/dev/disk/by-path/ is the useful one for physical location: a path like pci-0000:18:00.0-scsi-0:0:5:0 names the controller and the target, which maps onto an enclosure slot far more reliably than sdc does.

For drives behind a RAID controller, the controller tool is the authority, because the OS only sees the virtual disk:

# Broadcom / Dell - enclosure and slot for every physical drive
sudo storcli /c0/eall/sall show all

That output gives /c0/e252/s5 style identifiers, and the enclosure-slot pair is what the drive caddy is labelled with.

The System Event Log

The SEL is the BMC’s own record of hardware events. It matters because it records things the operating system structurally cannot:

• Events that happened while the host was powered off.
• Events that happened before the OS booted — a failed PSU at POST, a memory training failure.
• Events that happened during the crash, when the kernel had already stopped writing to disk.
• Power supply, fan, chassis intrusion and voltage events that the OS is never told about.

It also survives a reinstall, because it lives in the BMC, not on the disk. A host that has been re-imaged three times still has the record of the PSU that failed last year.

$ BMC=192.0.2.50
CRED="-I lanplus -H $BMC -U admin -f /etc/ipmi/bmc.pw"

ipmitool $CRED sel info
ipmitool $CRED sel elist last 5

SEL Information
Version          : 1.5 (v1.5, v2 compliant)
Entries          : 512
Free Space       : 0 bytes
Percent Used     : 100%
Last Add Time    : 03/14/2026 02:11:47
Last Del Time    : Not Available
Overflow         : true
Supported Cmds   : 'Reserve' 'Get Alloc Info'

1fe | 03/14/2026 | 02:04:12 | Power Supply PSU2 | Failure detected | Asserted
1ff | 03/14/2026 | 02:04:13 | Power Supply PSU2 | Power Supply AC lost | Asserted
200 | 03/14/2026 | 02:09:55 | Memory DIMM_A1 | Correctable ECC | Asserted
201 | 03/14/2026 | 02:10:31 | Temperature Inlet Temp | Upper Non-critical | Asserted
202 | 03/14/2026 | 02:11:47 | Memory DIMM_A1 | Correctable ECC | Asserted

The BMC clock is not the host clock

SEL timestamps come from the BMC’s own real-time clock, which is set independently of the host and is not disciplined by whatever NTP client is running on the operating system. A BMC that has never had its clock set — or that lost it during a firmware update — timestamps events from 2001, or from the moment it last booted.

That makes correlation impossible in the exact investigation where it matters most: matching a kernel panic in the journal against the thermal event that preceded it.

# Substitute your own values before running:
BMC=192.0.2.50
CRED="-I lanplus -H $BMC -U admin -f /etc/ipmi/bmc.pw"

# What does the BMC think the time is? Compare against date -u.
ipmitool $CRED sel time get
date -u

Many BMCs can be pointed at an NTP server through their own web interface or Redfish, and on a fleet that is the right answer. Where it is not available, set it during the build and re-check it in the same drill that tests OOB access. The linux-time-skew-operational-impact lesson covers what skew does to authentication; this is the same failure in a place nobody thinks to look, and its cost is a misdiagnosis rather than an outage.

Export the inventory off the host

An inventory that only exists on the machine is unavailable exactly when the machine is. Collect it on a schedule and ship it somewhere else:

#!/usr/bin/env bash
# /usr/local/sbin/hw-inventory - run from cron, ship the output
set -euo pipefail

out="/var/backups/hw/$(hostname -s)-$(date -u +%Y%m%d).txt"
install -d -m 0750 "$(dirname "$out")"

{
  echo "=== system ==="   ; dmidecode -t system
  echo "=== bios ==="     ; dmidecode -t bios
  echo "=== memory ==="   ; dmidecode -t memory
  echo "=== storage ==="  ; lsblk -o NAME,MODEL,SERIAL,SIZE,TYPE
  echo "=== summary ==="  ; lshw -short
} > "$out"

lshw -short is the one-page overview — bus path, device class, description — and is the fastest way to answer “what NIC is in this box” without opening it.

Knowledge check