System metrics — CPU, memory, disk, temperature, and the prometheus-vyos-exporter

A “router is unhealthy” report is not a system issue until the operator has CPU per-core, memory, disk, and sensor evidence. The Linux kernel exposes the canonical system metrics in /proc/stat, /proc/meminfo, /sys/class/hwmon/, and the per-device sysfs nodes. The operator who cannot read those metrics in five minutes will spend an hour guessing whether the router is overloaded, the disk is full, or the temperature is too high.

This lesson is the production reference for system metrics on VyOS 1.5 LTS: the four transport options (prometheus-vyos-exporter, node_exporter, gNMI, SNMP HOST-RESOURCES-MIB), what each one gives the operator, what each one costs, and the validation discipline that prevents the system telemetry layer from becoming the next source of incidents.

The four transports

flowchart LR
  subgraph ROUTER["VyOS 1.5 router (Linux kernel)"]
    K["/proc/stat, /proc/meminfo, /sys/class/hwmon/"]
    F["FRR zebra, FRR bgpd, FRR ospfd"]
  end
  subgraph COLLECT["Collection plane"]
    VYOS["prometheus-vyos-exporter<br/>(VyOS-specific)"]
    NE["node_exporter<br/>(standard Linux)"]
    GNMI["gNMI telemetry<br/>subscription"]
    SNMP["SNMP HOST-RESOURCES-MIB<br/>(RFC 2790)"]
  end
  subgraph CONSUME["Consumption plane"]
    TS["Time-series DB<br/>(Prometheus)"]
    NMS["NMS / Grafana"]
  end
  K --> VYOS
  K --> NE
  F --> VYOS
  K --> GNMI
  K --> SNMP
  VYOS --> TS
  NE --> TS
  GNMI --> TS
  SNMP --> NMS

prometheus-vyos-exporter is the canonical VyOS-specific exporter. It exposes FRR state (BGP sessions, OSPF neighbours, route counts), VRRP state, and the VyOS configuration tree. The cost: the exporter is community-maintained (not part of the base VyOS image), and the operator must install it as a separate package.

node_exporter is the canonical standard Linux exporter. It exposes CPU per-core, memory, swap, disk usage, filesystem usage, network interface counters, and the per-mount-point filesystem metrics. The cost: the exporter does not expose VyOS-specific state (FRR state, VRRP state, route counts); the operator must use prometheus-vyos-exporter for those.

gNMI telemetry is the streaming-evidence alternative. The operator subscribes to the system state YANG model and receives state updates on a configurable cadence. The cost: gNMI requires the vyos-gnmi package, and the system state YANG model is still maturing.

SNMP HOST-RESOURCES-MIB (RFC 2790) is the legacy alternative. The hrSystemLoad, hrMemorySize, hrStorageSize, hrStorageUsed provide aggregate state. The cost: the MIB does not provide per-CPU or per-mount-point detail, and the data is 32-bit.

The canonical system metrics

The operator must know which system metrics to monitor. The four canonical metric categories:

flowchart TD
  subgraph SYS_METRICS["System metrics"]
    CPU["CPU per-core<br/>(user, system, iowait, softirq, idle)"]
    MEM["Memory<br/>(used, free, available, swap, cache, buffer)"]
    DISK["Disk<br/>(usage, inodes, read/write IOPS, latency)"]
    SENS["Sensors<br/>(temperature, fan speed, voltage, power)"]
  end
  CPU --> MEM
  MEM --> DISK
  DISK --> SENS

CPU per-core is the canonical evidence of CPU saturation. The operator monitors node_cpu_seconds_total (Prometheus) or mpstat -P ALL 1 (Linux) to see the per-core CPU breakdown. A single core at 100% indicates interrupt pinning (Part L-02); all cores at 100% indicates the router is overloaded.

Memory is the canonical evidence of memory pressure. The operator monitors node_memory_MemTotal_bytes, node_memory_MemAvailable_bytes, node_memory_SwapTotal_bytes, node_memory_SwapFree_bytes. A MemAvailable of less than 10% of MemTotal indicates memory pressure; a swap usage of more than 10% indicates the system is swapping.

Disk is the canonical evidence of disk usage. The operator monitors node_filesystem_size_bytes, node_filesystem_avail_bytes, node_filesystem_files (inodes), node_disk_reads_completed_total, node_disk_writes_completed_total. A filesystem usage of more than 90% indicates the disk is full; an inode usage of more than 90% indicates the filesystem is out of inodes.

Sensors is the canonical evidence of hardware health. The operator monitors node_hwmon_temp_celsius (temperature), node_hwmon_fan_speed_rpm (fan speed), node_hwmon_in_volts (voltage), node_hwmon_power_watts (power). A temperature of more than 80°C indicates the router is overheating; a fan speed of 0 RPM indicates the fan has failed.

Step 1 — Configure prometheus-vyos-exporter

The operator installs the prometheus-vyos-exporter package and configures the exporter:

configure
set service prometheus vyos-exporter listen-address '10.0.0.1'
set service prometheus vyos-exporter port '9102'
commit

The exporter exposes the following metrics:

$ curl -s http://10.0.0.1:9102/metrics | grep vyos_
vyos_bgp_session_up{peer="10.0.1.1",remote_as="65001"} 1
vyos_bgp_session_up{peer="10.0.1.2",remote_as="65002"} 1
vyos_ospf_neighbor_state{neighbor="10.0.1.1",area="0.0.0.0"} 8
vyos_vrrp_state{group="default",interface="eth0"} 2
vyos_route_count{address_family="ipv4",vrf="default"} 12345
vyos_config_applied{timestamp="2026-08-15T14:00:00Z"} 1

The vyos_bgp_session_up metric is a gauge (1 = up, 0 = down). The vyos_route_count metric is a gauge. The vyos_config_applied metric is a timestamp that the operator can use to detect configuration changes.

Step 2 — Configure node_exporter

The operator installs the node_exporter package and configures the exporter:

configure
set service prometheus node-exporter listen-address '10.0.0.1'
set service prometheus node-exporter port '9100'
set service prometheus node-exporter collectors 'network,filesystem,disk'
commit

The exporter exposes the standard Linux metrics:

$ curl -s http://10.0.0.1:9100/metrics | grep node_cpu_
node_cpu_seconds_total{cpu="0",mode="user"} 12345.67
node_cpu_seconds_total{cpu="0",mode="system"} 1234.56
node_cpu_seconds_total{cpu="0",mode="idle"} 987654.32
node_cpu_seconds_total{cpu="1",mode="user"} 12340.67
node_cpu_seconds_total{cpu="1",mode="system"} 1230.56
node_cpu_seconds_total{cpu="1",mode="idle"} 987000.32

The node_cpu_seconds_total metric is a counter (the operator computes the rate via rate(node_cpu_seconds_total[5m])). The operator can also use the node_cpu gauge, which is the same data but as a gauge.

Step 3 — Stream system state via gNMI

The operator enables the gNMI server and subscribes to the system state YANG model:

configure
set service gnmi listen-address '10.0.0.1'
set service gnmi port '57400'
set service gnmi authentication username 'gnmi-user'
set service gnmi authentication password 'GNm1-2026-Q3'
commit

The operator then uses a gNMI client to subscribe to the system state:

$ gnmic -a 10.0.0.1:57400 -u gnmi-user -p 'GNm1-2026-Q3' \
  subscribe --path '/system' \
  --stream-mode sample --sample-interval 30s

The stream produces system state updates every 30 seconds. The operator feeds the stream into a time-series database. The cost: gNMI requires the vyos-gnmi package, and the system state YANG model is still maturing.

Step 4 — Configure SNMP HOST-RESOURCES-MIB

The operator enables SNMPv3 and the HOST-RESOURCES-MIB on the router:

configure
set service snmp v3 engineid '0x80004f7e9c3a1b2c'
set service snmp v3 group default mode 'ro'
set service snmp v3 view full oid '1' include
set service snmp v3 user monitoring-auth group default auth plaintext-key 'AuthPass-2026-Q3'
set service snmp v3 user monitoring-auth group default auth type 'sha'
set service snmp v3 user monitoring-auth group default privacy plaintext-key 'PrivPass-2026-Q3'
set service snmp v3 user monitoring-auth group default privacy type 'aes'
commit

The operator then queries the HOST-RESOURCES-MIB:

$ snmpwalk -v3 -u monitoring-auth -l authPriv -a SHA -A 'AuthPass-2026-Q3' -x AES -X 'PrivPass-2026-Q3' 10.0.0.1 hrSystemLoad
HOST-RESOURCES-MIB::hrSystemLoad.0 = INTEGER: 5

The hrSystemLoad metric is the load average (not the CPU utilisation). The operator uses the load average as a quick check but should correlate it with node_cpu_seconds_total for the actual CPU utilisation.

Validation discipline

Every system telemetry metric must be validated before it becomes an alert. The validation discipline:

1. Confirm the exporter is running. The operator runs systemctl status prometheus-vyos-exporter and systemctl status node_exporter and confirms the daemons are active.
2. Confirm the metrics are present. The operator curls the exporter’s /metrics endpoint and confirms the metrics are returned.
3. Confirm the metrics are changing. The operator curls the metrics twice, 5 seconds apart, and confirms the counters have incremented.
4. Confirm the rate is sensible. The operator computes the rate (CPU, memory, disk) and compares it to the expected rate.
5. Confirm the alerts fire under the expected condition. The operator tests the alert by inducing the condition (e.g., running a CPU stress test) and confirming the NMS alerts.

Production failure modes

The system telemetry failure modes the operator encounters:

• Disk full. The router’s log partition is full; the keepalived daemon cannot write its log; the FRR daemon cannot write its operational state. Fix: configure log rotation (/etc/logrotate.d/vyos); alert on disk usage > 80%.
• Out of memory. The router is running out of memory; the kernel’s OOM killer is killing the FRR daemon. Fix: investigate the memory leak; upgrade the router’s RAM; cap the FRR daemon’s memory usage via cgroup.
• Temperature too high. The router is overheating; the kernel’s thermal throttling is reducing the CPU frequency. Fix: check the cooling; replace the fan; reduce the CPU load.
• CPU saturation. The router is overloaded; the FRR daemon is consuming 100% CPU. Fix: investigate the FRR daemon’s load (BGP update burst, OSPF SPF storm); add cores; reduce the routing daemon’s workload.
• Filesystem out of inodes. The router’s filesystem is out of inodes; new files cannot be created. Fix: investigate the inodes (small files, log files, temporary files); clean up the filesystem.
• Sensor failure. The router’s temperature sensor is failing; the kernel is reporting a temperature of 0°C or -1°C. Fix: replace the sensor; configure the alert to ignore the sensor’s failure mode.

Rollback

System telemetry changes are typically configuration-only, but the impact can be cross-cutting. The rollback discipline:

• prometheus-vyos-exporter, node_exporter, gNMI, SNMP — all are in the VyOS tree or in the operator’s filesystem. The rollback is rollback N and commit (for the tree changes) or rm (for the script files).
• The exporter binary — installed outside the VyOS tree. The rollback is to uninstall the package or kill the process.
• Cron entries — the script is in /usr/local/bin and the cron is in /etc/cron.d/. The rollback is to remove the script and the cron entry.

For every change, use commit-confirm:

configure
# ... make the change ...
commit-confirm 5
# If the change has unintended consequences, the auto-rollback
# fires after 5 minutes and the previous configuration is restored.

Production discipline

Cross-course references

• Part L (vyos-l-01-cpu-saturation) covers the CPU saturation diagnosis that the system metrics layer supports.
• Part III (vyos-iii-01-linux-base) covers the Linux kernel primitives that the system metrics layer exposes.
• The Observability course covers the consumer side: Prometheus, Grafana, alerting on system state deviation.
• The Ansible course’s XLII-Ansible-BeyondLinux covers the automation hand-off (rolling out the exporter to a fleet via a single playbook).
• The Linux course’s V-Linux-NetConfig covers the underlying kernel primitives (/proc/stat, /proc/meminfo) at the host level.

Quiz