CPU Metrics

A backend service responds in 4 seconds. Application panels say nothing. The host panel says node_cpu_seconds_total is at 12% user, 2% system, 0.6% iowait. “CPU is fine.” The on-call opens the saturation panel — load average 96 on a 32-core host, node_pressure_cpu_waiting_seconds_total climbing 80% of the time. The CPU was not busy; the run queue was. The investigation walks through the difference in this lesson.

CPU metrics from node_exporter come from three families: node_cpu_seconds_total (the modes), node_load1/node_load5/ node_load15 (the run queue depth), and node_pressure_* (the kernel pressure stall information). Each answers a different question. Each fails differently. The lesson is about reading them together.

What it is

node_cpu_seconds_total is a counter per (cpu, mode) pair, measured in seconds. The mode label takes one of a fixed set of values defined by /proc/stat:

# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="user"}    9.876e+03
node_cpu_seconds_total{cpu="0",mode="nice"}    0
node_cpu_seconds_total{cpu="0",mode="system"}  4.5e+03
node_cpu_seconds_total{cpu="0",mode="idle"}    1.234e+05
node_cpu_seconds_total{cpu="0",mode="iowait"}  12
node_cpu_seconds_total{cpu="0",mode="irq"}     1
node_cpu_seconds_total{cpu="0",mode="softirq"} 24
node_cpu_seconds_total{cpu="0",mode="steal"}   0
node_cpu_seconds_total{cpu="1",mode="user"}    9.901e+03
...

The modes mean:

• user — time spent in user-space code (anything outside the kernel).
• nice — user-space at a de-prioritised nice value.
• system — time spent inside the kernel on behalf of processes.
• idle — CPU had nothing to do.
• iowait — CPU was idle but waiting on block I/O. Counts as “the disk is the bottleneck”, not “the CPU is busy”.
• irq / softirq — interrupt handling. A spike here usually means a misbehaving device driver or a saturated NIC.
• steal — time the hypervisor gave to other VMs. Steal > 0 means the host is oversold.

The cardinalities are bounded: cpus * modes. A 64-core host produces 64 * 8 = 512 series. Negligible.

Why rate matters

A raw counter value is “total seconds of CPU time this mode since boot.” That is not operationally useful. What matters is the rate: “fraction of CPU time spent in this mode over the last 5 minutes.”

# Per-mode utilisation as a fraction of one core, 5m average.
rate(node_cpu_seconds_total{mode!="idle"}[5m])

# Whole-host utilisation as a fraction of all cores.
1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))

The first query returns “what fraction of one CPU is busy in non-idle work right now.” A value of 0.65 means 65% of one core. On a 32-core host that is 2% of total capacity.

The second query returns “what fraction of total capacity is busy,” expressed as a 0..1 number, by averaging the idle rates. This is the value to put on a dashboard.

Saturation: load average, run queue, PSI

“CPU at 100%” is not the same as “CPU saturated.” A 64-core host can run 64 single-threaded tasks at full utilisation without anyone waiting. A 64-core host with 512 runnable threads is saturated. The right saturation metric depends on the question:

• How many processes are runnable right now? node_load1 / node_load5 / node_load15 from /proc/loadavg. The convention: “load is OK when it is below the number of cores.” On a 32-core host, load=96 is three times the capacity.
• What fraction of time did at least one task wait to run? node_pressure_cpu_waiting_seconds_total (PSI). The kernel reports this directly. A value above 0.4 means 40% of the last 5 minutes had at least one task waiting for the CPU.
• What fraction of time did all tasks wait? node_pressure_cpu_some_total (the “some” pressure). This is even worse — it means the system as a whole stalled.

PSI is the modern signal. It catches saturation that load average misses, and it is the closest metric to “the user experienced a CPU stall.” node_exporter 1.8.x enables the pressure collector by default.

The run-queue length lives at node_schedstat_running_seconds_total (cumulative run time on the run queue) and node_schedstat_waiting_seconds_total (cumulative wait time before going onto the run queue). A ratio of waiting to running above 1 means the wait queue is growing. These are cheap and useful; they are part of the schedstat collector.

Steal time as a SaaS signal

Steal time is the mode labelled steal. It is CPU time the hypervisor wanted to give this VM but had to give to someone else because the host was oversold. On bare metal it is always zero. On Proxmox it is rare. On Amazon EC2, Google Compute Engine, and most public cloud it appears whenever the host is contended.

Operational meaning:

• steal = 0 sustained — fine.
• steal rising while user+system is moderate — the host is oversubscribed. Your VM is being throttled because a neighbour is busy.
• steal rising while idle is also falling — the VM is hitting capacity, not the host.

The detection query:

# CPU time stolen from this VM, last 5m, fraction of one core.
rate(node_cpu_seconds_total{mode="steal"}[5m])

A non-zero value is a signal to either reschedule (move to a less-contended host), resize (bigger instance), or accept the contention. It is not something the application can fix.

How to configure it

The CPU collector is enabled by default in node_exporter 1.8.x. There is no flag to tune. The configuration that matters is on the Prometheus side: how the rate is computed, how the recording rules are written, how the alerts fire.

A minimal recording rule group:

# /etc/prometheus/rules/cpu.yml
groups:
  - name: cpu
    interval: 30s
    rules:
      - record: instance:cpu_utilisation:rate5m
        expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

      - record: instance:cpu_steal:rate5m
        expr: avg by (instance) (rate(node_cpu_seconds_total{mode="steal"}[5m]))

      - record: instance:cpu_iowait:rate5m
        expr: avg by (instance) (rate(node_cpu_seconds_total{mode="iowait"}[5m]))

      - record: instance:load_per_core:rate1m
        expr: node_load5 / count by (instance) (node_cpu_seconds_total{mode="idle"})

The recording rules give stable names that dashboards and alerts can use without re-deriving the rate on every refresh. count by (instance) (node_cpu_seconds_total{mode= "idle"}) returns the number of cores (one series per idle-mode label, aggregated to one count per instance).

PromQL patterns the on-call uses

Three patterns appear repeatedly:

# 1. Top 10 hosts by CPU, right now.
topk(10, instance:cpu_utilisation:rate5m)

# 2. Hosts where CPU steal is non-zero.
instance:cpu_steal:rate5m > 0.01

# 3. Hosts where load is more than 2x core count.
instance:load_per_core:rate1m > 2

The first is for “the database is slow, which host is hot?” The second is for “is my SaaS host contended?” The third is for “is any host queued up?”

How to validate it

# READ-ONLY
# 1. The collector is enabled.
curl -sf http://localhost:9100/metrics | grep '^# HELP node_cpu_seconds_total'

# 2. The series exist for every core.
curl -sf http://localhost:9100/metrics \
  | awk -F'[="]' '/^node_cpu_seconds_total/ {print $3}' \
  | sort -u

# 3. The rate makes sense.
# In PromQL:
# rate(node_cpu_seconds_total{mode="user"}[5m])
# A healthy idle host returns ~0.02-0.05 per core.
# A busy host returns 0.6-0.9 per core.
# A CPU-saturated host returns ~1.0 per core.

# 4. Pressure stall information is exposed (if pressure collector enabled).
curl -sf http://localhost:9100/metrics | grep '^node_pressure_cpu'

Expected for a healthy 8-core host:

$ curl -sf http://localhost:9100/metrics | grep -c '^node_cpu_seconds_total'
8

$ curl -sf http://localhost:9100/metrics | grep '^node_pressure_cpu'
# HELP node_pressure_cpu_waiting_seconds_total ...
node_pressure_cpu_waiting_seconds_total 1.234e+03
# HELP node_pressure_cpu_some_total ...
node_pressure_cpu_some_total 0.876e+03

If node_pressure_cpu_* is missing, the pressure collector is disabled. It is enabled by default in 1.8.x; older versions needed --collector.pressure. Verify with node_exporter --help 2>&1 | grep pressure.

How it can fail

1. Counter reset on reboot. Symptom: rate() returns a spike at reboot time. Fix: ignore in dashboards with a reset adjustment (rate() already handles counter resets, but a long-window rate across a reboot is still noisy). Use increase() with care, or alert only after the host has been up for 10 minutes.
2. Missing the steal label. Symptom: rate(...mode= "steal"...) returns empty. Cause: bare-metal host or non-virtualised container. This is correct behaviour, not a bug. Detect by inspecting the host type before alerting on steal.
3. Per-core cardinality explosion. Symptom: Prometheus TSDB head grows. Cause: a non-CPU metric is being grouped by cpu. Fix: drop cpu from non-CPU label sets.
4. Saturated CPU but no alerts fire. Symptom: the host is slow, but node_cpu_utilisation is at 30%. Cause: the saturation metric is missing — load or pressure was not recorded. Fix: add the saturation recording rules.
5. Confusing iowait with CPU busy. Symptom: the alert fires on node_cpu_utilisation because iowait was included. iowait is not CPU work; it is “the CPU is idle waiting for the disk.” The fix is to alert on mode != "idle" but investigate whether iowait is dominant.
6. A 1-second scrape interval on a 30-day retention. Symptom: Prometheus out-of-memory. Cause: per-core series times the scrape interval. Fix: keep scrape at 15s or 30s; use recording rules for finer resolution.

Security implications

The CPU metrics themselves are low-risk — they do not reveal process names or user data. The risk surface is:

• The /metrics endpoint is open by default on 0.0.0.0:9100. Anyone with network access can read CPU utilisation, and from there infer workload shape. Bind to a private interface or use --web.config for TLS + auth.
• The processes collector (opt-in) exposes per-process CPU usage with full command line. A command line that contains tokens or query strings is a leak. Disable it unless the operational value justifies the leak.
• CPU metrics are not a write path. node_exporter reads /proc/stat; it does not write anywhere. The textfile collector is the only write path, and it is opt-in.

Performance implications

Reading /proc/stat and /proc/pressure is cheap — single milliseconds per scrape. The CPU collector adds nothing measurable to scrape duration. The cost appears elsewhere:

• The cardinality: per-core counter series. On a 96-core host with 8 modes that is 768 series. Negligible.
• The rate computation: rate() is O(samples in window). Recording rules fix the cost at evaluation time.
• Dashboard queries that aggregate across instances without recording rules. A 1000-instance fleet with avg(rate(...)) in a dashboard panel is fine; a dashboard panel with topk(20, rate(...)) across the fleet is fine too. The slow case is a panel that expands to thousands of series per host.

The schedstat collector adds a small amount of additional scrape work. Worth enabling; not worth worrying about.

Production guidance

• Three recording rules are the minimum: utilisation, steal, load-per-core. Add iowait and pressure-stall if your fleet has either pain point.
• Alert on saturation (load_per_core > 2 for 10m) before utilisation. Utilisation can be at 30% on a saturated host.
• Alert on steal (cpu_steal > 0.05 for 5m) on cloud hosts. On bare-metal hosts, do not.
• Expose the recording-rule values on a dashboard alongside the raw counter, so the on-call can compare.
• Use 5m rates for dashboards, 1m rates for alerts that care about spikes. Never alert on the raw counter.

Verification

You should now be able to answer:

• What does the mode label in node_cpu_seconds_total mean, and what does iowait actually tell you?
• Why is rate(node_cpu_seconds_total{mode="idle"}[5m]) the starting point for utilisation, and why does it need to be averaged across cores?
• What does node_pressure_cpu_waiting_seconds_total tell you that node_load5 does not?
• When does steal time appear, and what does it mean?
• On Proxmox, how does CPU measurement on the PVE host differ from CPU measurement inside a VM?

Quiz