CPU saturation — top, mpstat, softirq vs hardirq vs userspace, Linux scheduler

A “router CPU at 100%” report is not a fact. It is a starting question. Three different things can saturate a VyOS 1.5 LTS router’s CPU and the fix for each is different: the FRRouting zebra daemon consuming userspace cycles because of a BGP update burst; the kernel’s softirq handler consuming cycles because the NIC is feeding packets faster than the routing engine can process them; or a single NIC interrupt pinned to a single core that is being overrun. The first operator to look at the report and guess which one it is will reach the wrong fix; the operator who runs mpstat -P ALL 1 and pidstat -t 1 first will reach the right fix in five minutes.

This lesson is the diagnostic reference for Part L: the three saturation sources, the evidence that distinguishes them, and the production discipline that prevents a CPU investigation from becoming a CPU outage.

The three saturation sources

A Linux CPU can spend cycles in three classes of work, and a VyOS operator must keep them separate:

flowchart LR
  subgraph US["Userspace (FRR, ssh, vtysh)"]
    Z["zebra<br/>(RIB → FIB)"]
    B["bgpd / ospfd"]
    S["ssh, snmp, telemetry"]
  end
  subgraph SI["Softirq (kernel NAPI)"]
    K["ksoftirqd/N<br/>packet processing"]
    X["net_rx_action"]
  end
  subgraph HI["Hardirq (device)"]
    N["eth0 IRQ"]
    I["eth1 IRQ"]
  end
  HI --> SI
  SI --> US
  US --> SI

Userspace CPU is where the FRRouting daemons run. zebra consumes userspace CPU when it processes route updates from BGP, OSPF, or static configuration. bgpd and ospfd consume userspace CPU when they receive and validate routing-protocol packets. SSH, telemetry exporters, the SNMP agent, and the vyos-configd daemon also run in userspace. When top shows a python3, bgpd, or zebra process at the top, the saturation is userspace.

Softirq CPU is where the kernel processes packets that have been DMA’d from the NIC into the kernel ring buffer but have not yet been handed to a userspace consumer. The kernel runs net_rx_action in softirq context; for high-rate flows this becomes the dominant CPU consumer. When top shows ksoftirqd/N (one per CPU) at the top, the saturation is in the kernel packet path, not in FRR.

Hardirq CPU is where the NIC’s interrupt service routine runs. The IRQ fires when the NIC has packets in its ring buffer; the ISR does the absolute minimum (DPDK-style drivers do even less). Hardirq CPU is usually a small fraction of total CPU; if it is a large fraction, the IRQ is being serviced too often or too long, which usually means the softirq pass is failing to drain the ring.

The first three commands

When the report is “CPU at 100%”, the operator runs three commands before any configuration change:

top -bn1 | head -30
mpstat -P ALL 1 5
pidstat -t -p $(pgrep -d, zebra) 1 5

top shows the process view. mpstat shows the per-CPU breakdown of userspace / softirq / hardirq / iowait / idle. pidstat -t shows the per-thread breakdown of a specific daemon (here zebra), so the operator can see whether zebra is consuming CPU because of one specific thread or because of all threads.

The mpstat output is the canonical evidence:

$ mpstat -P ALL 1 5
Linux 6.6.32-vyos (router1)  08/15/26  _x86_64_  (4 CPU)

13:42:00  CPU    %usr   %nice   %sys  %soft  %steal  %idle   %wait
13:42:01  all    8.50    0.00   1.75  72.50    0.00  17.25    0.00
13:42:01    0    4.00    0.00   1.00  92.00    0.00   3.00    0.00
13:42:01    1   10.00    0.00   2.00  78.00    0.00  10.00    0.00
13:42:01    2   9.50    0.00   1.50  65.00    0.00  23.50    0.00
13:42:01    3   10.50    0.00   2.50  55.00    0.00  31.50    0.00

The %soft column tells the operator where the saturation is. In the example above, all four cores are spending 55% to 92% of their time in softirq. This is packet-processing saturation, not userspace saturation. The fix is not in FRR; it is in the NIC’s interrupt distribution and ring buffer sizing.

A different pattern tells a different story:

$ mpstat -P ALL 1 5
13:50:00  CPU    %usr   %nice   %sys  %soft  %steal  %idle   %wait
13:50:01  all   87.50    0.00   1.00   0.50    0.00  11.00    0.00
13:50:01    0   95.00    0.00   1.50   0.50    0.00   3.00    0.00
13:50:01    1   90.00    0.00   0.50   0.50    0.00   9.00    0.00
13:50:01    2   85.00    0.00   1.00   0.50    0.00  13.50    0.00
13:50:01    3   85.00    0.00   1.00   0.50    0.00  13.50    0.00

Here %usr is at 85-95% across all cores. This is userspace saturation. The next step is top or pidstat to identify which process is consuming the userspace CPU.

Distinguishing the source

flowchart TD
  S["Symptom<br/>CPU 100%"]
  S --> Q1{mpstat %usr or %soft dominates?}
  Q1 -->|"%usr dominates"| U["Userspace<br/>top, pidstat"]
  Q1 -->|"%soft dominates"| K["Softirq<br/>ksoftirqd"]
  Q1 -->|"%irq dominates"| H["Hardirq<br/>/proc/irq"]
  U --> UQ{Which process?}
  UQ -->|FRR zebra| U1["Route-update burst"]
  UQ -->|bgpd| U2["BGP update burst"]
  UQ -->|ssh, snmp| U3["Management traffic"]
  K --> KQ{Single-core or all-core?}
  KQ -->|single-core| K1["Interrupt pinning<br/>see Part L-02"]
  KQ -->|all-core| K2["Total rate exceeds capacity"]
  H --> HQ{Single IRQ dominating?}
  HQ -->|yes| H1["Single-NIC overload"]
  HQ -->|no| H2["Many small IRQs"]

Userspace saturation

The diagnostic is top -bn1 | head -20 and pidstat -t -p <pid> 1. The output reveals:

• zebra at the top → route-update burst. Common during BGP convergence or OSPF LSA floods. The fix is route damping (BGP), route summarisation, or stub area design (OSPF).
• bgpd → BGP update burst. Common during peer reset, route-leak correction, or a misconfigured peer flapping. The fix is peer isolation, neighbor maximum-prefix, or neighbor route-map to bound what the peer can advertise.
• ospfd → OSPF SPF recalculation. Common during a flapping link or an area design that is too large. The fix is area range, stub areas, or link stability.
• sshd, snmpd, node_exporter → management-plane saturation. Common during an SNMP walk storm or a misconfigured telemetry exporter. The fix is rate limiting, dedicated management interface, or removing the offending exporter.
• python3 (telemetry, NetFlow, custom) → application-level loop. The fix is to find and kill the loop.

Softirq saturation

The diagnostic is top -bn1 | head -20 to see ksoftirqd/N at the top, plus cat /proc/softnet_stat (covered in Part L-03) and cat /proc/net/softnet_stat to see the per-CPU queue depth.

The two distinct softirq saturation patterns:

1. Single-core softirq saturation. One ksoftirqd/N is at 100% while the others are idle. This means the NIC’s interrupt is pinned to a single core (the default) and the softirq pass on that core cannot keep up. The fix is interrupt affinity (Part L-02) and Receive Side Scaling (RSS).

2. All-core softirq saturation. All ksoftirqd/N are saturated. This means the total packet rate exceeds the router’s capacity. The fix is not in software; it is in hardware (faster NIC, more cores, larger pipe). Software tuning (interrupt coalescing, ring buffer size) can help at the margin.

Hardirq saturation

The diagnostic is mpstat showing %irq at the top (rare; usually a small fraction) and cat /proc/interrupts to see which IRQ is firing. Modern NICs do almost no work in hardirq; if %irq is high, the driver is doing too much in hardirq, which usually indicates an IRQ storm from a misbehaving device or a very low interrupt-coalescing threshold.

The Linux scheduler and VyOS

VyOS 1.5 LTS inherits the Linux 6.6 LTS kernel CFS scheduler. Three scheduler concepts the operator must understand:

Process priority. The CFS scheduler runs the highest-priority runnable process. Linux has 40 priority levels (nice -20 to nice 19); lower nice values mean higher priority. The kernel itself runs at the highest priority (in softirq and hardirq context, preempting userspace). FRR daemons run at the default priority (nice 0); the operator can use nice and renice to de-prioritise a non-critical process.

CPU affinity. The taskset command and /proc/<pid>/cpu_allowed restrict a process to a subset of CPUs. For VyOS, the operator typically sets the FRR daemons’ affinity to a specific core set (a “control-plane core set”) and the softirq handling to a different core set (“data-plane cores”). This is the foundation of the “control-plane pinning” pattern discussed in Part L-02.

Cgroups. Linux control groups (/sys/fs/cgroup) provide hierarchical resource accounting and limiting. VyOS uses systemd-managed cgroups by default; the operator can constrain a process to a percentage of CPU via cpu.max in cgroup v2. For a misbehaving telemetry exporter that consumes 100% of one core, constraining it to 10% via cpu.max prevents it from saturating the router.

The operator’s discipline: scheduler tuning is justified only when the diagnostic shows a single source of saturation and the fix is local. A router that is saturated because it is receiving 10 Gbps of traffic that needs to be encrypted cannot be fixed by nice; it needs faster hardware or fewer packets.

Production failure modes

The CPU-saturation failure modes the operator encounters:

• BGP update burst saturates zebra. A peer sends a 100,000-route refresh; zebra consumes 100% CPU; routing-protocol convergence stalls; downstream BGP sessions flap. Fix: peer-side neighbor maximum-prefix and neighbor route-map to bound what the peer can advertise.
• OSPF LSA flood saturates ospfd. A flapping link triggers continuous SPF recalculation. Fix: link stability, stub area design, or area range summarisation.
• Single-NIC interrupt pinning. A 10 Gbps NIC’s IRQ is pinned to one core; that core’s softirq saturates; the other cores are idle. Fix: RSS and interrupt affinity (Part L-02).
• SNMP walk storm saturates the management plane. A NMS polls the router every 5 seconds for 10,000 OIDs. Fix: SNMP rate limit, dedicated management interface, or monitor bandwidth baseline.
• Telemetry exporter loop. A custom exporter crashes and restarts in a tight loop, consuming userspace CPU. Fix: systemd Restart= policy; cap restart rate; isolate to its own cgroup.
• TCPDump on the data path. A tcpdump capture on a busy interface can saturate the softirq path. Fix: SPAN port on the switch instead of tcpdump on the router.

Rollback

CPU-saturation fixes are typically small but the impact is large. The rollback discipline:

• taskset and renice — these affect a running process; they are not in the VyOS configuration tree. The rollback is to kill the affected process or reboot.
• IRQ affinity (/proc/irq/<n>/smp_affinity) — also not in the VyOS tree. Rollback by restoring the original mask or rebooting. Part L-02 covers persistent configuration via system irqbalance or udev rules.
• Systemd cgroup limits — systemctl set-property modifies the running cgroup; rollback by clearing the limit (cpu.max=) or rebooting.
• BGP / OSPF knobs — these are in the VyOS tree. Use rollback N and commit.

For all changes, use commit-confirm:

configure
# ... make the change ...
commit-confirm 5
# If the change has unintended consequences, the auto-rollback
# fires after 5 minutes and the previous configuration is restored.

Production discipline

Cross-course references

• Part L-02 (L-VyOS-Performance / interrupt affinity) covers the RSS, NUMA, and IRQ affinity primitives that this lesson assumes when softirq is the source.
• Part L-03 (L-VyOS-Performance / packet drops) covers softnet_stat and drop reasons, which are the next step after softirq saturation is identified.
• Part L-05 (L-VyOS-Performance / route churn) covers FRR-specific causes of userspace saturation (BGP bursts, OSPF floods).
• The Linux course’s V-Linux-NetConfig and XXII-Linux-NetTroubleshoot parts cover the same primitives from the host perspective (mpstat, pidstat, cgroups).
• The Observability course covers the telemetry side (Prometheus, Grafana, alerting on CPU deviation).
• The Ansible course’s XLII-Ansible-BeyondLinux covers the automation hand-off (CPU-saturation detection in automated playbooks).

Quiz