Runbook: Investigate Router CPU Saturation

“The router’s CPU is at 100%” is not a finding. It is three completely different findings that a single graph cannot tell apart, and the fix for each lives in a different subsystem.

A VyOS router spends cycles in three classes of work. Userspace is where the FRR daemons run — zebra maintaining the RIB and pushing it into the kernel FIB, bgpd and ospfd handling protocol messages — along with SSH, SNMP and any telemetry exporter. Softirq is where the kernel processes packets that the NIC has already delivered into memory: routing lookups, firewall, NAT, and the encryption pass for tunnelled traffic. Hardirq is the NIC’s own interrupt handler, which on a modern driver does almost nothing.

The classes are not interchangeable, and confusing them is expensive in a specific way: it sends you to tune the wrong subsystem, which takes an hour and occasionally makes things worse. Tuning BGP timers when the real problem is a NIC interrupt pinned to one core changes nothing except the configuration. Raising netdev_max_backlog when a peer is sending a full table changes nothing except your confidence.

So this runbook is deliberately read-only. It ends with a written attribution, not with a change, because the attribution is the part that is hard and the change is the part that has a blast radius.

When to use this runbook

• A monitoring alert on router CPU has fired.
• Convergence after a topology change is taking noticeably longer than this router’s baseline.
• SSH to the router is slow or intermittent, or new sessions are refused.
• Throughput has fallen short of what the link and the hardware should deliver, with no obvious link fault.

When not to use it

• Packet loss with idle CPU. Drops and utilisation decouple. Packets can be dropped at the ring, at the per-CPU backlog, or at a socket while the CPU has headroom. That is a drop investigation, and it starts at the counters rather than at top.
• A single flapping session or prefix. That is route instability. It can raise CPU as a side effect, and treating the side effect wastes the window.
• A planned upgrade or a known burst. Convergence load during a full-table refresh is the router doing its job. Wait for it.
• “The router feels slow” with no measurement. Get one first. Half the reports that arrive as CPU problems are DNS, an asymmetric path, or an application.

Blast radius

Reading is free — except that on a saturated router, some diagnostics are not.

tcpdump on a busy forwarding interface adds a copy of every matching packet to the softirq path you are already investigating, and can turn a router that was coping into one that is not. Verbose FRR debugging adds work to a control plane that is already behind. A full SNMP walk against a router whose management plane is the thing saturating is a diagnostic that reproduces the fault.

Use the cheap observers — mpstat, top, pidstat, the /proc counters — and reach for packet capture only after the class of work is already known, and preferably on a switch SPAN port rather than on the router.

Inputs

• The alert, with its threshold and the time it fired.
• This router’s recorded baseline: normal per-core %usr and %soft, normal route count, normal throughput.
• What changed in the estate in the preceding hour — commits, peer resets, upgrades, new traffic.
• Whether this router terminates IPsec or WireGuard, and roughly how much.
• The interface layout: which NIC carries the transit, how many queues it has, how many cores the router has.

Step 1: Is anything actually being harmed?

Attribution is the second question. The first is whether this matters at all, because a router can be legitimately busy.

show ip bgp summary
show ip ospf neighbor
show ip route summary
show interfaces

cat /proc/net/softnet_stat
ethtool -S eth1 | grep -Ei 'drop|miss|error|discard'
cat /sys/class/net/eth1/statistics/rx_dropped

Three outcomes, and they diverge sharply:

• Busy, no harm. Adjacencies up, drop counters flat, latency normal. The router is at capacity for the work it is being given. This is a capacity conversation with an owner and a timescale — not an incident, and not something to tune under pressure.
• Busy, with harm. Drops climbing, or adjacencies bouncing, or latency up. Now it is an incident, and the attribution below decides what kind.
• Busy, and the harm has a different cause. Common. The CPU is elevated because something else is wrong — a flap storm, a leaked full table, an interface error rate driving retransmits. Fixing the cause fixes the CPU; tuning the CPU fixes nothing.

Step 2: Which class of work

mpstat -P ALL 1 5

Read the columns, and read the rows — the per-core spread carries as much information as the totals.

One core at 95% while the others idle is the signature of work that cannot be spread: a NIC IRQ landing on a single core, or a single-threaded daemon. All cores evenly loaded is the signature of work that is being spread and is simply too much.

%steal deserves its own note. On a virtualised VyOS instance, steal time means the hypervisor is not giving this guest the CPU it is asking for. Nothing configured inside the router changes that, and every minute spent tuning the guest is a minute not spent talking to whoever owns the host.

Step 3a: Userspace — name the process

top -bn1 | head -30

ZEBRA_PID=$(pgrep -x zebra)
pidstat -t -p "$ZEBRA_PID" 1 5

pidstat -t breaks a daemon down by thread, and for zebra that is the observation that matters: zebra is single-threaded for RIB processing. Every route update flows through one thread regardless of how many cores the router has. A zebra thread at 100% on an eight-core router is a fully saturated route-processing path even though mpstat shows the box at 12% overall — and adding cores does nothing for it. The only lever is fewer updates.

The process name maps to a cause, and each has a different owner:

• zebra — route churn. The RIB is being rewritten faster than it can be pushed to the kernel. Look at what is generating the updates.
• bgpd — protocol message load: a peer reset, a full-table refresh, a peer sending far more than expected, or a flap storm.
• ospfd — SPF recalculation, driven by LSA churn. Usually a flapping link somewhere in the area, and the cost scales with the area’s size.
• sshd, snmpd, a telemetry exporter — the management plane. An SNMP walk storm or an exporter in a restart loop, both of which belong to whoever runs the monitoring.
• A crypto daemon or a heavy %sys — go to Step 3c.

show ip route summary
show ip bgp summary
show ip bgp flap-statistics
show ip ospf database

show ip route summary against the baseline is the fastest way to separate “a peer sent us far more than it should” from “the same routes keep being rewritten”. The first shows as a table that is much larger than normal; the second as a normal-sized table with a high flap count.

Step 3b: Softirq — one core or all of them

cat /proc/net/softnet_stat
cat /proc/interrupts | head -20
ethtool -l eth1
ethtool -g eth1

/proc/net/softnet_stat has one row per CPU. The first column is packets processed, the second is packets dropped because that CPU’s input queue was full, and the third is the number of times the softirq pass ran out of budget before draining the backlog. The columns are hex, and the pattern across rows is what matters more than any single value.

• One row with drops, the rest clean. The NIC’s interrupts are landing on one core and that core cannot drain the ring. This is a distribution problem. ethtool -l shows how many queues the NIC actually has, and /proc/interrupts shows how many are in use. Remediation is RSS plus interrupt affinity — and it belongs to a change, because it is fiddly and because the runtime forms do not survive a reboot.
• Every row with drops. The offered rate exceeds what this router can process. No software knob fixes that. Raising the ring buffer moves the drop later and adds latency; raising the backlog absorbs bursts but not a sustained excess. The honest answers are more hardware, offload, or less traffic.
• Squeeze climbing with no drops. The softirq pass is being cut short repeatedly. Often it is being starved by a CPU-bound userspace process — which brings you back to Step 3a on the same box.

Step 3c: Crypto — a distinct signature

A router terminating IPsec or WireGuard does its encryption in the kernel, in softirq context, on whichever core handled the packet’s arrival. So heavy tunnel traffic looks like softirq saturation — with one extra question to ask before concluding the hardware is too small.

grep -m1 '^flags' /proc/cpuinfo | tr ' ' '
' | grep -x aes
lsmod | grep aes

If the CPU advertises aes and the module is loaded, hardware AES is available and the crypto is as fast as this CPU can make it. If either is missing, the kernel is falling back to software AES — which is an order of magnitude slower, and which presents exactly as “this router is too small for the tunnel”. That is a very cheap check to run before anyone specifies replacement hardware.

The distribution problem from Step 3b applies here with more force: because the encryption happens on the core that received the packet, a NIC whose interrupts all land on one core caps tunnel throughput at one core’s worth of crypto, no matter how many cores the box has.

Step 4: Transient, at capacity, or faulty

This classification is the output of the runbook. Everything above exists to support it.

Write it as a sentence: the class, the subsystem, the process or queue, the evidence. “Softirq, single-core, eth1 interrupts on CPU 0 only, softnet drops on CPU 0 and nowhere else” is an attribution somebody can act on. “CPU high, probably traffic” is not, and it is what the next person will find in the ticket.

Step 5: Stop before changing anything

Every remediation this investigation can point at is a change with its own blast radius:

• RSS and interrupt affinity — improves distribution, does not persist, interacts with irqbalance, and can move a hot spot rather than removing it.
• Ring buffer size — absorbs bursts, adds worst-case latency. Latency-sensitive traffic pays for it.
• Backlog and socket buffer sysctls — the running value and the configured value diverge the moment you use sysctl -w.
• Bounding a peer — a change to a peering relationship, with a peer-visible effect if it is wrong.
• Renice or cgroup limits on a routing daemon — trades convergence time for CPU headroom, silently, in a way no routing protocol knows about.
• Restarting FRR — drops every session and rebuilds the table. It reliably clears the symptom, and it also destroys the evidence and costs a full reconvergence across the estate.

Each deserves a change record and a rollback. None of them belongs in an investigation.

Rollback

The procedure is read-only, so there is normally nothing to undo. What needs undoing is anything reached for under pressure:

The runtime forms are the trap. They vanish at the next reboot, which looks like a rollback and is actually a fix quietly ceasing to exist.

Common patterns

Escalation

Escalate when:

• The attribution is a peer’s behaviour. Bounding a peer is a change to a relationship, not an operator action mid-incident.
• Every core is saturated in softirq at the offered rate. That is capacity, and capacity is bought, not configured.
• %steal is significant. The conversation is with whoever owns the hypervisor.
• The load is unsolicited traffic aimed at the control plane. That is a security and control-plane-protection question.
• Anyone proposes restarting FRR or rebooting. Both clear the symptom and destroy the evidence, and both should be a decision with a name against it.