Router CPU Saturation · Break/Fix · VyOS

Reported symptoms

edge1 is a four-core VyOS router at a regional edge. It holds two eBGP transit sessions to different providers, an OSPF area 0 adjacency into the data-centre fabric, and a customer aggregation port, eth3, carrying tagged sub-interfaces.

The ticket has four complaints stacked on it, filed by three different people over two hours: monitoring paging on load average 14 against four cores since 01:40; both transit sessions resetting several times at different moments, with neither provider reporting anything on their side; the OSPF adjacency to the fabric cycling Full, Init, Full every few minutes; and a router that is unpleasant to use, where show ip route takes the best part of ten seconds and a commit took long enough that the operator assumed it had hung.

Two details keep the investigation from converging. Forwarding is fine — transit throughput is at its normal overnight level, no customer has reported loss, and the packets going through the box are not the problem. And somebody already restarted FRR at 02:20: everything came clean for roughly four minutes, then went back to exactly the previous state, which was read on the bridge as “the restart did not work” rather than as the clue it is.

Evidence provided

Read-only / Safethe number the alert fired on

vyos@edge1:~$ uptime

 03:05:12 up 61 days,  4:22,  2 users,  load average: 14.21, 13.88, 12.40

Illustrative output

Read-only / Safeone core pegged, three idle, and the softirq columns are quiet

vyos@edge1:~$ mpstat -P ALL 1 5

03:05:20     CPU    %usr   %nice    %sys  %iowait    %irq   %soft  %steal   %idle
03:05:21     all   25.31    0.00    1.02     0.00    0.00    2.55    0.00   71.12
03:05:21       0   99.00    0.00    1.00     0.00    0.00    0.00    0.00    0.00
03:05:21       1    3.06    0.00    1.02     0.00    0.00    4.08    0.00   91.84
03:05:21       2    1.01    0.00    1.01     0.00    0.00    3.03    0.00   94.95
03:05:21       3    2.02    0.00    1.01     0.00    0.00    3.03    0.00   93.94

Illustrative output

Read-only / Safethe userspace consumer is named, and it is not the packet path

vyos@edge1:~$ top -bn1 | head -14

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2147 frr       20   0  412360 118204   6120 R  98.4   2.9  418:33.12 zebra
2159 frr       20   0  866112 502440   6284 S  41.2  12.3  204:11.87 bgpd
2153 frr       20   0  198744  46012   5880 S  18.7   1.1   77:52.40 ospfd
1219 root      20   0   58312   9204   7440 S   1.3   0.2    3:11.05 vyos-configd

Illustrative output

Read-only / Saferun this twice, a minute apart, and compare the totals

vyos@edge1:~$ show ip route summary

Route Source         Routes               FIB  (vrf default)
connected            14                   14
static               22                   22
ospf                 318                  318
ebgp                 962184               962184
------
Totals               962538               962538

Illustrative output

Read-only / Safeone interface, and then a hold timer on an unrelated peer

vyos@edge1:~$ journalctl -u frr --since "-30 min" | tail -12

Aug 18 03:02:41 edge1 ospfd[2153]: Interface eth3.410 [1] changed state to Down
Aug 18 03:02:41 edge1 zebra[2147]: interface eth3.410 vrf default index 19 is down
Aug 18 03:02:45 edge1 ospfd[2153]: Interface eth3.410 [1] changed state to Up
Aug 18 03:02:45 edge1 zebra[2147]: interface eth3.410 vrf default index 19 is up
Aug 18 03:02:52 edge1 ospfd[2153]: Interface eth3.410 [1] changed state to Down
Aug 18 03:02:57 edge1 ospfd[2153]: Interface eth3.410 [1] changed state to Up
Aug 18 03:03:04 edge1 bgpd[2159]: %ADJCHANGE: neighbour 203.0.113.1(isp-a) in vrf default Down BGP Notification send (Hold Timer Expired)
Aug 18 03:03:06 edge1 ospfd[2153]: Interface eth3.410 [1] changed state to Down

Illustrative output

Read-only / Saferun it again in sixty seconds

vyos@edge1:~$ cat /sys/class/net/eth3/carrier_changes

Illustrative output

Work the evidence before reading on

Four failures are on the ticket. Before reading further, decide how many faults you think there are.

The alert fired on load average 14 against four cores. mpstat shows three of those cores idle. Which of the two is measuring utilisation, and what is the other one measuring?
Exactly one core is at 100% %usr, and the process at the top of top is zebra. What does “exactly one core” tell you about the shape of the work, and what does it rule out about the fix?
%soft is in single digits on every core, and forwarding is healthy. What whole category of cause does that eliminate in one line?
The FRR restart produced four clean minutes. What has to be true of the input for a restart to help for exactly that long and then stop helping?

Then the question that resolves the incident: the two transit sessions reset at different moments, and neither provider has anything to report. What single local condition would make two unrelated peers miss their hold timers without either peer doing anything wrong?

Root cause

1. One physical port is flapping

eth3 carries a customer aggregation on tagged sub-interfaces. Its optic is failing, and the link is bouncing every few seconds — thousands of carrier transitions, still incrementing while you look at it. show interfaces ethernet eth3 at any given moment shows the port up with unremarkable counters, which is why the interface was cleared early in the investigation: a flapping port looks healthy in roughly half of all snapshots you take of it. The counter in sysfs is cumulative and does not lie about history the way a point-in-time reading does.

2. Redistribution turns one port’s instability into the estate’s

The sub-interface eth3.410 contributes a connected route and two statics. Those are redistributed into OSPF, and OSPF is redistributed into BGP. That chain is not exotic; it is how most edge routers are built.

The consequence is that a carrier transition is not a local event. Each one makes ospfd regenerate and flood LSAs and rerun SPF, makes bgpd recompute and re-advertise the affected prefixes to every peer, and makes zebra install and withdraw routes in the kernel FIB. At one flap every four seconds, none of the three daemons ever reaches idle.

3. zebra is single-threaded, which is what “one core” means

Every RIB-to-FIB operation on the router passes through zebra, and zebra processes them on one thread. That is why a four-core box shows one core at 100% and three at nothing. It is also why the box being “only 25% utilised in aggregate” is meaningless: the resource that is exhausted is one thread, and the other three cores cannot help with it.

This is where the wrong fix gets proposed: a router at 100% of one core with an aggregate figure of 25% invites “we have outgrown this box”, and more cores would not move this number at all.

4. The BGP and OSPF instability is a symptom, not a second fault

FRR’s daemons are event loops. A daemon spending its time regenerating updates services its timers late — and a keepalive sent late enough is a keepalive that did not arrive before the peer’s hold timer expired. So sessions with no relationship to eth3 reset. Each reset causes a full re-advertisement in both directions, which is more work, which makes the next timer later still.

That is why the two transit resets happened at different moments, and why both providers are correct when they say their side is clean. It is also why the restart bought exactly four minutes: restarting FRR drains the accumulated queue, and then the port — which nobody had touched — refills it.

Under the hood

The single most useful line in this incident is the split between %usr and %soft in mpstat, and it costs one command.

Packets the router forwards are processed in softirq context: the NIC raises an interrupt, the kernel polls the ring, and the packet goes through routing, firewall and NAT without ever reaching a userspace process. If the router were out of forwarding capacity, %soft would dominate and ksoftirqd would be at the top of top. Control-plane work — BGP updates, OSPF LSAs, RIB-to-FIB programming — happens in userspace, in the FRR daemons, and shows up in %usr.

Here %soft is in single digits and %usr is 100% on one core. That single comparison eliminates the entire “we are out of packet-forwarding capacity” branch, along with every fix hanging off it: interrupt affinity, receive-side scaling, ring-buffer sizing, faster NICs. None would have touched this incident, and all are plausible-sounding changes to make to a router at 100% CPU.

The per-core breakdown does the second half. An aggregate figure cannot distinguish “four cores at 25%” from “one core at 100%”, and those two states have nothing in common.

A restart that works for four minutes is evidence, not a fix

The FRR restart at 02:20 was not useless. It was a measurement, and it was read as a failed repair.

A restart clears accumulated state. If the fault is accumulated state — a leaked resource, a wedged session, a corrupted table — the restart fixes it and it stays fixed. If the fault is a live input that keeps arriving, the restart clears the backlog and the input refills it, and the time it takes to refill tells you the rate of that input.

Four minutes of quiet followed by a return to exactly the previous state says: whatever drives this is still running, and it is external to the process you restarted. That should have redirected the investigation at 02:24. Instead it went into the ticket as “tried restarting FRR, no improvement”, which is both true and the opposite of what happened.

When a restart helps temporarily, the useful question is never “why did it not work” but “how long did it last, and what does that duration tell me about the thing refilling it”.

Resolution

Confirm the source before touching routing. Read cat /sys/class/net/eth3/carrier_changes twice, sixty seconds apart. A counter that moves is the fault; a counter that does not means you are looking at the wrong port and should re-read the journalctl -u frr interface-state lines for the one that is named there.
Decide who owns the outage you are about to cause, because every option below costs somebody service. This is a commercial decision as much as a technical one, and at 03:00 it needs a name attached to it.
Option A, the surgical one: administratively disable the failing port. set interfaces ethernet eth3 disable and commit-confirm 5. Churn stops immediately and the control plane recovers within a convergence interval. The customer on eth3.410 is down until an engineer replaces the optic.
Option B, if that customer cannot be dropped: leave the link up and stop the flap propagating. Bound the redistribution with a route-map so the prefixes behind eth3.410 are not injected into OSPF. The customer keeps their link; their prefixes stop being reachable from the rest of the estate. Same conversation, smaller blast radius.
Option C, hold. Change nothing in the routing, dispatch a field engineer with a replacement optic, and run degraded until they arrive. This is a legitimate choice when the transit resets are not causing customer-visible loss — but it is a decision to operate a router that cannot converge promptly, so it gets a named owner and a review time, not an implicit "we will see how it goes".
Whichever you choose, use commit-confirm with a short timer. The router is slow and your commit will take longer than usual; a change that makes things worse on a box in this state is difficult to back out by hand.
Do not restart FRR again. It will look like it worked for about four minutes.
Raise the hardware fault against eth3 with the transition count and the log excerpt attached. It needs replacing whichever routing option you took tonight.

Verification

The source is quiet. cat /sys/class/net/eth3/carrier_changes returns the same value across several minutes. Verify at the source first — every check below can look correct during a lull between waves.
No core is pegged. mpstat -P ALL 1 5 shows no single core at 100% %usr, and zebra has dropped out of the top of top -bn1.
The route table has stopped moving. show ip route summary run twice, a minute apart, returns identical totals. This is the check that distinguishes "the churn stopped" from "the CPU figure happens to be low right now".
Adjacencies are aging. show ip ospf neighbor shows every neighbour Full with an uptime past the previous flap interval, and show ip bgp summary shows both transit sessions with uptimes past their previous reset interval. Uptime is the evidence; the state column reads Established on a session that reset thirty seconds ago.
The management plane is usable again. show ip route returns promptly and a trivial commit completes in normal time. This is the symptom the operator on shift will judge you on.
If you took Option B, prove the filter does something. Confirm the eth3.410 prefixes are absent from show ip route ospf on a second router in the area — a redistribution route-map that has never been observed to exclude anything has not been tested.
If you took Option C, the verification is a calendar entry, not a command: the review time is set and the owner knows they hold it.

Prevention

Bound every redistribute statement with a route-map naming the prefixes it may carry. Unbounded redistribution is what gave one customer port estate-wide reach, and it is the difference between a flapping port and a flapping network:

set policy prefix-list PL-CUSTOMER-AGG rule 10 action permit
set policy prefix-list PL-CUSTOMER-AGG rule 10 prefix 198.51.100.0/24
set policy route-map RM-CONNECTED-TO-OSPF rule 10 action permit
set policy route-map RM-CONNECTED-TO-OSPF rule 10 match ip address prefix-list PL-CUSTOMER-AGG
set protocols ospf redistribute connected route-map RM-CONNECTED-TO-OSPF

A route-map on the redistribution does not stop the port flapping. It stops the flap being an OSPF event, and therefore stops it being a BGP event.

Alert on route churn, not only on CPU. Routes added and withdrawn per minute, or the delta between successive show ip route summary totals, identifies this class of incident in seconds. A CPU alert only tells you the router is busy, which is the least useful true statement available.
Alert on carrier transitions per interface. /sys/class/net/*/carrier_changes is cheap to poll and turns a failing optic into a hardware ticket days before it becomes a routing incident.
Retire the load-average alert on routers. Load average is run-queue depth, not utilisation; a load average of 14 on this box coexisted with three idle cores, and it is what pointed the first hour of the investigation at capacity. Alert on per-core utilisation split by mode instead, so %usr and %soft are distinguishable at alert time and not only during the postmortem.
Write down that zebra is single-threaded, somewhere an operator at 03:00 will find it. Buying more cores is an expensive wrong answer that takes weeks to disprove.

What if BFD had been enabled on the affected paths?

It would have made this incident arrive faster and hit harder, which is worth sitting with, because BFD is otherwise exactly the right thing to deploy.

BFD detects a failed path in milliseconds instead of seconds, so traffic stops being sent into a hole. Against a link that fails cleanly that is a large improvement. Against a link that oscillates, faster detection means more transitions reach the routing protocols per unit time, and every transition is the same fan-out: LSA regeneration, BGP re-advertisement, FIB reprogramming. The mechanism that protects you from a blackhole amplifies you into a churn storm.

That is not an argument against BFD. It is an argument that fast detection and churn damping are a pair, and that deploying the first without the second moves the failure mode rather than removing it — the damping half being flap damping for BGP-learned prefixes, area design and summarisation so an internal flap does not reach the backbone, and route-maps so an unstable prefix is never redistributed.

Any change that makes a router react faster to the network also makes it react faster to a network that is lying to it.

Why this matters

Four symptoms arrived from four directions — a capacity alert, two transit providers, an internal fabric adjacency, and an operator complaining the CLI was slow. Three of them were consequences of the fourth thing nobody reported, because nobody notices a port that comes back up every time.

The skill here is not “check the interfaces”. It is that mpstat -P ALL costs one command and splits the problem space in half before you have formed a hypothesis: %usr against %soft separates control plane from data plane, and per-core against aggregate separates a busy router from a saturated single thread. Both distinctions were available in the first thirty seconds, and both were skipped in favour of a load-average number that was never measuring what the alert implied.

On a router the interesting question is usually not which resource is exhausted but what is feeding it. CPU was exhausted here and CPU was not the fault: there was no fix at that layer to find. The fault was a failing optic, and the path from one to the other ran through a redistribution statement nobody had looked at in two years.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. One physical port is flapping

2. Redistribution turns one port’s instability into the estate’s

3. zebra is single-threaded, which is what “one core” means

4. The BGP and OSPF instability is a symptom, not a second fault

Resolution

Verification

Prevention