Lab: Performance Troubleshooting

Objective

“The router’s CPU is at 100%” is three different findings wearing one number, and the fix for each lives in a different subsystem. This lab makes you produce one of them on purpose, on hardware you own, so that you can watch the evidence separate it from the other two.

You will build the amplification chain first — a customer port whose connected route and static routes are redistributed into OSPF, and OSPF into BGP — and prove it exists before injecting anything. Then you will flap the port and attribute the result in the order the CPU runbook insists on: which class of work, then which daemon, then which thread. Each step eliminates fixes, and by the end three of the four things people reach for are off the table before anyone has typed a configuration command.

The mitigation at the end is deliberately unimpressive. It does not make the router faster. It stops one unstable interface from being an estate-wide event, which is the only kind of fix this class of problem has.

Architecture

Four VyOS VMs. edge is the router under test and it is deliberately built with two vCPUs, because the single most informative thing in this lab is one core pegged next to one core idle, and that pattern is invisible on a single-vCPU VM.

                     isp   (AS 65010)
              +--------------------------+
              | eth0 MGMT 192.0.2.12     |
              | eth1 203.0.113.2/30      |
              | dum0 203.0.113.129/32    |  the far service
              +--------------------------+
                           |
                eBGP  203.0.113.0/30
                           |
              +--------------------------+
              |   edge   (AS 65001)      |
              | eth0 MGMT 192.0.2.11     |
              | eth1 203.0.113.1/30      |
              | eth2 198.51.100.1/30     |
              | eth3 198.51.100.129/25   |  the unstable port
              +--------------------------+
                     |             |
     OSPF area 0     |             |   ACCESS 198.51.100.128/25
   198.51.100.0/30   |             |
                     |             |
       +--------------------+   +--------------------+
       |       core         |   |        ops         |
       | eth1 198.51.100.2  |   | eth1 198.51.100.130|
       | dum0 198.51.100.65 |   |                    |
       +--------------------+   +--------------------+

Every address is from a range RFC 5737 reserves for documentation. The customer prefixes are carved out of the same 192.0.2.0/24 block as the management segment, which is why management is a /26 — nothing in this lab may overlap anything else, because an overlap would produce exactly the kind of ambiguous routing evidence the lab is trying to teach you to avoid.

Requirements

• A hypervisor with roughly 5 GiB of free RAM and 32 GiB of free disk. edge gets 2 vCPU and 2 GiB; the other three can be 1 vCPU and 1 GiB.
• The VyOS 1.5 LTS ISO.
• Four isolated layer-2 bridges with no physical port: MGMT, TRANSIT, FABRIC, ACCESS.
• Root on the hypervisor, to bring a tap interface up and down. Task 4 needs it, and the guest-side alternative is in the callout there.
• Console access to all four VMs, per the callout above.
• sysstat on edge for mpstat and pidstat. Part L-06 states VyOS 1.5 ships it; if mpstat is not on your image, Task 1 has the fallback.
• Roughly two hours, of which about twenty-five minutes is measurement windows you have to actually wait out.

Scenario

edge is a regional edge router. It holds one eBGP transit session, one OSPF adjacency into the fabric, and a customer aggregation port. It has been in service for a year without anyone looking at it closely.

Monitoring has started paging on load average. Three complaints arrive over two hours from three different people: the transit session resetting at odd intervals with the provider reporting nothing on their side; the OSPF adjacency cycling; and a router that is unpleasant to use, where show ip route takes seconds. Forwarding, meanwhile, is fine — throughput is normal and no customer has reported loss.

Your job is not to fix it. It is to produce a sentence somebody can act on, and then to demonstrate that the obvious actions would not have produced it.

Tasks

Configuration blocks are written for the [edit] prompt. Where a block opens with configure and you are already at [edit], skip that line. Blocks tagged bash are the router’s operational-mode shell or the hypervisor host, as their comments say. Blocks made up of show commands are operational mode; from inside configure, prefix each one with run.

Task 1 — Build the estate, and capture the baseline that makes everything else readable

Install four routers, give each a hostname and a MGMT address on eth0, then build the routing.

edge — two vCPU, one eBGP session, one OSPF adjacency, one access port:

configure
set system host-name edge
set interfaces ethernet eth0 address 192.0.2.11/26
set interfaces ethernet eth0 description 'MGMT'
set interfaces ethernet eth1 address 203.0.113.1/30
set interfaces ethernet eth1 description 'TRANSIT to isp'
set interfaces ethernet eth2 address 198.51.100.1/30
set interfaces ethernet eth2 description 'FABRIC to core'
set interfaces ethernet eth3 address 198.51.100.129/25
set interfaces ethernet eth3 description 'ACCESS customer aggregation'

set protocols ospf parameters router-id 198.51.100.1
set protocols ospf area 0 network 198.51.100.0/30

set protocols bgp system-as 65001
set protocols bgp parameters router-id 203.0.113.1
set protocols bgp parameters log-neighbor-changes
set protocols bgp neighbor 203.0.113.2 remote-as 65010
set protocols bgp neighbor 203.0.113.2 description 'isp transit'
commit
save

isp — the transit peer and the far service:

configure
set system host-name isp
set interfaces ethernet eth0 address 192.0.2.12/26
set interfaces ethernet eth1 address 203.0.113.2/30
set interfaces dummy dum0 address 203.0.113.129/32
set interfaces dummy dum0 description 'far service'
set protocols bgp system-as 65010
set protocols bgp parameters router-id 203.0.113.2
set protocols bgp parameters log-neighbor-changes
set protocols bgp address-family ipv4-unicast network 203.0.113.128/25
set protocols bgp neighbor 203.0.113.1 remote-as 65001
set protocols bgp neighbor 203.0.113.1 description 'edge customer'
commit
save

core — the fabric neighbour:

configure
set system host-name core
set interfaces ethernet eth0 address 192.0.2.13/26
set interfaces ethernet eth1 address 198.51.100.2/30
set interfaces dummy dum0 address 198.51.100.65/32
set protocols ospf parameters router-id 198.51.100.2
set protocols ospf area 0 network 198.51.100.0/30
set protocols ospf area 0 network 198.51.100.65/32
commit
save

ops — the customer host, and the source of every end-to-end measurement:

configure
set system host-name ops
set interfaces ethernet eth0 address 192.0.2.14/26
set interfaces ethernet eth1 address 198.51.100.130/25
set protocols static route 0.0.0.0/0 next-hop 198.51.100.129
commit
save

Now the baseline. Part L-06 is explicit that a baseline captured at idle, with no note of the load it was taken under, produces alert thresholds that are wrong — so write the conditions at the top of the file and then take four readings.

# On edge, operational mode.
JOURNAL="$HOME/lab23"
mkdir -p "$JOURNAL"
B="$JOURNAL/baseline.txt"

{
  echo "# edge baseline"
  echo "# taken: $(date -Iseconds)"
  echo "# load:  idle -- one eBGP session, one OSPF adjacency, no customer traffic"
  echo "# kernel: $(uname -r)"
  echo
  echo "== uptime"
  uptime
  echo
  echo "== mpstat -P ALL 1 60"
  mpstat -P ALL 1 60 | tail -8
  echo
  echo "== zebra threads"
  pidstat -t -p "$(pgrep -x zebra)" 1 5 | tail -12
} | tee "$B"

Two more readings that only mean something as a pair. Run this, wait sixty seconds, run it again, and record both:

# On edge, operational mode. Run twice, sixty seconds apart.
date -Iseconds
vtysh -c 'show ip route summary'
cat /sys/class/net/eth3/carrier_changes

A single route count is a number. Two counts a minute apart are evidence: the delta between them is the churn rate, and it is the measurement that separates “the table is large” from “the table keeps being rewritten”. Those are different problems with different fixes, and Part L-05 puts them in different rows.

Finally, the number that comes from outside the router:

$ ping 203.0.113.129 count 20

--- 203.0.113.129 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19029ms
rtt min/avg/max/mdev = 0.612/0.884/1.402/0.171 ms

Task 2 — The number that starts every wrong investigation

The alert in the scenario fired on load average, and load average is the reason this class of incident so often begins in the wrong subsystem.

Read both numbers at rest, together:

$ uptime

 11:04:12 up 2:18,  1 user,  load average: 0.08, 0.11, 0.09

Load average is run-queue depth: how many processes were runnable, averaged over three windows. On a two-core router, a load average of 2 means both cores have work and nothing is queueing — a fully used router, not an overloaded one. A load average of 8 on the same box means work is waiting.

More importantly, it says nothing about class. It cannot distinguish a router burning cycles in bgpd from one burning them in the kernel packet path, and that distinction is the entire content of the rest of this lab. Write in the journal now, before there is any pressure to skip it: no change is opened on the strength of a load average.

Task 3 — Build the amplification chain, and prove it before you use it

One flapping port is a local event. What turns it into an estate-wide event is redistribution, and this is the chain most edge routers are genuinely built with — which is why the break/fix scenario for this subject opens with it.

On edge, give the access port some customer prefixes, and then wire the chain one hop at a time, verifying each hop before adding the next.

configure
set protocols static route 192.0.2.64/29 next-hop 198.51.100.130
set protocols static route 192.0.2.72/29 next-hop 198.51.100.130
set protocols static route 192.0.2.80/29 next-hop 198.51.100.130
set protocols static route 192.0.2.88/29 next-hop 198.51.100.130
set protocols static route 192.0.2.96/29 next-hop 198.51.100.130
set protocols static route 192.0.2.104/29 next-hop 198.51.100.130
set protocols static route 192.0.2.112/29 next-hop 198.51.100.130
set protocols static route 192.0.2.120/29 next-hop 198.51.100.130
commit
save
exit

Eight prefixes is not a production table and the lab does not pretend it is. What matters is that each one is recursive on the access port: their next-hop lives on eth3, so when eth3 loses carrier the connected route goes, the next-hop stops resolving, and all eight are withdrawn together. That is the multiplier, and in production the same structure carries thousands.

Hop one — statics and the access connected route into OSPF:

configure
set protocols ospf redistribute connected metric-type 2
set protocols ospf redistribute static metric-type 2
commit
save
exit

Verify it before continuing. On core:

show ip ospf database
show ip route ospf

core should now hold the eight customer prefixes and the access segment as OSPF externals. If it does not, stop here — the rest of the lab measures a chain that does not exist.

Hop two — OSPF into BGP, toward the transit peer:

configure
set policy prefix-list CUSTOMER-SPACE rule 10 action 'permit'
set policy prefix-list CUSTOMER-SPACE rule 10 prefix '192.0.2.64/26'
set policy prefix-list CUSTOMER-SPACE rule 10 le '29'
set policy route-map OSPF-TO-BGP rule 10 action 'permit'
set policy route-map OSPF-TO-BGP rule 10 match ip address prefix-list 'CUSTOMER-SPACE'
set protocols bgp address-family ipv4-unicast redistribute ospf route-map 'OSPF-TO-BGP'
commit
save
exit

Verify on isp:

$ show bgp ipv4 unicast

   Network          Next Hop         Metric LocPrf Weight Path
*> 192.0.2.64/29    203.0.113.1          20             0 65001 ?
*> 192.0.2.72/29    203.0.113.1          20             0 65001 ?
*> 192.0.2.80/29    203.0.113.1          20             0 65001 ?
*> 192.0.2.88/29    203.0.113.1          20             0 65001 ?
*> 192.0.2.96/29    203.0.113.1          20             0 65001 ?
*> 192.0.2.104/29   203.0.113.1          20             0 65001 ?
*> 192.0.2.112/29   203.0.113.1          20             0 65001 ?
*> 192.0.2.120/29   203.0.113.1          20             0 65001 ?

Stop and look at what you have built. A carrier transition on eth3 is now four events, not one: zebra withdraws the connected route and the recursive statics from the FIB, ospfd regenerates and floods external LSAs and reruns SPF, bgpd recomputes and re-advertises to a peer in another autonomous system, and every one of those has to be undone four seconds later when the port comes back.

Nothing here is exotic and nothing here is wrong. It is how the router was asked to work.

Task 4 — Inject the flap, and attribute in order

Flap the access port from the hypervisor, so the fault is external to the guest exactly as a failing optic is.

# On the hypervisor, as root. Substitute your own values before running:
VM=edge

virsh domiflist "$VM"
# Read the Interface column for the tap backing eth3 and set it below.
TAP=vnet7

for i in $(seq 1 90); do
  ip link set "$TAP" down
  sleep 2
  ip link set "$TAP" up
  sleep 2
done

While the loop runs, work the evidence in the order the CPU runbook fixes.

First: is anything being harmed?

show ip ospf neighbor
show bgp summary
show ip route summary
show interfaces

Second: which class of work. This is the one command that splits the problem, and everything after it is narrowing.

$ mpstat -P ALL 1 5

            CPU    %usr   %nice    %sys  %iowait    %irq   %soft  %steal   %idle
          all   49.24    0.00    1.51     0.00    0.00    1.51    0.00   47.74
            0   97.00    0.00    2.00     0.00    0.00    1.00    0.00    0.00
            1    1.01    0.00    1.01     0.00    0.00    2.02    0.00   95.96

Two facts in that output and they matter in different ways. %usr dominates and %soft is in single digits — the work is in userspace, not in the kernel packet path. And it is on one core, with the other idle, which is the signature of work that cannot be spread.

Note what the all row says: 49% aggregate. A router reported as “half idle” whose limiting resource is completely exhausted. That number is the reason this incident gets closed as “we have outgrown the box”.

Third: which daemon.

$ top -bn1 | head -12

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2147 frr       20   0  212360  58204   6120 R  96.7   2.9   18:33.12 zebra
2159 frr       20   0  266112  92440   6284 S  38.2   4.6   09:11.87 bgpd
2153 frr       20   0  148744  36012   5880 S  21.4   1.8   05:52.40 ospfd
1219 root      20   0   58312   9204   7440 S   1.1   0.4    0:31.05 vyos-configd

Fourth: which thread. This is the step that turns a diagnosis into a prognosis.

# On edge, operational mode.
pidstat -t -p "$(pgrep -x zebra)" 1 5

One zebra thread carries essentially all of it. Every RIB-to-FIB operation on the router passes through that thread, and it does not matter how many cores the box has. This is why “add cores” is not on the list of fixes, and it is worth writing down as a sentence rather than as a feeling.

Fifth: confirm the input. Two readings, sixty seconds apart, of the same two counters you took in Task 1:

# On edge, operational mode. Run twice, sixty seconds apart.
date -Iseconds
vtysh -c 'show ip route summary'
cat /sys/class/net/eth3/carrier_changes

The route totals barely move; the carrier counter climbs steadily. That pair is the whole diagnosis in two lines: the table is not growing, so this is not a peer sending more than it should. The same routes are being written and unwritten, driven by a counter that is still incrementing while you look at it.

# On edge, operational mode.
journalctl -u frr --since "-10 min" | tail -20

Now write the attribution as a sentence, in the form the runbook asks for: class, subsystem, process, evidence. “Userspace, single-core, zebra thread at 97% on CPU 0, driven by 400-plus carrier transitions on eth3 in ten minutes, route totals flat” is something a colleague can act on. “CPU high, probably traffic” is not.

Task 5 — Eliminate the packet path with one reading

You have already done this and it is worth making explicit, because it is the cheapest elimination available and it is routinely skipped.

%soft in single digits, across both cores, rules out the entire kernel packet-path category: interrupt affinity, RSS, ring buffer sizing, backlog sysctls, NIC offload, and the “buy a faster NIC” conversation. All of it. One column.

Confirm it holds under the flap rather than assuming:

# On edge, operational mode.
cat /proc/net/softnet_stat
ethtool -S eth1 | grep -Ei 'drop|miss|error|discard'
cat /sys/class/net/eth1/statistics/rx_dropped

And confirm from outside the router that forwarding is genuinely healthy, which is the scenario’s most confusing detail and its most informative one:

$ ping 203.0.113.129 count 20

--- 203.0.113.129 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19031ms
rtt min/avg/max/mdev = 0.641/0.951/2.884/0.402 ms

Compare that against the Task 1 baseline. Latency has moved a little; loss has not appeared. A CPU incident in which the traffic is fine is not a contradiction — the forwarding path is in the kernel and it is not the thing that is busy.

Task 6 — The peer that resets although nobody touched it

Let the flap loop keep running and watch the transit session.

show bgp summary
show ip ospf neighbor

# On edge, operational mode.
journalctl -u frr --since "-15 min" | grep -Ei 'adjchange|hold|neighbor'

Somewhere in there a session resets on a hold timer. Check isp — nothing has happened on its side, its interface is clean, and it is entirely correct when it says so.

The mechanism is worth naming precisely, because it is the part that makes this class of incident look like several unrelated faults. FRR’s daemons are event loops. A daemon spending its time regenerating updates services its timers late, and a keepalive sent late enough is a keepalive that did not arrive before the peer’s hold timer expired. So a session with no relationship to the flapping port resets. Each reset causes a full re-advertisement in both directions, which is more work, which makes the next timer later still.

Time it if you can: note the interval between the resets in the log. In the scenario this is the detail that keeps the investigation from converging — two problems that look independent, filed by two different people, with one cause.

Task 7 — The restart that helps for exactly four minutes

Somebody on the call will propose restarting FRR. Do it, and time it — the duration is evidence about the input, not about the fix.

# On edge, operational mode. SERVICE-IMPACT: drops every adjacency on the box.
date -Iseconds
sudo systemctl restart frr

Watch mpstat -P ALL 1 5 and top for the next several minutes and record when the CPU returns to its saturated state.

It comes clean, briefly. Restarting FRR drains the accumulated queue of work and starts every daemon from an empty state. Then the port — which nobody has touched, and which is still flapping — refills it, and the router returns to exactly where it was.

Three things to take from that.

Read the duration as a measurement of the input. A fix that helps for a bounded time and then stops helping is not a partial fix; it is a fix aimed at the symptom while the cause continues unchanged. The length of the relief tells you how long the input takes to rebuild the backlog.

The restart destroyed evidence. The FRR logs before the restart, the session uptimes, the accumulated counters — all reset. If you had not captured them in Task 4 you would now be investigating a router with no history, and you would be doing it under more pressure than before.

It also cost a full reconvergence across the estate. Every adjacency on edge dropped and re-formed, which is visible in other people’s graphs and looks, to them, like a new incident. Note it in the journal with a name against it, because that is the standard the anti-patterns lesson holds a reboot or a restart to.

Stop the flap loop now, and confirm the router settles:

# On the hypervisor, as root.
ip link set "$TAP" up

# Then, on edge, operational mode:
mpstat -P ALL 1 10
cat /sys/class/net/eth3/carrier_changes

Task 8 — Bound the churn at its source, and prove it with a number

Now the mitigation, and the disappointing shape of it: nothing here makes the router faster. It stops one unstable interface from being an estate-wide event.

The durable fix is the optic, and it is not yours. What is yours is the amplification. Part L-05 lists redistribution as a churn source in its own right and names the fix: a route-map on the redistribution, so an unstable prefix does not propagate.

Restart the flap loop, take a measurement, then apply the change and take another. One variable at a time.

configure
set policy prefix-list STABLE-ONLY rule 10 action 'deny'
set policy prefix-list STABLE-ONLY rule 10 prefix '192.0.2.96/27'
set policy prefix-list STABLE-ONLY rule 10 le '29'
set policy prefix-list STABLE-ONLY rule 20 action 'permit'
set policy prefix-list STABLE-ONLY rule 20 prefix '192.0.2.64/26'
set policy prefix-list STABLE-ONLY rule 20 le '29'
set policy route-map STATIC-TO-OSPF rule 10 action 'permit'
set policy route-map STATIC-TO-OSPF rule 10 match ip address prefix-list 'STABLE-ONLY'
set protocols ospf redistribute static metric-type 2 route-map 'STATIC-TO-OSPF'
compare
commit-confirm 10
exit

Read the prefix-list ordering carefully, because it is the part that goes wrong. Rule 10 denies 192.0.2.96/27 — the four customer prefixes this exercise is treating as the unstable half — and rule 20 permits the rest. Prefix-list rules are evaluated in numeric order and the first match wins, so the deny has to be numbered below the permit it is meant to override. Write them the other way round and rule 10 permits everything, rule 20 never runs, the commit succeeds, the configuration reads correctly, and the mitigation measurably does nothing.

Note also what this does not stop. The access segment’s own connected route is still redistributed, so one prefix still moves on every transition. Halving the churn is the honest description of this change; a follow-up that replaces redistribute connected with explicit network statements would remove the rest, and it is a separate change with its own review.

Now re-measure, the same way you measured before:

# On edge, operational mode. Run twice, sixty seconds apart.
date -Iseconds
vtysh -c 'show ip route summary'
cat /sys/class/net/eth3/carrier_changes

mpstat -P ALL 1 30

The carrier counter still climbs — you have not fixed the port and you were never going to. What should have changed is the delta in the route totals and the per-core %usr, because fewer prefixes now move on each transition and fewer of those movements leave the router.

Write the before and after side by side in the journal. If the numbers did not move, the mitigation did not work, and the honest thing is to record that and check the prefix-list ordering rather than to describe it as “improved”.

configure
confirm
save
exit

Finally, do the step Part L-06 names as a failure mode in its own right: update the baseline. The router’s normal has legitimately changed — it now carries a redistribution route-map that did not exist this morning — and a baseline that still describes this morning will alert on the change you just made.

# On edge, operational mode. Re-run the Task 1 capture with a new header.
B2="$HOME/lab23/baseline-post-change.txt"

{
  echo "# edge baseline, after the redistribution route-map"
  echo "# taken: $(date -Iseconds)"
  echo "# load:  eth3 flapping stopped -- same conditions as the Task 1 capture"
  echo "# change: STATIC-TO-OSPF applied to protocols ospf redistribute static"
  echo
  mpstat -P ALL 1 60 | tail -8
  pidstat -t -p "$(pgrep -x zebra)" 1 5 | tail -12
} | tee "$B2"

diff "$HOME/lab23/baseline.txt" "$B2" | head -40

Validation

• The baseline file exists, carries the load it was taken under, and contains per-core CPU, zebra thread CPU, two route-summary readings a minute apart, and an end-to-end latency figure from ops.
• After Task 3, show bgp ipv4 unicast on isp listed all eight customer prefixes, proving the amplification chain reached another AS before any fault was injected.
• During Task 4, mpstat -P ALL 1 5 showed %usr dominant on one core with the other core largely idle, and %soft in single digits on both.
• top named zebra as the leading consumer and pidstat -t attributed it to a single thread.
• The two route-summary readings during the flap showed a flat total while /sys/class/net/eth3/carrier_changes climbed — the pair that separates churn from table growth.
• An attribution sentence exists in the journal naming class, subsystem, process and evidence.
• The FRR restart produced a measured period of relief, and its duration is recorded along with the reason it ended.
• A before/after pair exists across the Task 8 mitigation for both per-core %usr and the sixty-second route-summary delta.
• The baseline was re-captured after the accepted change, with a new header.

Expected Outcome

A four-router lab in which you built a routing amplification chain deliberately, proved it existed, drove it with an external fault, and attributed the result to one thread of one daemon using four commands — none of which changed anything.

More usefully, you have a worked example of elimination. %soft in single digits removed the entire packet-path category. A flat route total removed “a peer is sending too much”. One core busy next to one core idle removed “add hardware”. By the time a configuration command was typed, the candidate fixes had gone from a dozen to two, and the evidence for each removal is a line in your journal rather than an opinion.

Troubleshooting

mpstat reports one CPU. edge was built with one vCPU. The single-core-versus-all-cores distinction is the most informative reading in this lab and it is invisible on one. Shut it down, give it two, and re-take the baseline — a baseline from different hardware is not a baseline.

%steal is high throughout. The hypervisor is not delivering the CPU the guest is asking for, which on a nested lab is common. Reduce what else is running on the host, or accept it and note it: nothing configured inside edge will change that column, and every other measurement in the lab is riding on top of it.

core does not receive the customer prefixes as OSPF externals. Check that the OSPF adjacency is Full first — show ip ospf neighbor on both ends — and only then look at redistribution. An adjacency that never formed produces exactly the same empty database as a redistribution that was never configured.

isp has the BGP session up but no customer prefixes. The route-map is doing its job and rejecting them, which usually means the prefix-list does not match. show ip prefix-list CUSTOMER-SPACE and check the le bound: the statics are /29, so a list written without le 29 matches nothing.

The flap loop runs but carrier_changes on eth3 does not move. You are flapping the wrong tap. virsh domiflist edge lists them in the same order as the VM’s interfaces; match by MAC against cat /sys/class/net/eth3/address on the guest rather than by position.

CPU barely moves during the flap. Eight prefixes on a modern host may not be enough to produce a dramatic reading, and the lab does not need a dramatic one — it needs a distinguishable one. Shorten the sleep in the loop to one second, and read the relative change from your own baseline rather than looking for a particular number. If %usr moves and %soft does not, the lab has done its job.

The BGP session never resets in Task 6. It may not, on a fast host with a short table. The mechanism is still demonstrable from the logs: look for growing gaps between keepalives, or lower the hold timer on the session to make the margin smaller. Record honestly that you did not reproduce the reset rather than describing one you did not see.

After Task 8 the numbers are unchanged. Check the prefix-list rule order first. A deny numbered above the permit it was meant to override never runs, the route-map permits everything, and the configuration looks entirely correct. show ip prefix-list STABLE-ONLY prints the rules in evaluation order.

Cleanup

Nothing in this lab leaves persistent damage, but several things do outlive the session in ways that surprise the next person.

Step 1. Stop the flap loop and bring the tap up. A tap left down is an interface that stays down across a guest reboot and looks exactly like a hardware fault:

# On the hypervisor, as root.
ip link set "$TAP" up
ip -brief link show "$TAP"

Step 2. If you flapped from inside the guest instead, the runtime change is undone by bringing the link up or by a reboot — show configuration never mentioned it, so there is nothing to revert in the tree:

# On edge, operational mode.
sudo ip link set eth3 up
cat /sys/class/net/eth3/carrier_changes

Step 3. To return edge to its Task 1 state, remove the amplification chain and the mitigation together. Order matters: take the redistribution out before the policy it references, or the commit fails on a dangling reference.

configure
delete protocols bgp address-family ipv4-unicast redistribute
delete protocols ospf redistribute
delete policy route-map OSPF-TO-BGP
delete policy route-map STATIC-TO-OSPF
delete policy prefix-list CUSTOMER-SPACE
delete policy prefix-list STABLE-ONLY
delete protocols static
compare
commit
save
exit

Step 4. Keep the journal and the baseline file. They are the only artefacts of this lab worth anything, and the baseline in particular is the thing Part L-06 says most estates do not have.

Step 5. To remove the topology, delete the four VMs and the four bridges on the hypervisor. Confirm with ip -brief link show on the host that no lab bridge and no orphaned tap survives.

What You Learned

• Load average is run-queue depth, not utilisation, and it has no idea what class of work is running. It is the number the alert fires on and the worst number to act on.
• mpstat -P ALL is the cheapest instrument in the investigation. The %usr versus %soft split eliminates an entire category of fix in one reading, and the per-core spread eliminates another.
• An aggregate CPU figure hides a saturated single-threaded resource. Two cores at 97% and 2% report as 49% busy. zebra processes the RIB on one thread, so the router is full while the graph says it is half idle — and adding cores does nothing at all.
• Two counter readings beat one. A route total on its own is a number; the delta over sixty seconds is what separates “a peer sent too much” from “the same routes keep being rewritten”. They are different problems in different rows of Part L-05.
• Redistribution is what turns a local fault into an estate-wide one. The chain is not exotic; it is how most edge routers are built. It is also the only part of this incident that was actually yours to change.
• A fix that helps for a measured few minutes is a measurement of the input. The FRR restart drained a queue that the unchanged cause immediately refilled, and it cost a full reconvergence and every scrap of evidence to learn that.
• The mitigation does not make the router faster, and saying so is the point. Ranking the available levers by what each one costs is worth more in the write-up than the one that happened to be applied.