Skip to main content
RunBook Academy

← All break/fix scenarios in VyOS

advancedvyos-bfd~30 min

BFD Causes Excessive Flapping

Reported symptoms

  • ●The voice team reports bursts of one-way audio and dropped calls, several times an hour, starting at about 04:15 and still happening at 08:30
  • ●The eBGP session to Provider B on edge-03 resets roughly every 90 to 120 seconds; the session to Provider A on the same router is untouched
  • ●The interface facing Provider B shows no link-state transitions, no CRC errors and no discards, with 41 days of link uptime
  • ●The monitoring ICMP probe across that same link is 100% successful at under 2 ms for the whole period
  • ●Provider B confirms their planned maintenance completed at 05:00 and says the flapping is coming from our side
  • ●The platform team sees `bgpd` pegged and has opened a ticket asking to reboot the router onto a larger instance
  • ●Ten other routers received the identical BFD profile in the same rollout three days ago and every one of them is stable

Evidence

  • · `show ip bgp summary` on edge-03 shows the Provider B session Up/Down at 00:01:14 with roughly 980,000 prefixes received
  • · `show bfd peers` reports `Status: up`, an uptime in seconds, `Diagnostics: ok` and `Remote diagnostics: control detection time expired`
  • · `show bfd peers` local and remote timers both read Detect-multiplier 3, receive interval 50ms, transmission interval 50ms
  • · `journalctl -u bfdd` shows down/up state changes on a regular 90 to 120 second cadence, with no gaps since 04:12
  • · `mpstat -P ALL 1 5` shows `%usr` between 96 and 99 on both vCPUs and `%soft` under 2; `%steal` is zero
  • · `top -bn1` shows `bgpd` and `zebra` occupying most of one vCPU each on a two-vCPU virtual machine
  • · `show interfaces ethernet eth1` shows no error or drop counters incrementing and no link-state change in 41 days
  • · `tcpdump -nn -i any udp port 3784` shows outbound BFD packets with inter-packet gaps that stretch well past 150 ms during each convergence burst
  • · The Provider A session on the same router, which takes a default route only, has not reset once
Diagnosis and resolutionclick to reveal

Root cause

BFD did not detect a failure; it manufactured one, and then the failure re-triggered the detector. The rollout three days earlier enabled BFD on both transit peers with the default profile - 50 ms transmit, 50 ms receive, multiplier 3 - which gives a 150 ms detection window. edge-03 is a two-vCPU virtual machine that accepts the full table from Provider B, and when that session converges, `bgpd` runs best-path over roughly a million prefixes on one vCPU while `zebra` installs the result into the kernel FIB on the other. For the duration of that burst there is no CPU left for `bfdd` to keep a 50 ms transmit timer, so the packets go out late, the remote end's 150 ms detector expires, and it declares the session down. That is what `Remote diagnostics: control detection time expired` records, and the direction matters: our own diagnostics read `ok`, so nothing was lost coming towards us. BGP then tears the session down, the session re-establishes, and re-establishment triggers a full-table download - the single most expensive convergence event this router has - which starves `bfdd` again. Provider B's maintenance at 04:12 supplied the first reset; from that moment the loop has been self-sustaining and has not needed the maintenance, which is why it continued for three and a half hours after the maintenance ended. The link was never at fault. The data plane never dropped a packet, which is exactly why the ICMP probe stayed green: it is answered in the kernel, and the kernel was never the busy part. The other ten routers are stable because they take a default route rather than a full table, so their convergence costs milliseconds instead of seconds.

Remediation

Break the loop before diagnosing the CPU, because the CPU burst is a consequence and it will not stop on its own. The cheapest action that ends the incident is to remove BFD from the Provider B neighbour on edge-03 and commit, under `commit-confirm` because the session being changed is a transit path. Say the cost out loud rather than treating it as free: without BFD the failure detector for that peer is the 180 second BGP hold time, so a genuine link failure will blackhole for up to three minutes until BFD is restored. That is a deliberate, time-boxed trade with a named owner and an end time, not a fix. Once the session has been stable for long enough to prove the loop is broken, restore fast detection from measurement rather than from the default: capture the actual inter-packet gaps with tcpdump during a convergence and choose an interval and multiplier whose product comfortably exceeds the worst gap observed, applied as a named profile rather than per-peer values. Then remove the exposure instead of tuning around it. edge-03 is a backup transit path and does not need a million prefixes to make its forwarding decisions; an inbound filter that accepts a default route from Provider B removes the convergence cost entirely, and it is a better answer than a bigger instance because it makes the router cheap to converge rather than merely fast enough to survive converging.

Verification

`show bfd peers` uptime must climb past the old 90 to 120 second flap cadence and keep climbing, and `Remote diagnostics` must stop recording control detection time expired - the remote diagnostic is the field that named the fault, so it is the field that has to clear. `show ip bgp summary` Up/Down must pass one hour, then one day. `journalctl -u bfdd --since` over the window must contain no state changes at all; a reduced flap rate is not a fixed loop, it is a slower one. Re-run `mpstat -P ALL 1 5` and confirm the router now has idle CPU in steady state, because a control plane with no headroom will fail the next event whatever the BFD interval is. Then run the test that actually proves it: in a maintenance window, reset the Provider B session deliberately with `clear ip bgp 198.51.100.2` and watch, with tcpdump running, whether the BFD session survives the re-convergence it causes. That is the exact event the loop fed on, and a configuration that has not been watched through one of them is a hypothesis. Finally confirm the prefix count is what the new inbound policy intends - if the filter is meant to accept a default route, `show ip bgp summary` should report a single-digit prefix count, and if it does not, the filter is not doing what the change record says it does.

Prevention

Choose a BFD detection time that exceeds the worst control-plane scheduling jitter the device actually exhibits, and measure that jitter rather than assuming it. The default 150 ms window is a statement about link quality on a device with CPU to spare; it is not a safe default on a small control plane holding a large table. More generally, never attach a fast failure detector to a device whose failure response is expensive without bounding the cost of that response - the coupling between detection speed and convergence cost is what closes the loop, and it closes silently. Roll BFD profiles out per class of router rather than per fleet, and hold each class through one deliberate convergence before moving on; this incident would have been caught in the lab by resetting one session. Alert on BFD session uptime resets rather than only on BGP session state, because BFD is the earlier and more specific signal and it names the direction of the loss. Keep the data-plane and control-plane questions separate in monitoring: the ICMP probe was green for four hours through a total outage of the routing session, and it was not wrong, it was answering a different question. Finally, size what a router accepts to what it can converge. A backup transit that hands a full table to a two-vCPU virtual machine has built a control-plane load with no operational benefit, and every failure mode of that router gets worse in proportion.

Reported symptoms

At 08:30 there are three tickets open and none of them mentions the same subsystem.

The voice team has bursts of one-way audio and dropped calls, several times an hour, first noticed around 04:15. The NOC has an eBGP session to Provider B on edge-03 that resets roughly every 90 to 120 seconds; the Provider A session on the same router has not moved. The platform team has a CPU alert on edge-03 with bgpd at the top of top, and has asked for a maintenance window to reboot the router onto a larger instance.

Four facts make it hard to point at any of these.

  • The link is clean. show interfaces ethernet eth1 shows no link-state transitions in 41 days, no CRC errors, no discards, no input or output drops incrementing.
  • The path works. The monitoring system’s ICMP probe across that same link has been 100% successful at under 2 ms for the entire period, including during the resets.
  • The trigger has gone. Provider B ran planned maintenance from 04:00 and confirms it completed at 05:00. The flapping did not stop at 05:00. It is still going at 08:30, and Provider B’s NOC says the resets are being initiated from our side.
  • Ten identical routers are fine. The BFD rollout three days ago applied the same profile to eleven routers. Only edge-03 flaps, and it flapped for three days without incident before this morning.

Evidence provided

Read-only / SafeProvider B up for 74 seconds and holding a full table; Provider A up for three days on a default route
vyos@edge-03:~$ show ip bgp summary
Neighbor        V    AS   MsgRcvd  MsgSent  Up/Down   State/PfxRcd
198.51.100.2    4  64502   4127885    2214   00:01:14       981204
203.0.113.6     4  64501      8912     8874   3d02h14m            1

Illustrative output

Read-only / Saferead the two diagnostic lines against each other
vyos@edge-03:~$ show bfd peers
BFD Peers:
peer 198.51.100.2 vrf default
	Status: up
	Uptime: 71 second(s)
	Diagnostics: ok
	Remote diagnostics: control detection time expired
	Peer Type: configured
	Local timers:
		Detect-multiplier: 3
		Receive interval: 50ms
		Transmission interval: 50ms
	Remote timers:
		Detect-multiplier: 3
		Receive interval: 50ms
		Transmission interval: 50ms

Illustrative output

Read-only / Safeuserspace saturation on both vCPUs, and almost no softirq
vyos@edge-03:~$ mpstat -P ALL 1 5
08:31:02  CPU    %usr   %nice   %sys  %soft  %steal  %idle
08:31:03  all   97.50    0.00   1.00   1.00    0.00   0.50
08:31:03    0   98.00    0.00   1.00   1.00    0.00   0.00
08:31:03    1   97.00    0.00   1.00   1.00    0.00   1.00

Illustrative output

Read-only / Safetwo daemons, two vCPUs, and bfdd getting whatever is left
vyos@edge-03:~$ top -bn1 | head -12
  PID USER      PR  NI    VIRT    RES   %CPU  COMMAND
1893 frr       20   0  2418924 1984112   98.7  bgpd
1887 frr       20   0   198432  41220    91.4  zebra
1901 frr       20   0    41880   9204     0.7  bfdd

Illustrative output

The BFD daemon’s own log has the cadence in it:

Aug 18 08:26:11 edge-03 bfdd[1901]: state-change: [mhop:no peer:198.51.100.2] up -> down
Aug 18 08:26:13 edge-03 bfdd[1901]: state-change: [mhop:no peer:198.51.100.2] down -> up
Aug 18 08:28:02 edge-03 bfdd[1901]: state-change: [mhop:no peer:198.51.100.2] up -> down
Aug 18 08:28:04 edge-03 bfdd[1901]: state-change: [mhop:no peer:198.51.100.2] down -> up

And a packet capture on the BFD port shows the gaps directly: outbound control packets leaving on time at 50 ms intervals in steady state, and gaps stretching past 150 ms every time the session converges.

Work the evidence before reading on

Nothing here is a link problem, and no daemon has crashed. Work these questions before reading the root cause.

  1. Diagnostics: ok and Remote diagnostics: control detection time expired are two different statements. Which end stopped hearing the other, and what does that rule out?
  2. The ICMP probe is green and the BFD session is not. Both cross the same link in both directions. What is different about how the two are answered?
  3. Provider B’s maintenance ended at 05:00 and the flapping did not. What, other than the maintenance, could be supplying the next reset?
  4. Ten routers got the identical profile and are stable. Compare the two lines of show ip bgp summary above and name the difference between this router’s two sessions.

Before continuing: the BFD detection window is 150 ms. What does this router spend more than 150 ms doing, and what causes it to do that?

Root cause

1. BFD did not detect a failure; it manufactured one

The rollout enabled BFD on both transit peers with the default profile: 50 ms transmit interval, 50 ms receive interval, multiplier 3. Detection time is interval multiplied by multiplier, so the peer declares the session down after 150 ms of silence.

Remote diagnostics: control detection time expired says the remote end’s detector expired — the remote stopped receiving our packets. Our own Diagnostics: ok says we never stopped receiving theirs. So this is not a link that drops packets; a link that drops packets drops them in both directions and both diagnostics would say so. Something on this router stopped sending on time.

2. Two daemons, two vCPUs, and a third with nothing left

edge-03 is a two-vCPU virtual machine and it accepts the full table from Provider B. When that session converges, bgpd runs best-path over roughly a million prefixes and hands the result to zebra, which installs it into the kernel FIB. mpstat shows the consequence: 97 to 99% %usr on both vCPUs, and %soft under 2%, so this is userspace saturation and not packet-processing saturation.

bfdd is a separate process at the same scheduling priority as the two that are saturating the box. Its 50 ms transmit timer is a userspace timer, and a userspace timer on a fully contended run queue does not fire at 50 ms. Three consecutive slips is 150 ms, and 150 ms is the whole detection budget.

3. The loop closes on itself

This is the part that explains why the incident outlived its trigger.

BFD signals bgpd, bgpd tears the session down, and the session re-establishes. Re-establishment means downloading the full table again — the single most expensive convergence event this router performs. That burst starves bfdd again, the remote detector expires again, and the cycle repeats on the cadence of a full-table download, which is the 90 to 120 seconds in journalctl.

Provider B’s maintenance at 04:12 supplied the first reset. Every reset after it was supplied by the previous one. The loop has not needed the maintenance since 04:13, which is precisely why it continued for three and a half hours after the maintenance ended, and why asking Provider B to “check their side again” produces nothing.

4. Why the probe stayed green, and why ten routers did not flap

An ICMP echo is answered by the kernel. The kernel was never the busy part — %soft is under 2% — so the probe was answered on time throughout, and it was not wrong. It was answering “is the path alive”, which it was. Nobody was asking “can the control plane keep a 50 ms timer”, and that is the question BFD asks.

The other ten routers take a default route from their upstreams. Their convergence costs milliseconds, so bfdd never loses a slot and the same profile is entirely safe on them. The profile is not the fault either. The combination of that profile with this router’s table size and this router’s CPU is.

Resolution

  1. Break the loop before investigating the CPU. The CPU burst is a consequence of the loop, and the loop will not stop while it is feeding itself. Anything that diagnoses first and stabilises second extends the outage.
  2. Remove BFD from the Provider B neighbour on edge-03: delete protocols bgp neighbor 198.51.100.2 bfd, applied with commit-confirm 10 because the session being changed is a transit path and the router is reachable across it.
  3. State the cost of that action rather than treating it as free. The failure detector for that peer is now the 180 second BGP hold time, so a genuine link failure will blackhole for up to three minutes. This is a deliberate, time-boxed trade: it needs a named owner and an end time, and it is not the fix.
  4. Confirm the loop is broken before touching anything else. The session should establish once, download the table once, and stay up. Watch for at least three times the old flap cadence before believing it.
  5. Measure the jitter instead of guessing an interval. Run tcpdump -nn -i any udp port 3784 while deliberately resetting the session in a window, and read the worst outbound inter-packet gap during the convergence. That number, not the default, is the input to the profile.
  6. Restore fast detection as a named profile whose detection time comfortably exceeds the measured worst gap, and reference the profile from the neighbour rather than setting per-peer values, so the reasoning lives in one place and the next router inherits it.
  7. Remove the exposure rather than tuning around it. edge-03 is a backup transit path: apply an inbound prefix filter that accepts a default route from Provider B instead of a full table. This makes the router cheap to converge rather than merely fast enough to survive converging, and it fixes every other failure mode that scales with table size.
  8. Only then consider the instance size. More vCPU raises the threshold at which the loop closes; it does not remove the coupling, and it is the more expensive of the two answers.
  9. Re-check the other ten routers against the same question - table size against control-plane capacity - rather than against "did it flap". Nine of them may simply not have been triggered yet.

Verification

  1. The remote diagnostic clears. show bfd peers must stop reporting Remote diagnostics: control detection time expired. That field named the fault, so it is the field that has to clear; an uptime that looks healthier is not the same statement.
  2. The uptimes climb. BFD session uptime passes three times the old flap cadence, and show ip bgp summary Up/Down passes one hour and then one day.
  3. The log is silent, not quieter. journalctl -u bfdd --since over the window contains no state changes at all. A reduced flap rate is a slower loop, not a broken one.
  4. The control plane has headroom. mpstat -P ALL 1 5 in steady state shows idle CPU. A control plane with no headroom will fail the next event whatever the BFD interval is, and the interval will get the blame again.
  5. The fix survives the event it was built for. In a maintenance window, reset the session deliberately with clear ip bgp 198.51.100.2, with tcpdump running on UDP 3784, and confirm the BFD session stays up through the re-convergence and that the worst inter-packet gap stays inside the new detection budget.
  6. The inbound policy does what the change record says. If the filter is meant to accept a default route, show ip bgp summary reports a single-digit prefix count for that peer. If it still reports hundreds of thousands, the filter is configured but not applied.
  7. Forwarding still works after the filter. show ip route 0.0.0.0/0 shows the default via Provider B as a usable path, and a traceroute from behind the router still leaves by the intended provider. A filter that fixes the control plane by removing reachability is not a fix.

Prevention

  • Set the detection time from measured jitter, not from the default. 150 ms is a statement about link quality on a device with CPU to spare. On a small control plane holding a large table it is a statement about the scheduler, and the scheduler will lose.
  • Never attach a fast detector to an expensive recovery without bounding the recovery. The coupling between detection speed and convergence cost is what closes the loop, and nothing in the configuration displays that coupling.
  • Roll detection profiles out per class of router, not per fleet, and hold each class through one deliberate convergence. A single session reset in the lab reproduces this entire incident.
  • Alert on BFD session uptime resets, not only on BGP state. BFD is the earlier signal and the more specific one: its diagnostic fields name which direction lost packets, which is the fact the whole diagnosis turned on.
  • Keep the data-plane and control-plane questions separate in monitoring. The ICMP probe was green through four hours of a dead routing session and it was not faulty. A control plane needs a probe that fails when the control plane fails.
  • Size what a router accepts to what it can converge. A backup transit handing a full table to a two-vCPU virtual machine has bought a control-plane load with no forwarding benefit, and every failure mode of that router degrades in proportion to it.

Cross-course references

The Linux course’s XXXVIII-Linux-PerfFund covers the mpstat and pidstat evidence used here and the USE methodology behind it, and its XXII-Linux-NetTroubleshoot covers the interface counters that ruled the link out. The Observability course’s XCVIII-Observability-Troubleshooting covers the probe-design gap that let an ICMP check stay green through the outage. Within this course, vyos-xxxii-01-bfd-concept derives the interval-times- multiplier detection budget, vyos-xxxii-03-bfd-with-bgp covers how BFD signals bgpd to tear a session down, vyos-xxxi-03-flapping-session lists the four cadences a flapping BGP session can have, vyos-l-01-cpu-saturation distinguishes userspace from softirq saturation, and vyos-xxviii-01-prefix-list-concept covers the inbound filtering that removes the exposure.