Lab: BFD for Fast Failure Detection

Objective

By the end of this lab you will be able to say, for one BGP session, exactly how long it takes to notice that the peer has stopped forwarding — and you will have that number from the router rather than from a data sheet.

Three numbers appear along the way and they are all different. The session starts with a 180-second hold timer. Enabling BFD with no further configuration gets you roughly 900 milliseconds, not the 150 milliseconds the word “BFD” usually implies. Asking for 150 takes a profile, on both ends, and a look at what the two ends actually negotiated rather than at what you typed. Getting from the first number to the third is most of the work; knowing which one you are running is all of the value.

You will also break BFD twice, in the two ways that leave a router looking healthy: a probe that a firewall is eating, and a perfectly good BFD session that no BGP neighbour is listening to.

Architecture

The Lab 11 topology, unchanged. Three routers, three eBGP sessions, and a triangle — which is what makes a failover measurable at all, because every destination is reachable two ways.

                      AS 65001  site1
                      router-id 192.0.2.101
                      dum0 198.51.100.1/24
                        /                \
      eth1 192.0.2.1/30                   eth2 192.0.2.5/30
            /                                      \
  eth1 192.0.2.2/30                          eth1 192.0.2.6/30
   AS 65010  isp-a                            AS 65020  isp-b
   dum0 203.0.113.1/25                        dum0 203.0.113.129/25
        \                                          /
      eth2 192.0.2.9/30 ------------------- eth2 192.0.2.10/30

eth0 on all three routers is management and is never touched by any task here. The three transit links live on isolated bridges with no physical uplink.

The destination that matters for the measurement is 203.0.113.1, which isp-a originates. site1 reaches it directly across the isp-a session today, and can reach it the long way round through isp-b when that session goes away. The gap between those two states is the thing you are timing.

Requirements

• The completed Lab 11 topology: three routers, three eBGP sessions Established, site1 originating 198.51.100.0/24, isp-a originating 203.0.113.0/25 and isp-b originating 203.0.113.128/25. Task 1 rebuilds it from a fresh install if you no longer have it.
• 6 GB RAM and 24 GB disk across three VMs.
• VyOS 1.5 LTS with FRR 10.x. This matters more here than in most labs: on 1.3 the BFD timers were flat leaves (interval, min-rx, multiplier) and on 1.4 and later they are children of an interval node. A 1.3-era runbook does not commit, and the error names a configuration path rather than a timer, which is easy to misread.
• Console access to all three VMs. Tasks 5 and 6 tear down BGP sessions on purpose. Nothing touches eth0, so your management path should survive — but a BFD misconfiguration can drop a session in under a second, which is quicker than you can react.
• A stopwatch or a second terminal, and about 100 minutes.

Scenario

The site is dual-homed and the routing works. What nobody has measured is what happens between the moment a provider stops forwarding and the moment this router notices.

The question came up because of an incident that produced no alerts: isp-a had a fault two hops beyond the handoff, the physical link to the site stayed up throughout, and the site kept handing traffic to a next hop that was not going anywhere. BGP was Established for the entire event, because from BGP’s point of view nothing had happened — the TCP session was fine, the keepalives were arriving, and the hold timer never came near expiry.

Your job is to put a number on that gap, then shrink it, then prove the new number is real.

Tasks

Task 1: Confirm the topology, or rebuild it

If the Lab 11 routers are still running, confirm all three sessions and skip to Task 2:

$ show bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 192.0.2.101, local AS number 65001 vrf-id 0
BGP table version 3
RIB entries 5, using 960 bytes of memory
Peers 2, using 1447 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
192.0.2.2       4      65010       184       186        0    0    0 01:31:12            2        1
192.0.2.6       4      65020       181       183        0    0    0 01:30:58            2        1

Whichever route you took, take a restore point on all three routers before you change anything: configure, save /config/pre-lab-14.boot, exit.

Task 2: Measure what you have today

Read the timers the session is running on:

$ show ip bgp neighbors 192.0.2.2

BGP neighbor is 192.0.2.2, remote AS 65010, local AS 65001, external link
BGP version 4, remote router ID 192.0.2.102, local router ID 192.0.2.101
BGP state = Established, up for 01:34:07
Last read 00:00:21, Last write 00:00:19
Hold time is 180, keepalive interval is 60 seconds
Configured hold time is 180, keepalive interval is 60 seconds

180 seconds. Note that there is no BFD block anywhere in that output — its presence later is what will tell you bgpd has actually registered the session, as opposed to the configuration merely mentioning BFD.

Now produce the failure the scenario describes. Start a continuous ping on site1 and leave it running in its own terminal:

ping 203.0.113.1 count 200

In a second terminal, on isp-a, take down the site-facing interface. The far end stops forwarding and site1’s own link light stays on, which is the whole point of the test:

configure
set interfaces ethernet eth1 disable
commit

Watch what happens on site1, and watch what does not.

The ping stops answering immediately. show interfaces ethernet eth1 still reports the interface up, because the bridge keeps the local carrier alive — and that is not a lab artefact, it is the ordinary production case. A fault beyond the first switch, a provider-side outage, a peer that has stopped forwarding but is still powered: in all of them your link light stays on.

Check the session repeatedly while the ping fails:

show bgp summary
show ip route 203.0.113.0/25

The session is still Established. The route is still there, still via 192.0.2.2, still selected. Nothing has failed as far as this router is concerned, and nothing will for up to three minutes.

Restore the link on isp-a — configure, delete interfaces ethernet eth1 disable, commit — and confirm the ping recovers. Record the first row of your table: no BFD, detection 180 s.

Task 3: Turn BFD on, and read the number you actually got

Enable BFD on both ends of the site1–isp-a session. BFD is bilateral: one side alone gets you a session in down and no protection at all.

On site1:

configure
set protocols bgp neighbor 192.0.2.2 bfd
commit
save

On isp-a:

configure
set protocols bgp neighbor 192.0.2.1 bfd
commit
save

Now read the state. VyOS documents one BFD operational command and it carries the whole diagnostic load:

$ show bfd peers

BFD Peers:
peer 192.0.2.2 vrf default
	ID: 1721004311
	Remote ID: 2884109553
	Active mode
	Status: up
	Uptime: 0 day(s), 0 hour(s), 1 minute(s), 6 second(s)
	Diagnostics: ok
	Remote diagnostics: ok
	Peer Type: configured
	Local timers:
		Detect-multiplier: 3
		Receive interval: 300ms
		Transmission interval: 300ms
	Remote timers:
		Detect-multiplier: 3
		Receive interval: 300ms
		Transmission interval: 300ms

Read the timer blocks, not the fact that it says up. FRR’s defaults are a 300 ms transmit interval, a 300 ms receive interval and a detect multiplier of 3. Detection time is the multiplier times the effective receive interval, so this session detects failure in roughly 900 milliseconds — not the 150 that the phrase “sub-second BFD” tends to put in a change ticket.

900 ms is a genuine improvement of two orders of magnitude over 180 seconds and is worth having on its own. It is simply not the number somebody who wrote “sub-150ms failover” in a design document is budgeting for.

Confirm that bgpd has registered the session, which is a different question from whether the session is healthy:

show ip bgp neighbors 192.0.2.2

There is now a BFD block in that output, naming the type (single hop), the negotiated intervals, the detect multiplier and the status. Its presence is the evidence. A healthy BFD session that no BGP neighbour is listening to is entirely possible, and Task 6 builds one on purpose.

Record the second row: BFD, default timers, detection ≈ 900 ms.

Task 4: Ask for 150 ms — first on one end, then on both

Define a profile on site1 only, and attach it:

configure
set protocols bfd profile FAST description 'directly attached transit, low latency'
set protocols bfd profile FAST interval transmit '50'
set protocols bfd profile FAST interval receive '50'
set protocols bfd profile FAST interval multiplier '3'
set protocols bgp neighbor 192.0.2.2 bfd profile 'FAST'
commit
save

Note the shape of those three lines, because it is the part most often written from memory and rejected. On 1.4 and later the timers are children of an interval node — interval transmit, interval receive, interval multiplier. There is no bare interval leaf, no min-rx, and no top-level multiplier; those spellings belong to VyOS 1.3, to FRR’s own configuration language, and to Cisco respectively.

Re-read the session:

show bfd peers

The local block now says transmit 50 and receive 50. The remote block still says 300 and 300, and the detection time has barely moved. Work out why before reading on.

Each direction is resolved independently, to the slower of the sender’s transmit interval and the receiver’s receive interval. Your 50 ms transmit meets a peer that will only accept 300 ms, so packets go at 300 ms. In the other direction isp-a transmits at 300 ms regardless of what you are willing to receive. Both directions are governed by the end you have not configured.

This is the single most common surprise in a BFD deployment, and it is why the configured values are not the answer to “how fast is this session”. Now configure the far end:

configure
set protocols bfd profile FAST description 'directly attached transit, low latency'
set protocols bfd profile FAST interval transmit '50'
set protocols bfd profile FAST interval receive '50'
set protocols bfd profile FAST interval multiplier '3'
set protocols bgp neighbor 192.0.2.1 bfd profile 'FAST'
commit
save

Re-read show bfd peers on both routers. Both timer blocks should now report 50/50/3, giving a detection time of about 150 ms. Record the third row of the table, and record it from the output rather than from the profile.

Two more knobs are worth knowing about even though this lab does not need them. set protocols bgp neighbor 192.0.2.2 bfd check-control-plane-failure changes how a local control-plane restart is treated, which matters before a maintenance window where you restart routing on purpose and would rather the peer did not read it as the link going away. And a peer that is not directly attached needs set protocols bfd peer <ip> multihop with a source address, plus ebgp-multihop on the BGP neighbour — and multi-hop BFD uses UDP 4784 rather than 3784, which matters the moment there is a filter in the path.

Task 5: Measure the failover honestly

Start a ping on site1 and leave it running:

ping 203.0.113.1 count 100

Confirm the current path first:

show ip route 203.0.113.0/25

It resolves via 192.0.2.2 on eth1. Now note the wall-clock time and disable isp-a’s eth1 exactly as in Task 2. Then, on site1:

show ip route 203.0.113.0/25
show bgp summary

The route now resolves through isp-b, and the isp-a session shows an Up/Down timer of a few seconds. The ping lost one packet, maybe two.

That ping is your sanity check, not your measurement. At one packet per second it cannot tell 150 ms from 900 ms — it can only tell you the failover happened and roughly how much traffic it cost. For the number itself, read the journal:

# Run on site1.
sudo journalctl -u frr -o short-precise --since '-5 min' | grep -iE 'bfd|neighbor'

short-precise gives microsecond timestamps. You are looking for two lines: the BFD session leaving up, and the BGP neighbour leaving Established. The gap between them is the signalling, and it should be small; the gap between the last successful ping and the BFD event is the detection.

Restore isp-a’s eth1 and wait for the session to come back. Then run the whole test again with the profile removed from both ends (delete protocols bgp neighbor 192.0.2.2 bfd profile on site1, the matching line on isp-a) so that the session falls back to the 300/300/3 defaults, and compare the journal gaps. The difference between the two runs is the profile’s entire contribution, measured rather than assumed.

Put the profile back before continuing.

Task 6: Break BFD in the two ways that look healthy

6a — the probe nobody is delivering. On isp-a, drop inbound single-hop BFD:

configure
set firewall ipv4 input filter rule 10 action 'drop'
set firewall ipv4 input filter rule 10 description 'LAB ONLY - blackhole single-hop BFD'
set firewall ipv4 input filter rule 10 protocol 'udp'
set firewall ipv4 input filter rule 10 destination port '3784'
commit

Within a second or so the session is gone. Read it from both sides:

show bfd peers
show bgp summary

show bfd peers reports Status: down, and the Diagnostics and Remote diagnostics fields say what each end saw. show bgp summary shows the session no longer Established. Confirm the packets are leaving but not arriving:

# Run on site1, in its own terminal.
sudo tcpdump -nni eth1 udp port 3784

Outbound BFD packets at the configured rate, nothing inbound.

The link is perfectly healthy. Traffic to 203.0.113.0/25 still gets there, the long way round through isp-b. What tore the session down was a firewall rule, and that is the honest cost of BFD: it is a new dependency on the path, and a change to that path can now take a working BGP session with it. Nothing about this is a reason not to run BFD. It is a reason to keep UDP 3784 — and 4784 wherever multi-hop is in use — in the same change-control conversation as the sessions that depend on it.

Remove the rule and confirm both the BFD session and the BGP session come back:

configure
delete firewall ipv4 input filter rule 10
commit
save

6b — the session nobody is listening to. This one is quieter. On site1, define a BFD peer directly and take the flag off the BGP neighbour:

configure
set protocols bfd peer 192.0.2.2 profile 'FAST'
delete protocols bgp neighbor 192.0.2.2 bfd
commit

Now:

show bfd peers
show configuration commands | match 'neighbor 192.0.2.2 bfd'

The first says the session is up with the timers you asked for. The second returns nothing, which settles it: there is no bfd clause on the neighbour, so nothing signals bgpd. The BFD block is also gone from show ip bgp neighbors 192.0.2.2.

Prove the consequence. Disable isp-a’s eth1 again and watch show bgp summary on site1: the session stays Established, exactly as it did in Task 2, while show bfd peers reports the failure within 150 ms. You have a probe that works and a protocol that is not subscribed to it.

This is the failure most likely to survive a review, because every individual piece of evidence looks good. The check that catches it is the one-liner above — grep the configuration for the bfd clause on the neighbour, rather than reading the BFD state and inferring.

Restore isp-a’s link, then put site1 back:

configure
delete protocols bfd peer 192.0.2.2
set protocols bgp neighbor 192.0.2.2 bfd profile 'FAST'
commit
save

Task 7: Protect a static route, and write the budget

BFD is not only for dynamic protocols, and the static case is where its behaviour is easiest to see. Add a fallback default route on site1, tracked by the same BFD session:

configure
set protocols static route 0.0.0.0/0 next-hop 192.0.2.2 bfd profile 'FAST'
set protocols static route 0.0.0.0/0 next-hop 192.0.2.2 distance '200'
commit
save

Two things to notice. The bfd node hangs off the next-hop, not off the route, so a route with several next-hops tracks each one independently. And bfdd now has two consumers for the same session: bgpd, which resets the peer when it goes down, and staticd, which withdraws the route. One probe, two subscribers, and that is the normal arrangement rather than a conflict.

Confirm the route is installed:

show ip route 0.0.0.0/0

Now disable isp-a’s eth1 one last time and look again. The static default is gone from the table — not present-but-unselected, gone. That is conditional installation, and it is the direct contrast with Lab 13, where a static default in an alternate table stayed installed with a dead next hop and blackholed a whole traffic class. The difference is that something is now watching the next hop.

Restore the link. Then write the deliverable that makes the rest of this worth having — the failover budget, in three lines:

• Detection, from your journal measurement.
• Teardown and withdrawal, the gap between the BFD event and the BGP neighbour event.
• Reconvergence, the time from the neighbour event until the route resolves through isp-b.

On this topology the first line dominates. On a full-table peer the third one does, and a design that budgets only for detection is budgeting for the smallest of the three.

Validation

• show bfd peers on site1 reports the 192.0.2.2 session up, with an Uptime longer than the interval you have been watching, Diagnostics: ok on both sides, and local and remote timer blocks that both read 50/50/3.
• show ip bgp neighbors 192.0.2.2 carries a BFD block for the peer with the same status — its presence, not just the BFD state, is the proof that bgpd registered the session.
• show configuration commands | match 'neighbor 192.0.2.2 bfd' returns the bfd profile 'FAST' line on site1, and the matching line for 192.0.2.1 exists on isp-a.
• Your table has three rows with a measured detection time in each: no BFD, BFD with defaults, BFD with the profile on both ends.
• The journal extract for at least one failover is in the journal, with short-precise timestamps for the BFD transition and the BGP neighbour transition, and the arithmetic written out.
• During Task 4, show bfd peers after the one-sided profile showed a local block of 50/50 against a remote block of 300/300, and you recorded that the detection time did not improve.
• During Task 6b, show bfd peers reported the session up while show bgp summary kept the neighbour Established through a link failure.
• show ip route 0.0.0.0/0 on site1 shows the BFD-tracked static default present with the link up and absent with it down.

Expected Outcome

One eBGP session protected by a BFD profile that both ends agreed to, with a detection time you measured rather than inherited, and a static default route on the same next hop that withdraws itself when the peer stops answering.

More usefully: a written failover budget for that session, and the evidence behind each of its three lines. The configuration that produces it is about eight commands. The part that survives contact with a design review is the number, the method that produced it, and the note saying which of the three components dominates.

The topology is left running and unchanged for the BGP break/fix scenarios and the capstone.

Troubleshooting

The session sits in down and tcpdump shows outbound packets with nothing coming back. Check the far end before checking anything local. The three causes, in the order they occur in practice: the peer has no BFD configured, a filter in the path is dropping UDP 3784 (or 4784 for multi-hop), or the two ends are in different VRFs. The vrf field on the peer line of show bfd peers settles the third one in a glance.

The profile is configured and the timers have not changed. Read which block did not change. If the local block moved and the remote did not, the far end is not configured — see Task 4. If neither moved, check that the profile is actually referenced from the neighbour rather than merely defined; a profile nothing points at is inert.

The commit is rejected with a configuration-path error on the timer lines. The 1.3 spellings. On 1.4 and later the leaves are interval transmit, interval receive and interval multiplier, all children of interval. min-rx does not exist on this CLI at all.

show bfd neighbors returns nothing. It is not a command. Use show bfd peers. Likewise journalctl -u bfdd — use journalctl -u frr.

The session flaps, with an Uptime of a second or two that resets each time you run show bfd peers. The link or the peer cannot sustain the packet rate. Either slow the intervals or raise the multiplier — detection time is the multiplier times the effective receive interval, so both levers move the same number — and change both ends, because a one-sided change may not move the transmit rate at all. This is unlikely on an idle lab bridge and common on a congested WAN, which is the point: BFD flaps on congestion, and a congested link is exactly where somebody will want to deploy it.

BGP drops but you cannot find a BFD event in the journal. Check that log-neighbor-changes is configured. Without it the neighbour transitions are not logged and the measurement in Task 5 has nothing to work with.

Cleanup

Everything here runs on isolated bridges, so cleanup means returning to a known state rather than restoring service.

Step 1. Make sure nothing is left disabled. On isp-a:

configure
delete interfaces ethernet eth1 disable
delete firewall ipv4 input filter rule 10
commit
save

Confirm with show bgp summary on all three routers that every session is Established and every prefix count is what it was in Task 1. A lab left with an interface disabled is the commonest way the next lab on the same topology starts with a mystery.

Step 2. To keep BFD in place for later work — the recommended path, since the break/fix scenarios and the capstone both benefit from it — stop here and save on each router. Note in the journal that the site1–isp-a session is running a 50/50/3 profile, so the next person is not surprised by a session that drops in 150 ms.

Step 3. To remove BFD but keep the topology, on site1:

configure
delete protocols static route 0.0.0.0/0
delete protocols bgp neighbor 192.0.2.2 bfd
delete protocols bfd profile FAST
commit
save

and on isp-a:

configure
delete protocols bgp neighbor 192.0.2.1 bfd
delete protocols bfd profile FAST
commit
save

Order matters in both blocks. A profile that a neighbour or a static route still references will not delete, which is the system doing you a favour — remove the reference first.

Step 4. To reset a router completely, load the file saved in Task 1:

configure
load /config/pre-lab-14.boot
compare
commit
save

compare before commit, every time. It is the only thing between you and a stored revision that turns out not to be the one you meant.

What You Learned

• BGP’s hold timer is a liveness check on the session, not on the path. You watched a router hand traffic to a dead next hop for a full minute with the session Established and the route selected, because nothing in BGP was looking at forwarding.
• bfd on its own buys about 900 ms, not 150. FRR’s defaults are 300/300/3 and they are perfectly reasonable defaults. They are simply not what a design document means when it says “sub-second BFD”, and the gap is a factor of six.
• The negotiated timers are the real ones, and they are not local. Each direction resolves to the slower of the sender’s transmit and the receiver’s receive, so a profile on one end alone changes almost nothing. Read show bfd peers; do not compute from the configuration.
• Measure from the log, not from the ping. A one-second ping cannot distinguish 150 ms from 900 ms. journalctl -u frr -o short-precise can, and it also shows you the teardown and the reconvergence as separate events — which is what turns one number into a budget.
• BFD introduces a dependency, and dependencies fail. A firewall rule on UDP 3784 tore down a healthy session in under a second. That is not an argument against BFD; it is an argument for keeping the probe’s ports in the same change conversation as the sessions that rely on it.
• A healthy BFD session nobody is subscribed to is the quietest failure of the three. show bfd peers looks perfect, the timers are right, and the protection is not there. The check is show configuration commands | match 'neighbor <ip> bfd', and the evidence in the protocol’s own output is the BFD block, not the BFD state.
• On a static route, BFD changes installation, not preference. The tracked route withdraws when the peer stops answering, which is the behaviour Lab 13’s untracked table could not produce. Administrative distance is a separate mechanism and still has to be set deliberately.