BGP best-path troubleshooting — missing routes, suboptimal path, why MED is ignored

A best-path ticket arrives in one of three forms: “the route is missing”, “the wrong path is selected”, or “the upstream is not honouring my MED”. Each form has a canonical triage path; each leaves different evidence in the BGP table. This lesson walks the operator through the discipline: define the failure, gather evidence, hypothesise the cause, test the hypothesis, document.

The BGP best-path algorithm is deterministic — the algorithm produces the same output for the same inputs. If the operator sees an unexpected output, the inputs are different from what the operator expected. The troubleshooting is the operator’s job of finding the input that differs from the expectation.

The troubleshooting discipline

The discipline is the same as for any routing problem, with the best-path dimension layered on top:

1. Define the failure. “The wrong path is selected for 198.51.100.0/24” — not “the network is slow”. The first is testable; the second is a guess.
2. Gather evidence. Three layers — BGP table, received-routes, advertised-routes, FRR log. Each tells a different story.
3. Hypothesise the cause. State a specific candidate cause (“the MED is being ignored because the upstreams are in different ASes”) before changing configuration.
4. Test the hypothesis. Make one change; observe the result; commit or rollback.
5. Document the evidence. The next operator on call at 03:00 will want to know what you found.

flowchart TD
  S1["1. Define: what fails"] --> S2["2. Evidence: BGP table, received-routes, advertised-routes"]
  S2 --> S3["3. Hypothesise: which step and which attribute"]
  S3 --> S4["4. Test: one change at a time"]
  S4 --> S5["5. Document: ticket, runbook, post-mortem"]
  S5 -.-> S1

The lesson assumes the BGP session is up. If the session is down, the troubleshooting starts with the session (covered in vyos-xxxi-01-session-states).

Failure 1 — Missing route

The operator sees a prefix in the upstream’s show ip bgp but not in the local BGP table. The BGP session is up. The first question is “where is the route — in the received-routes, in the BGP table, or in neither?”.

vyos@vyos:~$ show ip bgp summary
Neighbor        V    AS   MsgRcvd  MsgSent  Up/Down   State/PfxRcd
192.0.2.2       4  65001      1247      892   03:45:12       42
198.51.100.1    4  65002      1238      876   03:45:09       38

The peers are up and advertising prefixes. The local router sees 42 prefixes from peer 192.0.2.2 and 38 from peer 198.51.100.1.

vyos@vyos:~$ show ip bgp 198.51.100.0/24
% Network not in table

The route is not in the BGP table. The first question: did the peer send it?

vyos@vyos:~$ show ip bgp neighbors 192.0.2.2 received-routes | grep 198.51.100.0/24

received-routes is the pre-policy view: what the peer actually sent, before any inbound filter ran. routes is the post-policy view: what survived. The pair is the whole diagnosis — in received-routes but not in routes means your filter dropped it; in neither means the peer never sent it, so the problem is at the far end.

The operator then reads the neighbour’s policy:

vyos@vyos:~$ show configuration commands | match 'neighbor 192.0.2.2'

show ip bgp neighbors 192.0.2.2 names the applied route-map and prefix-list in its own output, and it is worth reading for one detail the configuration cannot tell you: FRR marks a policy name it cannot resolve with a leading *, so a route-map that was applied but never defined shows up there and nowhere else.

Here the inbound filter is a prefix-list, so the operator checks it:

vyos@vyos:~$ show configuration commands | grep -A5 "PL-INBOUND-192.0.2.2"
set policy prefix-list PL-INBOUND-192.0.2.2 rule 10 action 'permit'
set policy prefix-list PL-INBOUND-192.0.2.2 rule 10 prefix '203.0.113.0/24'

The prefix-list only permits 203.0.113.0/24. The route 198.51.100.0/24 is denied (the implicit deny at the end of the prefix-list). The operator found the bug.

The fix is to add the prefix to the prefix-list:

configure
set policy prefix-list PL-INBOUND-192.0.2.2 rule 20 action 'permit'
set policy prefix-list PL-INBOUND-192.0.2.2 rule 20 prefix '198.51.100.0/24'
commit
save

Failure 2 — Suboptimal path selected

The operator sees a prefix in the BGP table but the best-path (>) winner is not the expected path. The first question is “which step decided this?”.

vyos@vyos:~$ show ip bgp 198.51.100.0/24
Paths: (3 available, best #1)
  Path 1 (best):
    192.0.2.2 from 192.0.2.2 (10.255.0.1)
      Origin IGP, metric 0, localpref 100, weight 0, external
  Path 2:
    198.51.100.1 from 198.51.100.1 (10.255.0.2)
      Origin IGP, metric 0, localpref 100, weight 0, external
  Path 3:
    203.0.113.5 from 10.0.0.5 (10.255.0.5)
      Origin incomplete, metric 0, localpref 100, weight 0, internal

The operator expected path 2 to win (per the route-map that set Local Preference 200 on it). The actual winner is path 1. The walk:

• Step 1 — Weight. All paths have weight 0. Tie.
• Step 2 — Local Preference. All paths have localpref 100. The route-map that set Local Preference 200 is not visible. Tie.

The walk falls through to step 3 and beyond. The Local Preference is not set on path 2 — the route-map is not applied. The hypothesis: the route-map is configured but not applied to the peer.

vyos@vyos:~$ show configuration commands | match 'neighbor 198.51.100.1'
set protocols bgp neighbor 198.51.100.1 address-family ipv4-unicast route-map export 'RM-LP-198.51.100.1'
set protocols bgp neighbor 198.51.100.1 remote-as '65002'

There it is: the route-map exists and is applied, but as export. Local Preference is a non-transitive attribute that only means something to routers inside your own AS, so setting it on the way out to an eBGP peer changes nothing you can observe. The fix is to apply it on import:

configure
delete protocols bgp neighbor 198.51.100.1 address-family ipv4-unicast route-map export
set protocols bgp neighbor 198.51.100.1 address-family ipv4-unicast route-map import 'RM-LP-198.51.100.1'
commit
save

Two shapes of this command are worth holding onto, because a config written for an older release will not commit on 1.5:

• The AS number is not part of the path any more. It is declared once, as set protocols bgp system-as 64512, and peers live under set protocols bgp neighbor ....
• Policy is per address family and the directions are named import and export, not in and out.

A route-map change does not re-evaluate routes that are already in the table. Clear the session softly to make it take effect:

vyos@vyos:~$ clear ip bgp 198.51.100.1 soft in

The decision tree for “wrong path selected”:

flowchart TD
  A["Wrong path is `>` best"] --> B["Step 1: Weight<br/>(look for set weight route-map)"]
  B --> C["Step 2: Local Preference<br/>(look for set local-preference route-map)"]
  C --> D["Step 3: Locally Originated"]
  D --> E["Step 4: AS Path<br/>(look at Path field)"]
  E --> F["Step 5: Origin<br/>(look at Origin field)"]
  F --> G["Step 6: MED<br/>(only if same AS)"]
  G --> H["Step 7: eBGP over iBGP"]
  H --> I["Step 8: IGP cost to next-hop"]
  I --> J["Step 9-11: deterministic tiebreakers"]
  J --> K["The step that decided is the first one with a difference"]

The operator walks the tree in order. The first step that produces a difference is the one that decided.

Failure 3 — MED is ignored

The operator sets MED on routes advertised to upstream A and expects the upstream to prefer the lower MED. The upstream does not — the route selection is unchanged. The cause: the MED is being compared between paths from different ASes, which is the default behaviour to NOT do.

# Local configuration
set policy route-map MED-LOW rule 10 action 'permit'
set policy route-map MED-LOW rule 10 set metric '50'
set policy route-map MED-HIGH rule 10 action 'permit'
set policy route-map MED-HIGH rule 10 set metric '100'
set protocols bgp neighbor 192.0.2.2 address-family ipv4-unicast route-map export 'MED-LOW'
set protocols bgp neighbor 198.51.100.1 address-family ipv4-unicast route-map export 'MED-HIGH'

The two neighbours are in different ASes (192.0.2.2 is in AS 65001, 198.51.100.1 is in AS 65002). The MED is set on the outbound route-maps. The MED is sent to the upstream. The upstream’s best-path algorithm sees two routes for the same prefix:

• Route 1: from AS 65001 (peer 192.0.2.2), MED 50
• Route 2: from AS 65002 (peer 198.51.100.1), MED 100

The leftmost AS is different (65001 vs 65002). The MED is not compared. The upstream falls through to step 7 (eBGP over iBGP — both external), step 8 (IGP cost), step 9 (lowest router-id).

The fix is to recognise that MED only works between paths from the same AS. The operator who wants to influence the upstream’s selection when the upstreams are in different ASes must use AS Path prepending (lesson vyos-xxvii-03-as-path-prepending) or communities (lesson vyos-xxvi-06-communities).

The upstream could enable always-compare-med on its own routers and then the comparison would happen — on VyOS that is set protocols bgp parameters always-compare-med — but the upstream’s routers are not yours, and asking a transit provider to change a global best-path parameter for one customer is not a plan.

Failure 4 — Different best-path on different routers

The operator’s AS has two border routers. The two routers see the same paths for a prefix but pick different best-path winners. The cause: the default MED comparison is order-dependent. The router that sees the lower-MED path first wins.

# Router R1
vyos@vyos:~$ show ip bgp 198.51.100.0/24
  Path 1 (best): from 192.0.2.2, MED 50
  Path 2: from 198.51.100.1, MED 100

# Router R2
vyos@vyos:~$ show ip bgp 198.51.100.0/24
  Path 1 (best): from 198.51.100.1, MED 100
  Path 2: from 192.0.2.2, MED 50

R1 sees the lower-MED path first and picks it. R2 sees the higher-MED path first and picks it. The two routers have different best-path winners.

The fix is bgp bestpath deterministic-med on both routers:

configure
set protocols bgp parameters deterministic-med
commit
save

The knob groups the paths by neighbouring AS and compares within each group before comparing across groups, so the outcome no longer depends on the order the paths happened to arrive in. Both routers pick the same path.

Note where the node sits: deterministic-med and always-compare-med are directly under protocols bgp parameters, while the MED-handling flags such as missing-as-worst are under protocols bgp parameters bestpath med. They read as one family and are not one subtree, which is a common source of a command that will not complete.

After the change, the operator clears the sessions softly (clear ip bgp * soft in) or waits for the next BGP UPDATE for the change to take effect.

Failure 5 — Missing MED treated as most preferred

The upstream sends a route without MED. The operator’s router treats the missing MED as MED 0 (the most preferred) and installs the route. The operator expected the route to be deprioritised.

The default behaviour is to treat missing MED as MED 0. The operator who wants “no MED is the worst MED” configures bgp bestpath med missing-as-worst:

configure
set protocols bgp parameters bestpath med missing-as-worst
commit
save

The knob is global. Every path with missing MED is treated as MED 4294967295. The operator who has a mix of peers (some send MED, some do not) and wants the “no MED” peers to be deprioritised gets the expected effect.

Reading the show ip bgp output to identify the deciding step

The operator who sees a > winner and wants to know which step decided walks the eleven steps in order and looks at the attributes:

vyos@vyos:~$ show ip bgp 198.51.100.0/24
Paths: (3 available, best #2)
  Path 1: 192.0.2.2 from 192.0.2.2 (10.255.0.1)
    Origin IGP, metric 50, localpref 100, weight 0, external
  Path 2 (best): 198.51.100.1 from 198.51.100.1 (10.255.0.2)
    Origin IGP, metric 20, localpref 200, weight 0, external
  Path 3: 203.0.113.5 from 10.0.0.5 (10.255.0.5)
    Origin incomplete, metric 0, localpref 100, weight 0, internal

The walk:

• Step 1 — Weight. All paths have weight 0. Tie.
• Step 2 — Local Preference. Path 2 has localpref 200; paths 1 and 3 have 100. Path 2 wins at step 2.

The walk terminates. The algorithm never evaluates steps 3-11. The operator who sees “best #2” and “localpref 200” on path 2 knows step 2 decided.

If the operator changed the localpref on path 2 to 100 (matching the others), the walk would continue:

• Step 3 — Locally Originated. None of the paths are locally originated. Tie.
• Step 4 — AS Path. All paths have the same AS Path length (the diagram doesn’t show the full path, but assume they are all 1-hop). Tie.
• Step 5 — Origin. Paths 1 and 2 are IGP; path 3 is incomplete. Path 1 and 2 win at step 5; path 3 loses.
• Step 6 — MED. Paths 1 and 2 are from different ASes (assuming 192.0.2.2 is in AS 65001 and 198.51.100.1 is in AS 65002). The MED is not compared. Tie.
• Step 7 — eBGP over iBGP. Both paths 1 and 2 are external. Tie.
• Step 8 — IGP cost to next-hop. Whichever has the lower IGP cost to its next-hop wins.

The walk would continue until one step produces a difference. The operator who predicts the walk is the operator who can debug the best-path selection without changing configuration.

The FRR log as evidence

Be clear about what this log does and does not contain, because it is where most best-path investigations waste their first hour.

FRR does not log best-path decisions at its normal log level. A prefix whose winner changed at 03:00 leaves no line saying so and no line saying why. What the log carries is the events around the decision, and those are usually enough to date it: session establishment and teardown, NOTIFICATIONs and their subcodes, prefix-limit warnings, graceful-restart transitions, and errors from installing the resulting route into the kernel.

sudo journalctl -u frr --since '-2 hours' | grep -i bgp

Correlating that timeline with the commit history is the actual technique — show system commit gives you the times your side changed, and the FRR log gives you the times the peer’s side did.

If you need the decision itself, you have to ask for it. From vtysh, per-prefix best-path debugging exists and its output goes to the FRR log:

vyos@vyos:~$ vtysh
vyos# debug bgp bestpath 198.51.100.0/24

Turn it off with the matching no debug bgp bestpath 198.51.100.0/24 as soon as you have what you need. It is a debug, not configuration — it does not survive a restart and it does not appear in show configuration commands — and on a router carrying a full table it is expensive enough to matter.

How the result is validated

show ip bgp 198.51.100.0/24
show ip bgp 198.51.100.0/24 bestpath
show ip bgp neighbors 192.0.2.2 received-routes
show ip bgp neighbors 192.0.2.2 routes
show ip route 198.51.100.0/24
show configuration commands | match 'neighbor 192.0.2.2'

The first shows every candidate path and which one carries >. The second narrows it to the winner. The third and fourth are the pre-policy and post-policy views of what the peer sent — the pair that separates “they did not send it” from “we dropped it”, and the third needs soft-reconfiguration inbound to return anything. The fifth confirms the winner actually reached the RIB, which is a separate question from winning the BGP decision. The sixth is the configuration side of the same story.

The same views are available under their FRR names, which is what you will find in FRR’s own documentation:

vyos@vyos:~$ vtysh
vyos# show bgp ipv4 unicast 198.51.100.0/24
vyos# show bgp ipv4 unicast 198.51.100.0/24 bestpath
vyos# show bgp ipv4 unicast neighbors 192.0.2.2 received-routes

How it fails

The production failure modes the engineer must recognise:

• The wrong route-map is applied. The operator applies the route-map to the wrong peer or in the wrong direction. The fix is to apply the route-map to the correct peer in the correct direction.
• The route-map has a wrong match clause. The route-map matches a different prefix than the operator expected. The fix is to correct the match clause.
• The route-map has a wrong set clause. The route-map sets the wrong attribute (e.g., set metric when the operator wanted set local-preference). The fix is to correct the set clause.
• The route-map is denied instead of permitted. The route-map has action 'deny' instead of action 'permit'. The route is rejected at the filter. The fix is to change the action to permit.
• The new BGP knob changes the algorithm. The operator configures bgp always-compare-med and the best-path winners change across the AS. The fix is to coordinate the knob change across all routers.
• The FRR log is not consulted. The operator changes configuration without reading the FRR log. The change has an unexpected effect and the operator cannot find the cause. The fix is to read the FRR log first.

Rollback

Best-path troubleshooting changes are configuration changes. The standard rollback path applies:

• compare before commit to see the configuration change.
• commit-confirm <timeout> for any remote change.
• rollback N; commit; save to revert to the previous configuration.
• load /config/archive/<known-good-file>; commit; save to revert to a specific snapshot.

The operator who is troubleshooting a wrong-path-wins failure must also know that the rollback restores the previous best-path behaviour. The standard rollback is rollback 1; commit; save.

Production discipline

Cross-course references

The Linux course’s XIX-Linux-NetFoundations covers the kernel FIB. The OPNsense course’s XXX-OPNsense-DynamicRouting covers the same FRR BGP troubleshooting on the firewall side. The BGP lessons vyos-xxvii-01-best-path-algorithm through vyos-xxvii-05-igp-cost-tiebreak cover the individual steps and attributes. The lesson vyos-xxxi-03-flapping-session covers the broader route-not-selected troubleshooting. The Observability course’s XII-Observability-HostAgents covers scraping the FRR log for monitoring.

Quiz