Skip to main content
RunBook Academy

← All break/fix scenarios in VyOS

advancedvyos-bgp-prefix~30 min

BGP Route Selected but Not Installed

Reported symptoms

  • ●From 01:15, roughly two in five Internet destinations are unreachable from every host in the estate; the rest are fine, and nobody can find what distinguishes the two sets
  • ●Both border routers can reach everything when tested from the routers themselves, so the first conclusion is that transit is healthy and the problem must be internal
  • ●Every BGP session is Established with an unchanged uptime, and the per-peer prefix counts are the same as yesterday
  • ●OSPF is fully converged on all five routers - every adjacency Full, no LSA churn, no alarm
  • ●`traceroute` from a client dies at the first core router, which is also the router the network team is most confident is healthy
  • ●The only change in the window was an automated configuration run whose diff removed a handful of static routes annotated as unused

Evidence

  • · On rtr-core-1, `show ip bgp 198.51.100.0/24` shows the prefix present and selected, with next hop 192.0.2.2
  • · On rtr-core-1, `show ip route 198.51.100.0/24` returns nothing at all
  • · On rtr-core-1, `show ip route 192.0.2.2` - the next hop itself - also returns nothing
  • · A destination that still works, 203.0.113.0/24, resolves through next hop 10.64.0.2, which is present in `show ip route`
  • · `show ip route ospf | match 192.0.2` is empty on every core router; `show ip route ospf | match 10.64.0` is not
  • · `show system commit` shows the overnight automation run, and three weeks earlier a change commented "ospf: stop redistributing WAN transit links"
  • · `show configuration commands | match redistribute` shows OSPF redistributing connected routes through a route-map that permits loopbacks and the 10.0.0.0/8 internal supernet
Diagnosis and resolutionclick to reveal

Root cause

Paths learned from ISP-A carry a NEXT_HOP of 192.0.2.2 - the address on rtr-edge-1's external link - and that attribute is passed unchanged across iBGP, because no router in the estate sets next-hop-self. A core router can therefore only install those paths if it can resolve 192.0.2.2 in its own routing table. For years it could, but not for the reason anyone believed: the resolution came from a legacy static summary for 192.0.2.0/24, a leftover from a decommissioned addressing plan, and not from the IGP. Three weeks ago a deliberate and correct hardening change stopped OSPF from redistributing WAN transit links, which removed 192.0.2.0/30 from the IGP. Nothing broke, because the static summary was still resolving the next hop. Last night an automated configuration run removed static routes annotated as unused, including that summary, and the last thing resolving 192.0.2.2 disappeared. The paths remain in the BGP table because BGP still has them and still considers them the best available; they are simply not installable, so the kernel FIB has no entry and the router drops the traffic. The split between working and failing destinations is not arbitrary: destinations whose best path is via rtr-edge-1 fail, and destinations whose best path is via rtr-edge-2 work, because rtr-edge-2's transit link is numbered out of 10.64.0.0/30 and slipped through the same route-map that correctly excluded the publicly numbered one.

Remediation

There are two real fixes and one that will make this happen again, and the choice belongs to whoever owns the hardening decision, not to whoever is on call. The structural fix is next-hop-self on rtr-edge-1's iBGP sessions, which rewrites the NEXT_HOP to rtr-edge-1's loopback - an address the IGP does carry and is designed to carry - so the estate stops depending on external link addresses being visible internally at all. Its cost is real: it changes the next hop for every prefix rtr-edge-1 advertises internally, so it is a large convergence event, and it must be applied on the router that is not reporting a symptom, which is a hard thing to agree at 01:30. The fast fix is to re-admit 192.0.2.0/30 to the IGP, which reverses the trigger in one commit; but it also partly reverses a deliberate hardening decision, so it is a hold rather than a resolution - take it if you need service back now, record the owner and the date by which next-hop-self replaces it, and do not let the ticket close on it. The fix to refuse is re-adding the static summary: it works, it is invisible, it explains nothing, and it rebuilds the exact latent dependency that made a routine cleanup into an outage.

Verification

Verify the next hop first and the prefix second, because that is the order the router resolves them in: `show ip route 192.0.2.2` must return a route on every core router, and only then does `show ip route 198.51.100.0/24` mean anything. Check every router in the estate rather than the one that reported the fault - the failure was never local to rtr-core-1, and a fix verified on one box tells you nothing about the other four. Prove forwarding from a client rather than from the router: a router can have a FIB entry and still be the wrong hop, and the symptom that was reported was reachability, not table contents. Confirm both directions, since a return path resolved through the same missing next hop fails the same way and is easy to miss when the forward test passes. If next-hop-self was the fix, confirm the rewrite actually happened by reading the next hop on a core router - it must now be rtr-edge-1's loopback and not 192.0.2.2 - because a next-hop-self applied to the wrong session commits cleanly and changes nothing. Finally, confirm that the hardening the change was protecting is still intact: `show ip route ospf` must not have regained a set of transit links nobody asked for.

Prevention

Set next-hop-self on every iBGP session from a border router as an estate rule, not as an incident response. It converts a dependency on external link addressing into a dependency on loopback addressing, which the IGP already carries deliberately and which nobody cleans up by accident. Monitor the gap between the BGP table and the installed FIB: a count from `show ip bgp summary` next to a count from `show ip route summary`, alerting when they diverge, would have caught this at 01:15 with a precise cause rather than at 01:40 through a reachability complaint. Treat "remove unused configuration" as a change with a blast radius rather than as tidying - a static route is unused only if you can name what resolves through it, and that question is answerable in advance with `show ip route <next-hop>` for every BGP next hop in the table. Keep latent dependencies from accumulating by validating hardening changes against the thing they might silently be propping up: the OSPF change three weeks earlier was correct, was reviewed, and left the estate one commit away from an outage, which is a review outcome worth designing against. And record, per prefix source, which next hop it resolves through, so the question "what breaks if this route disappears" has a written answer.

Reported symptoms

The service desk starts taking calls at 01:40. The reports are useless in a specific and recognisable way: some things work and some things do not, and the people reporting cannot tell you which is which. One team cannot reach a supplier portal. Another cannot pull container images. A third is fine. Nobody can find the pattern because there is no pattern in the customer-visible facts.

The network on-call arrives and finds a healthy estate:

  • Every BGP session is Established, with the same uptime as yesterday, and the per-peer prefix counts are unchanged.
  • OSPF is fully converged on all five routers. Every adjacency is Full. No LSA churn, no alarm, nothing in the log.
  • From either border router, everything is reachable. Curl works, ping works, the transit providers are visibly healthy.
  • traceroute from a client dies at the first core router — which is the box the network team is most confident about, because they just looked at it and it is perfect.

The only change anywhere in the window is a configuration-management run at 01:14 whose diff removed a handful of static routes annotated as unused. It was reviewed by two people. It removed nothing that any show command on any router refers to.

The estate

AS 64512, five VyOS 1.5 LTS routers, a shape you will recognise:

  • rtr-edge-1 — eBGP to ISP-A. External link 192.0.2.0/30; the router holds 192.0.2.2, ISP-A holds 192.0.2.1.
  • rtr-edge-2 — eBGP to ISP-B. External link 10.64.0.0/30, a privately numbered handoff, which is common and will matter.
  • rtr-core-1, rtr-core-2, rtr-core-3 — no external sessions.
  • Full-mesh iBGP between all five. OSPF area 0 carries loopbacks out of 10.255.0.0/24 and the internal LANs out of 10.0.0.0/8.

No router in the estate sets next-hop-self. That is not a mistake anyone made; it is a thing nobody did, which is a different and more durable kind of problem.

Evidence provided

Read-only / SafeBGP has the path and considers it best
vyos@rtr-core-1:~$ show ip bgp 198.51.100.0/24
BGP routing table entry for 198.51.100.0/24
Paths: (1 available, best #1)
Path 1 (best):
192.0.2.2 from 10.255.0.1 (10.255.0.1)
Origin IGP, metric 0, localpref 100, weight 0, valid, internal

Illustrative output

Read-only / Safeno output at all - the kernel has never heard of it
vyos@rtr-core-1:~$ show ip route 198.51.100.0/24
Read-only / Safeno output either - and this is the next hop
vyos@rtr-core-1:~$ show ip route 192.0.2.2
Read-only / Safea destination that still works - note the next hop
vyos@rtr-core-1:~$ show ip bgp 203.0.113.0/24
BGP routing table entry for 203.0.113.0/24
Paths: (1 available, best #1)
Path 1 (best):
10.64.0.2 from 10.255.0.2 (10.255.0.2)
Origin IGP, metric 0, localpref 100, weight 0, valid, internal

Illustrative output

Read-only / Safeone transit link is in the IGP; the other is not
vyos@rtr-core-1:~$ show ip route ospf | match '192.0.2|10.64.0'
O   10.64.0.0/30 [110/20] via 10.255.0.2, eth1, 21d04h11m

Illustrative output

Read-only / Safetwo changes, three weeks apart
vyos@rtr-core-1:~$ show system commit
0   2026-08-18 01:14:09 by ansible via api  cleanup: remove unused static routes
1   2026-07-28 10:31:55 by vyos via cli     ospf: stop redistributing WAN transit links
2   2026-07-14 08:55:31 by vyos via cli     monitoring: add snmp community

Illustrative output

Work the evidence before reading on

The BGP table is not the forwarding table. Everything above is a statement about which of the two you are looking at.

  1. Read the two show ip bgp outputs side by side and find the one field that differs between the broken prefix and the working one. It is not an attribute, a metric or a peer.
  2. Ask what has to be true before a router can install a route whose next hop is not directly connected, and then check whether it is true here.
  3. The OSPF change is three weeks old and the outage is one hour old. Resist discarding the older change: ask what was holding the estate up in between, and why nothing broke on the day of the OSPF change.
  4. The overnight diff removed static routes that no show command refers to. Consider whether “referred to” and “depended on” are the same test.
  5. Two of the three core routers are broken and two of five destinations work. Decide which of those two ratios is a coincidence.

Before continuing: why does this router forward to 203.0.113.0/24 and not to 198.51.100.0/24, when its BGP table is equally happy about both?

Root cause

1. The next hop is an external address, carried unchanged across iBGP

When rtr-edge-1 learns 198.51.100.0/24 from ISP-A, the NEXT_HOP attribute is 192.0.2.2 — the address on rtr-edge-1’s own external link. When rtr-edge-1 re-advertises that path over iBGP, the NEXT_HOP is passed on unchanged. That is the default: an iBGP speaker does not rewrite NEXT_HOP unless next-hop-self tells it to.

So every core router holds a path whose usability depends on a question that has nothing to do with BGP: can I reach 192.0.2.2? If the answer is no, the path stays in the BGP table and never reaches the kernel. show ip bgp looks perfect, show ip route returns nothing, and the router silently drops the traffic it just told everyone else it could carry.

2. The IGP stopped carrying the external link three weeks ago — correctly

The change on 28 July replaced a bare redistribute connected in OSPF with a route-map permitting only loopbacks and the internal supernet. This is a good change. Transit handoff subnets have no business in your IGP: they are addressed by someone else, they change without your involvement, and carrying them expands the IGP for no routing benefit.

It removed 192.0.2.0/30 from OSPF, and nothing broke, so the change was recorded as a success. Three weeks later nobody connects it to anything.

Note which link survived. rtr-edge-2’s handoff is numbered 10.64.0.0/30, inside the 10.0.0.0/8 supernet the route-map permits, so it is still redistributed. The correctly hardened link is the one that broke; the one that slipped through the filter is the one still working. That is the entire explanation for “two in five destinations” — the split is simply which border router wins the best path.

3. A static route nobody could account for was the load-bearing wall

Between 28 July and last night, 192.0.2.2 resolved through a static route for 192.0.2.0/24 — a summary left over from a decommissioned addressing plan, present on the core routers, referenced by nothing, explained by no one.

The overnight automation removed static routes annotated as unused, and by every test available it was unused: no other route pointed at it, no interface used it, no policy matched it. The test that would have caught it is not “is anything referring to this route” but “is any BGP next hop resolving through it”, and that test is one command per next hop.

Two changes, three weeks apart, each correct in isolation, neither sufficient alone. That is the normal shape of a serious outage, and it is why the review of the second change could not have found the problem by reading the second change.

Resolution

  1. Capture the state before touching anything: show ip bgp 198.51.100.0/24, show ip route 198.51.100.0/24 and show ip route 192.0.2.2 on each core router. The third is the one that will look like nothing and is the one that matters.
  2. Establish the true blast radius from the next hop, not from the complaints. Every prefix whose next hop is 192.0.2.2 is affected, whether or not anybody has noticed yet; show ip bgp filtered by that next hop gives you the list.
  3. Decide, explicitly and with a name attached, between the structural fix and the hold. Structural: next-hop-self on rtr-edge-1 iBGP sessions. Hold: re-admit 192.0.2.0/30 to the IGP now and schedule the structural fix.
  4. If you take the hold, take it deliberately: adjust the OSPF redistribution route-map to permit 192.0.2.0/30 specifically, not to permit transit links generally, and write the replacement date into the ticket before you commit.
  5. If you take the structural fix, apply it on rtr-edge-1 - the router with no symptom - and expect a convergence event, because the next hop changes for every prefix it advertises internally.
  6. Either way, compare and then commit-confirm 10. This is a change to next-hop resolution for a large part of the table; if it is wrong you want the router to undo it without you.
  7. Watch the next hop resolve before you look at any prefix: show ip route 192.0.2.2 on a core router, then show ip route 198.51.100.0/24. The order is the diagnosis in miniature.
  8. Do not re-add the static summary. It restores service, it is invisible to review, and it rebuilds the exact dependency that turned a routine cleanup into an outage.
  9. Confirm and save on every router, then close the loop on the automation: the removed static was correctly identified as unreferenced and was still load-bearing, and that gap belongs in the change process rather than in this ticket.

Verification

  1. The next hop resolves. show ip route 192.0.2.2 returns a route on every core router. Check all of them - the failure was never local to the box that reported it.
  2. The prefix is installed. show ip route 198.51.100.0/24 shows a BGP route with a usable next hop, and show ip route summary shows the BGP count back where it belongs.
  3. Forwarding works from a client, not from the router. The reported symptom was reachability; a FIB entry is necessary and not sufficient.
  4. The return path works. A return route that resolves through the same missing next hop fails identically and is invisible to a one-way test.
  5. If next-hop-self was the fix, the rewrite actually happened: show ip bgp 198.51.100.0/24 on a core router shows rtr-edge-1 loopback as the next hop, not 192.0.2.2. Applied to the wrong session it commits cleanly and changes nothing.
  6. The hardening survived. show ip route ospf must not have regained a set of transit links nobody asked for; if you took the hold, exactly one /30 came back and it is named in the route-map.
  7. The whole affected set recovered, not just the prefixes in the tickets. Re-run the next-hop query from step two of the resolution and confirm every prefix in that list is now installed.

Prevention

  • Make next-hop-self an estate rule on every border router’s iBGP sessions. It converts a dependency on somebody else’s link addressing into a dependency on your own loopbacks, which the IGP carries deliberately and which nobody removes as tidying.
  • Alert on the gap between the BGP table and the FIB. The counts from show ip bgp summary and show ip route summary should track each other. A divergence is precisely this class of fault, and it is visible at the moment of breakage rather than forty minutes later through a service-desk call.
  • Redefine “unused” for routing configuration. A static route is unused only when you can name what resolves through it. For every BGP next hop in the table, show ip route <next-hop> answers the question in advance, and the answer belongs in the change record.
  • Validate hardening changes against what they might be propping up. The OSPF change was correct and left the estate one commit from an outage. The review that would have caught it asks not “is this change right” but “what currently depends on the thing this removes”.
  • Write down which next hop each prefix source resolves through. It is a short table, it changes rarely, and it turns a two-hour diagnosis into a one-command check.