Wrong Default Route · Break/Fix · VyOS

Reported symptoms

edge2 is the router for a regional site with two transit providers. ISP-A on eth0 is the contracted primary and sends a default over eBGP. ISP-B on eth1 is the backup and sends one too, held back by an inbound route-map. The site announces its own prefix to both.

The ticket did not come from the network team. Finance opened it, because ISP-B has spent two months billing above its 95th-percentile commit while ISP-A — the circuit the site pays a primary’s price for — bills at close to nothing.

Two more complaints arrived separately and were logged as separate problems:

A partner’s API started refusing the site two days ago with an IP-allowlist error. The partner had just tightened their allowlist to the single address the site documented to them years ago.
A handful of long-lived sessions through a stateful middlebox drop intermittently. Inbound traffic still arrives over ISP-A, so those flows are asymmetric: out one provider, back the other.

The confounding facts are what kept this open. BGP is clean — both sessions Established for weeks, prefix counts as expected, no flaps. And it has already been fixed once: a week ago an engineer raised the local-preference on the ISP-A default to 200, confirmed the change was live in the BGP table, and closed the ticket. Nothing moved.

Evidence provided

Read-only / Safetwo candidates; read which one carries the selected marker

vyos@edge2:~$ show ip route 0.0.0.0/0

Codes: K - kernel route, C - connected, S - static, R - RIP,
     O - OSPF, B - BGP, > - selected route, * - FIB route

S>* 0.0.0.0/0 [1/0] via 198.51.100.1, eth1, weight 1, 06w0d05h
B   0.0.0.0/0 [20/0] via 203.0.113.1, eth0, weight 1, 05w6d22h

Illustrative output

Read-only / Safethe deployed fix is present, correct, and the phrase to notice is 'not in RIB'

vyos@edge2:~$ show ip bgp 0.0.0.0/0

BGP routing table entry for 0.0.0.0/0
Paths: (2 available, best #1, table default, not in RIB)
64510
  203.0.113.1 from 203.0.113.1 (203.0.113.254)
    Origin IGP, localpref 200, valid, external, best (Local Pref)
64520
  198.51.100.1 from 198.51.100.1 (198.51.100.254)
    Origin IGP, localpref 50, valid, external

Illustrative output

Read-only / Safeone line, and nobody on the call put it there

vyos@edge2:~$ show configuration commands | match "static route 0.0.0.0/0"

set protocols static route 0.0.0.0/0 next-hop 198.51.100.1

Illustrative output

Read-only / Saferevision 3 is the fault; revision 4 is why it was reasonable

vyos@edge2:~$ show system commit | head -6

0   2026-08-11 09:22:04 by admin via cli   raise localpref on ISP-A default
1   2026-08-04 14:05:51 by admin via cli   firewall rule 40 for monitoring
2   2026-07-30 11:47:19 by admin via cli   ntp server change
3   2026-07-07 22:14:38 by admin via cli   temp default during ISP-A window
4   2026-07-07 21:58:02 by admin via cli   shutdown ISP-A session for window

Illustrative output

Read-only / Safefirst hop is the backup provider, from the router itself

vyos@edge2:~$ traceroute -n 198.18.0.1

traceroute to 198.18.0.1 (198.18.0.1), 30 hops max, 60 byte packets
1  198.51.100.1  1.412 ms  1.388 ms  1.401 ms
2  198.51.100.254  2.907 ms  2.884 ms  2.910 ms

Illustrative output

Work the evidence before reading on

The BGP table and the routing table disagree, and both are telling the truth.

Read the two lines of show ip route 0.0.0.0/0 again. Which marker means “selected”, and which of the two candidates has it? What decides between two routes for the same prefix from two different sources?
show ip bgp 0.0.0.0/0 says best next to the ISP-A path, and it also says something about the RIB. What is BGP claiming to be best among?
The local-preference change is live, visible and correct. If a knob is set in the right place and has no effect, what does that tell you about which layer the decision is actually being made in?
Revision 3 in the commit log was a sensible change when it was made, and the session it was made for is listed one line below it. What was supposed to happen to revision 3, and what event would have made anybody notice that it had not?

Then the question that resolves it: if you raised the ISP-A local-preference to 4,294,967,295, what would change? Your answer tells you where the fault is.

Root cause

1. A static default outlived the window it was made for

Revision 4 in the commit log shut the ISP-A session for a maintenance window. Revision 3, sixteen minutes later, added a static default toward ISP-B to keep the site online while it was down. That was correct, and it worked.

The window closed. The ISP-A session came back. The static did not go away.

2. Administrative distance decided the rest, and it knows nothing about intent

When two routes cover the same prefix, the router picks the one whose source it trusts more, and that trust is a fixed number: connected 0, static 1, eBGP 20, OSPF 110, iBGP 200. Lower wins. A static route at distance 1 beats an eBGP-learned route at distance 20 every time, in every direction, regardless of what the operator meant.

So from the moment the session recovered, the site’s entire egress moved to the backup provider. Nothing broke, because ISP-B is a working transit. There was a default; it was selected; traffic flowed. No monitoring anywhere in the estate asks whether the right traffic is leaving by the right door, so the fault surfaced through an invoice and a partner’s allowlist, eight weeks later, as two unrelated tickets.

3. Local-preference was the right idea in the wrong layer

This is the part that cost a week, and it is the most transferable thing in the scenario.

Local-preference is a BGP path attribute. It orders paths within BGP, and its job is to decide which of several BGP paths for a prefix becomes BGP’s best path. Having decided, BGP hands the RIB exactly one candidate — and that candidate arrives carrying administrative distance 20.

The contest that determines the selected route is between that one BGP candidate and the static route at distance 1. Local-preference is not an input to that contest and never sees it. Set to 200 or to four billion, it changes which BGP path is best and cannot change which route is selected.

The line to notice in the evidence is not in RIB, sitting in the same output that says best. Both are accurate. BGP is reporting that it has chosen a best path and that the RIB did not take it.

Under the hood

Three layers decide a route, and they are consulted in a fixed order: prefix length first, then administrative distance, then metric.

Prefix length is the strongest: a /24 beats a /16 for any address matching both, whatever the sources are. Administrative distance is consulted only when two routes cover the same prefix, and it compares sources. Metric is consulted only when the sources are the same, and it compares paths within one source.

Almost every “the route I configured is not being used” incident is a knob turned in the wrong one of those three. Local-preference, MED, AS-path prepending, IGP cost, OSPF interface cost — all of them live in layer three, inside a single protocol. They cannot influence layer two. Conversely, distance lives in layer two and is indifferent to everything a protocol has to say about its own paths.

The diagnostic that separates them is two commands, and it is worth running both even when you are confident. show ip bgp <prefix> is BGP’s own view: which paths exist and which one BGP nominated. show ip route <prefix> is the RIB’s view: which candidates were submitted and which one was selected. When they agree, the protocol got its way. When they disagree — a path marked best in one and absent from the other — the decision was made above the protocol, and nothing you change inside the protocol will move it.

Verifying a fix at the layer the knob lives in

The local-preference change was verified by running show ip bgp 0.0.0.0/0 and confirming that localpref read 200 and the ISP-A path read best. Both were true. The site kept sending every packet out of ISP-B.

The check confirmed the knob had been set. It could not confirm the knob had an effect, because it asked the question inside the subsystem that owns the knob, and the fault was in the arbitration above it. This shape recurs constantly: correct content in a place that is not consulted, or is consulted too late. A sysctl overridden by a later file in sysctl.d. A firewall rule below a terminating rule that already matched. A route-map on a peer that is not the peer carrying the traffic.

The discipline that catches all of them is to verify a routing change at the forwarding layer and never at the configuration layer. Not “is local-preference 200” but “which route is selected, and where does a packet actually go”. show ip route 0.0.0.0/0 and a traceroute from a host behind the router answer that in ten seconds, and they would have closed this on day one.

Resolution

Establish what now depends on the wrong path before you correct it. Six weeks of egress via ISP-B is long enough for dependencies to accrete: firewall or NAT rules written against eth1, allowlists at third parties updated to the ISP-B address, VPN peers with a pinned local address. Correcting the route without checking turns one broken partner into several.
Decide whether this is a window change or an immediate one, and say which out loud. Removing the static re-translates every source-NATed flow to the ISP-A address on its next packet, so sessions in flight break. That cost is real, and it is smaller than the allowlist rejection that is already happening — but it is a decision, not a side effect.
Remove the static default and the inert fix together, under commit-confirm 5: delete protocols static route 0.0.0.0/0, and delete policy route-map RM-FROM-ISP-A rule 10 set local-preference for the knob added a week ago. Leaving the local-preference behind is worse than useless, because the next engineer will read it as evidence that path preference is managed here.
Confirm the selection moved before the confirm timer expires. show ip route 0.0.0.0/0 must show the BGP route selected. If it does not, let the auto-rollback fire rather than adding another change on top.
If ISP-B must remain primary for commercial reasons, do not re-add the static. Express it in BGP by raising local-preference on the ISP-B default, so the path is still tied to the liveness of the BGP session and still fails over.
Re-run the partner call from the site and confirm the source address they see is the documented one. That is the symptom the business raised, and the routing table is not evidence for it.
Close the loop on the window that started this: find the change record for revision 3 and record what the removal step should have said.

Verification

show ip route 0.0.0.0/0 shows the eBGP default via 203.0.113.1 selected, and shows no static candidate at all. Not an inactive static — no static. A leftover inactive route is the next incident, because it also outranks the correct one the moment its next-hop becomes reachable again.
show ip bgp 0.0.0.0/0 is unchanged. The BGP table was correct throughout, and a fix that changed it would mean you fixed something else.
Both resolvers agree. traceroute -n from the router and from a host behind it must both leave via 203.0.113.1. Router and host resolve the route by different paths, and agreement across the two is the evidence, not either one alone.
The outside world sees the right address. Check the observed public source address from an external service, and re-run the partner call that was being rejected. This is the only check independent of the router that produced the fault.
Egress has moved. show interfaces ethernet eth0 and eth1 show the byte counters crossing over within a few minutes.
Failover works. In a window, fail the ISP-A session and confirm the default moves to ISP-B and returns on recovery. For six weeks this site believed it had a redundant edge and had a single static route; the only way to know that has been repaired is to exercise it.

Prevention

Give temporary configuration an expiry. A change made to survive a window is not done when it works; it is done when it is gone. Put the removal in the same checklist, with the same owner, and do not close the window until the removal is verified.
When a temporary default really is needed, make it floating — a static at a distance above the dynamic source, which is inert while BGP holds the default and takes over automatically when BGP withdraws it:

set protocols static route 0.0.0.0/0 next-hop 198.51.100.1 distance 210

Left behind, that line is harmless. Left behind, the same line without distance 210 is this incident. One habit removes the entire failure mode.

Alert on the identity of the selected default, not its existence. “A default exists and traffic flows” was true for eight weeks. The check that would have caught this is that the selected 0.0.0.0/0 has source BGP and next-hop 203.0.113.1.
Monitor egress bytes per WAN against design intent, and monitor the public source address the estate presents. Third parties contract on the address, not on the routing table, and the address is observable from outside.
Diff the running configuration against the intended configuration on a schedule. A line nobody remembers adding should be found by a comparison, not by an invoice.

What if ISP-B's transit had failed while the static was in place?

The site would have gone dark, and the redundancy it was paying for would not have helped.

A BGP-learned default is tied to the liveness of the session that carries it. If ISP-B’s transit fails, the session drops or the prefix is withdrawn, the route leaves the RIB, and the remaining default takes over. That is the whole mechanism, and it is why the site has two providers.

A static default has no such tie. It is selected because it is configured, and it stays selected as long as its next-hop is resolvable — which, for a next-hop on a directly connected segment, means as long as the Ethernet link is up. A provider whose upstream has failed will usually keep the customer link up. The static would have kept pointing into it, the eBGP default from ISP-A would have stayed at distance 20 and stayed unselected, and every packet would have been handed to a provider that had nowhere to send it.

So the invoice and the allowlist were not the severity of this incident. The severity is that for six weeks the site had no failover and no way to know, because everything that could have told it was reporting healthy.

Why this matters

Administrative distance is usually taught as a table to memorise, and the table is not the skill. The skill is recognising that the table encodes trust in a source, and that a source’s trustworthiness has nothing to do with whether the operator wanted that route today. Static beats eBGP because a human typed it, and the router has no way to know the human typed it eight weeks ago for a reason that expired the same evening.

The second half is the more expensive lesson. A change can be committed, correct, visible in the subsystem it belongs to, and completely without effect, because the decision it was meant to influence is made a layer above it. BGP reported the ISP-A path as best and reported, in the same breath, that the RIB had not taken it. Every fact was on the screen; the engineer read the half that answered the question they had asked.

Verify routing changes by where packets go, never by what the configuration says. show ip route and a traceroute from a host behind the router cost ten seconds and would have ended this on the day it was first reported.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. A static default outlived the window it was made for

2. Administrative distance decided the rest, and it knows nothing about intent

3. Local-preference was the right idea in the wrong layer

Resolution

Verification

Prevention