Skip to main content
RunBook Academy

← All break/fix scenarios in VyOS

advancedvyos-bgp-attribute~30 min

BGP Attribute Change Misroutes Traffic

Reported symptoms

  • ●Ingress on the Provider B link rose from roughly 40% to 95% of capacity overnight and is now dropping during the morning peak
  • ●A subset of customer-facing services fails intermittently: short requests mostly succeed, long-lived sessions establish and then stall
  • ●Ingress on the Provider A link has fallen to near zero while egress towards Provider A is unchanged, and the NOC has raised a suspected optics fault
  • ●The change made the previous evening did exactly what it was written to do, was verified against its stated goal, and was closed
  • ●Both eBGP sessions have been Established for three days and the received-prefix counts are identical to last week
  • ●`show ip route` and traceroutes from the router are exactly as designed; nothing on the router looks wrong
  • ●Provider A reports no changes on their side and their looking glass no longer lists any of our prefixes

Evidence

  • · `show ip bgp neighbors 203.0.113.6 advertised-routes` returns one prefix; the archived capture from Sunday returns fourteen
  • · `show ip bgp summary` shows both peers Established for 3d, with unchanged received-prefix counts
  • · `show ip bgp neighbors 203.0.113.6` reports an outbound route-map named TO-PROVIDER-A
  • · `show configuration commands | match TO-PROVIDER-A` returns three lines, all of them sequence 10
  • · Interface counters on the Provider A interface show received bytes at about 2% of the previous week and transmitted bytes unchanged
  • · `show configuration commit` lists two commits at 21:44 and 21:58 the previous evening, both by the on-call engineer
  • · The 21:44 commit set local-preference on the Provider B import map and was reverted at 21:52 with no observed effect
  • · Provider A's looking glass returns no results for any of the AS's originated prefixes
Diagnosis and resolutionclick to reveal

Root cause

A route-map is an advertisement filter first and an attribute editor second, and the change treated it as only the second. The intent was to move ingress traffic for the partner range 192.0.2.0/24 onto Provider B, which is inbound traffic engineering and therefore has to be expressed in what the AS advertises rather than in what it accepts - the reverted 21:44 attempt set local-preference on the Provider B import map, which could never have worked because LOCAL_PREF is carried between iBGP peers and stripped when advertised to an eBGP peer, so it steers this AS's own outbound traffic and is invisible to Provider B. The 21:58 attempt reached for the right attribute, an AS_PATH prepend on the export towards Provider A, and created a route-map with a single rule: sequence 10, action permit, matching the partner prefix and prepending the local AS three times. A route-map evaluates its rules in sequence, the first matching rule decides the disposition, and a route that matches no rule hits the implicit deny at the end and is filtered. Attaching that map to the Provider A neighbour in the export direction therefore changed that peer from "advertise everything" to "advertise only 192.0.2.0/24", withdrawing the other thirteen prefixes in the same commit. The change's own goal was met, which is why the verification passed: partner traffic does now arrive via Provider B. It arrives there because it is the only path left, along with everything else. Nothing on the router looks wrong because outbound policy towards a provider does not touch the local RIB: received counts, `show ip route`, and every traceroute originated on the router are unchanged, and the only view that changed is one nobody monitors. The intermittent session failures are the second-order effect - egress still prefers Provider A because the import policy was never touched, while ingress now arrives entirely on Provider B, so any stateful device in either path sees one direction of a flow and drops the other.

Remediation

Restore the advertisement before touching the prepend; the prepend is not the emergency and the withdrawal is. Add a terminating rule to the route-map - a high sequence number, action permit, with no match and no set clause - so that everything which does not match the prepend rule is advertised as it was before, and apply it under `commit-confirm` because the session being changed carries transit. Say the cost of that restore out loud: re-advertising thirteen prefixes moves a large volume of ingress traffic from one provider to the other within a minute or two, which is itself a capacity event on Provider A, so it is done deliberately with someone watching both links rather than fired and forgotten. Holding is not a real option here because Provider B is already dropping traffic, but the choice of when is: if the morning peak is minutes away, restoring into the peak is still better than a congested link, and that judgement belongs to a named owner rather than to whoever is typing. Once ingress has rebalanced, re-do the partner change properly with the terminating rule present from the start, and verify both what the change intended and what else it moved. If Provider A publishes a community that requests local depreference, that is a cleaner expression of the same intent than a prepend - but only if they publish one, which is a fact to look up rather than to assume. Finally, close out the stalled sessions: asymmetric flows through a stateful device do not always recover on their own once the asymmetry is removed, and some will need to be re-established.

Verification

Count what is advertised, not what is intended. `show ip bgp neighbors 203.0.113.6 advertised-routes` must return the full documented origination set - fourteen prefixes - and the check is a comparison against that documented list, not against "more than it was an hour ago". Then read the AS_PATH per prefix in the same output: the prepend must be present on 192.0.2.0/24 and absent on the other thirteen, because a terminating rule placed at the wrong sequence number would restore the advertisement and prepend everything. Confirm from outside the AS as well as inside it, using Provider A's looking glass, since the router's own view of what it advertises is a statement of intent and the looking glass is a statement of arrival. Watch the interface counters on both providers until ingress on Provider A returns to its historical share and Provider B drops below its congestion threshold; the traffic move is the outcome the incident is measured by, not the prefix count. Confirm the failing services from the application side rather than from the router, because the sessions that broke were broken by asymmetry and the routing table never showed it. Last, prove the new guard can fail: in the lab, remove the terminating rule from a copy of the map, run the change process, and confirm the advertised-count check refuses the change rather than reporting it.

Prevention

Give every neighbour route-map an explicit terminating rule, permit or deny, written deliberately and reviewed as part of the map. A map without one is a map whose behaviour at the end is decided by a default that nobody wrote down, and the same implicit deny applies to prefix-lists, so the habit transfers. Record the advertised-prefix count per eBGP peer in the change record before and after every policy change, and alert on it continuously: advertisement is the only direction of a BGP relationship that leaves no evidence on the local router, which is exactly why it is the direction that goes unnoticed for a whole night. Verify blast radius separately from intent; this change was verified thoroughly against what it was supposed to do, and the question that was never asked is what else it did. Teach the direction rule explicitly, because the reverted 21:44 commit was the cheap early warning that nobody read: outbound traffic engineering is expressed in what you accept, with local-preference and weight, and inbound traffic engineering is expressed in what you advertise, with AS_PATH, MED and communities. Finally, treat any change that moves which provider carries ingress as a change to path symmetry, and enumerate the stateful devices in both paths before making it - the capacity consequence of this change was visible on a dashboard within minutes, and the asymmetry consequence was a set of application tickets that nobody connected to routing for hours.

Reported symptoms

AS 64500 is dual-homed on border-01: Provider A in AS 64501 across eth2, Provider B in AS 64502 across eth1. The AS originates fourteen prefixes. By 09:00 on Tuesday there are three tickets and they read like three different incidents.

  • Capacity. Ingress on the Provider B link went from about 40% of capacity to 95% overnight, and it is now dropping traffic through the morning peak. The capacity team wants an emergency upgrade with Provider B.
  • Application. A subset of customer-facing services is failing intermittently. Short requests mostly succeed; anything long-lived establishes and then stalls part-way. The pattern does not follow any one service or any one customer.
  • Optics. Ingress on the Provider A link has fallen to almost nothing while egress towards Provider A is exactly what it was last week. The NOC dashboard renders this as a half-dead link and someone has already ordered a replacement SFP.

Against that, the routing evidence is reassuring. Both eBGP sessions have been Established for three days. The received-prefix counts are identical to last week. show ip route is exactly as designed and traceroutes launched from border-01 leave by the provider the design says they should.

There is one change. CR-3312 was applied at 21:58 the previous evening: prepend the partner range 192.0.2.0/24 towards Provider A so that the partner’s traffic arrives on Provider B instead. It was verified — the partner’s traffic does arrive on Provider B — and closed at 22:05.

Evidence provided

Read-only / Safeboth sessions healthy, both receiving what they always received
vyos@border-01:~$ show ip bgp summary
Neighbor        V    AS   MsgRcvd  MsgSent  Up/Down   State/PfxRcd
203.0.113.6     4  64501     41822     3319   3d04h12m       981204
198.51.100.2    4  64502     40977     3301   3d04h09m       981198

Illustrative output

Read-only / Safeone prefix advertised to Provider A; Sunday's archived capture shows fourteen
vyos@border-01:~$ show ip bgp neighbors 203.0.113.6 advertised-routes | tail -5
   Network          Next Hop            Metric LocPrf Weight Path
*> 192.0.2.0/24     203.0.113.5              0         32768 64500 64500 64500 i

Total number of prefixes 1

Illustrative output

Read-only / Safethe whole change: one rule, and the line that attaches it
vyos@border-01:~$ show configuration commands | match TO-PROVIDER-A
set policy route-map TO-PROVIDER-A rule 10 action 'permit'
set policy route-map TO-PROVIDER-A rule 10 match ip address prefix-list 'PARTNER-RANGE'
set policy route-map TO-PROVIDER-A rule 10 set as-path prepend '64500 64500 64500'
set protocols bgp neighbor 203.0.113.6 address-family ipv4-unicast route-map export 'TO-PROVIDER-A'

Illustrative output

Read-only / Safethree commits, not one — 21:44 set local-preference and 21:52 reverted it
vyos@border-01# show configuration commit
1   2026-08-17T21:58:41+01:00   oncall   192.0.2.51   config.20260817-215841.boot
2   2026-08-17T21:52:06+01:00   oncall   192.0.2.51   config.20260817-215206.boot
3   2026-08-17T21:44:19+01:00   oncall   192.0.2.51   config.20260817-214419.boot

Illustrative output

Provider A’s looking glass returns no results for any of the AS’s originated prefixes. Provider A’s NOC confirms nothing changed on their side, and they are right.

Work the evidence before reading on

Every command run on border-01 returns healthy output. Work these questions before reading the root cause.

  1. Received-prefix counts are unchanged and both sessions are up. The change touched policy in one direction only. Which direction, and which command reports it?
  2. A route-map is a list of numbered rules. What happens to a route that matches none of them?
  3. Egress towards Provider A is unchanged and ingress has collapsed. Which attribute steers which direction of traffic — and what does the resulting asymmetry break?
  4. There were three commits that evening, not one. What was the 21:44 commit trying to do, and why did reverting it change nothing?

Before continuing: the change was verified and it passed. Name the question the verification did not ask.

Root cause

1. A route-map filters first and edits attributes second

The route-map that was created has exactly one rule:

set policy route-map TO-PROVIDER-A rule 10 action 'permit'
set policy route-map TO-PROVIDER-A rule 10 match ip address prefix-list 'PARTNER-RANGE'
set policy route-map TO-PROVIDER-A rule 10 set as-path prepend '64500 64500 64500'

Route-map evaluation is sequential and first-match-wins: rules are tried in ascending sequence order, the first rule whose match clauses are satisfied decides the disposition, and later rules are not consulted. A route that satisfies no rule reaches the end of the map and hits the implicit deny, which filters it.

Attaching that map in the export direction therefore changed the Provider A neighbour from “advertise everything eligible” to “advertise only what matches PARTNER-RANGE”. The prepend worked perfectly on the one prefix it was written for. The other thirteen were withdrawn in the same commit, by a rule nobody wrote.

The diff was three added lines plus one line to attach the map. The effect was one prepend and thirteen withdrawals.

2. The reverted commit was the warning nobody read

The 21:44 commit set local-preference 200 on the Provider B import map, and it had no observable effect, which is why it was reverted eight minutes later.

It could not have had one. LOCAL_PREF is carried in UPDATE messages between iBGP peers and stripped when a route is advertised to an eBGP peer; it expresses this AS’s own outbound preference and is invisible to Provider B by design. The goal — move the partner’s inbound traffic — was never reachable through an attribute that never leaves the AS.

That is the direction rule, and it is the whole of the confusion in this incident: outbound traffic engineering is expressed in what you accept, with local-preference and weight; inbound traffic engineering is expressed in what you advertise, with AS_PATH, MED and communities. The second attempt reached for the right family of attribute. It just also picked up a filter.

3. Why nothing on the router looks wrong

Outbound policy towards a provider does not touch the local RIB. Every diagnostic border-01 offers about itself describes either what it received or where it will send packets, and neither changed:

  • received-prefix counts, unchanged, because import policy was not touched;
  • show ip route, unchanged, for the same reason;
  • traceroutes from the router, unchanged, because egress still prefers Provider A.

The only view that changed is show ip bgp neighbors ... advertised-routes, and outside a change window nobody runs it. This is the property that makes advertisement errors long-lived: they are silent on the device that causes them and only visible from the other side of the peering.

4. The second failure that the first one caused

Ingress arrives where you are advertised, and egress leaves where you prefer. The change moved the first and left the second alone, so from 21:58 the AS was egressing via Provider A and ingressing entirely via Provider B.

That is the application ticket. A flow that leaves by one provider and returns by another passes through any stateful device in either path in one direction only, and a device that never sees the other half of a conversation drops it. Short requests that complete inside a single exchange often survive; long-lived sessions establish and then stall, which is exactly the reported pattern, and it points at routing only if you already know the paths went asymmetric.

Resolution

  1. Restore the advertisement before touching the prepend. The withdrawal is the incident; the prepend is a preference. Anything that debates the traffic-engineering design first extends a congested link.
  2. Add a terminating rule to the map: a high sequence number, action permit, no match clause and no set clause, so every prefix that does not match the prepend rule is advertised exactly as it was before the change. Apply it with commit-confirm 10 because the session carries transit.
  3. Say the cost of the restore out loud before committing it. Re-advertising thirteen prefixes moves a large volume of ingress from one provider to the other inside a minute or two, which is a capacity event on Provider A in its own right. Have someone watching both links while it happens.
  4. Choose when, deliberately, and give the choice an owner. Restoring into the morning peak is still better than leaving Provider B dropping traffic, but that is a judgement about two bad options and it belongs to a named person rather than to whoever is at the keyboard.
  5. Watch the traffic move rather than the prefix count. The count tells you the configuration is right; the interface counters tell you the incident is over.
  6. Re-do the partner change once ingress has rebalanced, with the terminating rule present from the start, and verify the intent and the blast radius as two separate checks.
  7. Consider expressing the intent as a community rather than a prepend, but only if Provider A publishes one for local depreference. That is a fact to look up in their published policy, not one to assume; a community the provider does not honour is an advertisement with no effect and no error message.
  8. Close out the stalled application sessions. Flows broken by asymmetry through a stateful device do not always recover once the asymmetry is removed, and some will need to be re-established from the application side.

Verification

  1. The advertised set is complete. show ip bgp neighbors 203.0.113.6 advertised-routes returns the documented origination set of fourteen prefixes, and the check is a comparison against that documented list. "More than it was an hour ago" is not a check.
  2. The prepend is on one prefix and not on fourteen. Read the AS_PATH per prefix in the same output: 192.0.2.0/24 carries the three prepends and the other thirteen do not. A terminating rule placed at the wrong sequence number restores the advertisement and prepends everything, and the prefix count alone cannot tell the two apart.
  3. The advertisement arrives. Confirm from Provider A's looking glass, not only from border-01. The router's view of what it advertises is a statement of intent; the looking glass is a statement of arrival, and the difference between them is where filtering on the provider side would show up.
  4. The traffic rebalanced. Ingress on Provider A returns to its historical share and Provider B falls below its congestion threshold. Hold this over a full peak before calling it, because an overnight measurement proves nothing about 09:00.
  5. The applications recovered. Confirm the previously failing services from the application side. They broke because of path asymmetry, which the routing table never displayed, so the routing table cannot be the evidence that they are fixed.
  6. Symmetry is restored deliberately, not by accident. Confirm that ingress and egress for the affected prefixes now use the same provider, and if the design intends them not to, confirm every stateful device in both paths is known and tolerant of that.
  7. The guard can fail. In the lab, delete the terminating rule from a copy of the map and run the change process. The advertised-count check must refuse the change. A check that has never rejected anything is a comment.

Prevention

  • Every neighbour route-map ends with an explicit terminating rule, permit or deny, written on purpose. A map without one behaves according to a default nobody recorded, and a reader cannot tell a deliberate omission from a forgotten one.
  • Record advertised-prefix counts per peer in the change record, before and after, and alert on them continuously. Advertisement is the only half of a BGP relationship that leaves no evidence on the local router, which is why it went unnoticed for eleven hours.
  • Verify blast radius separately from intent. This change was verified carefully against what it was meant to achieve, and it achieved it. The missing question is “what else moved”, and it is a different command from the one that proves the goal.
  • Know which attribute steers which direction, and treat a no-effect change as a signal. The reverted 21:44 commit was a local-preference change made in pursuit of an inbound outcome. It did nothing, which was the cheapest possible warning that the mental model was inverted.
  • Treat any change to who carries ingress as a change to path symmetry. Enumerate the stateful devices in both paths first. The capacity effect of this change was on a dashboard within minutes; the asymmetry effect arrived as application tickets that nobody connected to routing until the afternoon.
  • Keep a weekly archived capture of advertised routes per peer. The only reason the fourteen-to-one drop was provable at all is that Sunday’s capture existed to compare against.

Cross-course references

The Ansible course’s LVI-Ansible-Breakfix carries the same shape at the inventory layer, where a change that adds no code changes the set of machines a run touches. The Observability course’s XCVIII-Observability-Troubleshooting covers the monitoring gap that let an advertisement collapse go unalerted for eleven hours. Within this course, vyos-xxxiii-05-route-map-composition derives the first-match-wins evaluation and the implicit deny, vyos-xxvi-01-local-preference establishes that LOCAL_PREF is stripped towards eBGP peers, vyos-xxvii-03-as-path-prepending covers the attribute the change should have used and how, vyos-xxvi-06-attribute-anti-patterns catalogues the neighbouring mistakes, and vyos-lii-05-return-path covers the asymmetry that broke the applications.