Outbound BGP Filter Blocks Intended Prefix

Reported symptoms

The maintenance was routine and had been on the calendar for a fortnight. ISP-B was replacing a line card and asked for the session to be drained; the network team shifted traffic to ISP-A at 01:00, watched the graphs move, and confirmed the estate was healthy.

Nine minutes later the payments team escalated. Their new API — the one behind 203.0.113.0/24, live for three weeks — was returning nothing from the internet. Not slow, not erroring: unreachable.

What made it hard:

• Everything else was fine. Twenty-odd services ride the same border. Not one of them noticed the drain.
• From inside, the service was healthy. The route was in the RIB, the servers answered, and the internal synthetic monitor stayed green for the entire event. The application team could see nothing wrong and said so.
• External traceroutes died before reaching us, at the last hop in ISP-A’s network. That reads as an upstream black hole, and it is exactly what a missing origination looks like from outside.
• Undoing the drain fixed it in under two minutes. This is the dangerous one. Restoring the ISP-B session restored the service instantly and completely, which is powerful evidence for “ISP-B’s maintenance broke it” and no evidence at all for what actually happened.
• Nothing had changed. The prefix passed acceptance testing three weeks ago and had not been touched since. No commit today.
• Both sessions look right. Established, expected prefix counts, ISP-A up sixty-three days without a flap.
• RPKI and IRR were checked first and were clean. The ROA and the route object for 203.0.113.0/24 are published and correct, which took prefix validation off the table early.

The incident was thirty seconds from being closed against the carrier.

The estate

rtr-border-1 is a VyOS 1.5 LTS router in AS 64512 with two transits and one job: originate the company’s address space and carry it to both upstreams.

• ISP-A, peer 192.0.2.1, AS 64500. Outbound policy is the route-map RM-TO-ISP-A.
• ISP-B, peer 192.0.2.5, AS 64600.

The company originates 198.51.100.0/22, two more-specifics inside it, and — since three weeks ago — 203.0.113.0/24 for the payments API. Origination is by network statement against prefixes present in the RIB.

RM-TO-ISP-A exists to do traffic engineering: a subset of the prefixes carry a community that means “prepend twice towards ISP-A”, and the rest go out unmodified. Both of its rules gate on membership of the prefix-list PL-OUR-AGGREGATES, which is meant to be the inventory of everything this AS originates.

Evidence provided

vyos@rtr-border-1:~$ show ip route 203.0.113.0/24

Routing entry for 203.0.113.0/24
Known via static, distance 1, metric 0, best
* 10.60.4.2, via eth2

vyos@rtr-border-1:~$ show ip bgp 203.0.113.0/24

BGP routing table entry for 203.0.113.0/24
Paths: (1 available, best #1, table default)
Advertised to non peer-group peers:
192.0.2.5
Local
0.0.0.0 from 0.0.0.0 (10.255.0.1)
Origin IGP, metric 0, weight 32768, valid, sourced, local, best

vyos@rtr-border-1:~$ show ip bgp neighbors 192.0.2.1 advertised-routes

   Network            Next Hop     Metric  LocPrf  Weight  Path
*> 198.51.100.0/22    0.0.0.0           0           32768  i
*> 198.51.100.0/24    0.0.0.0           0           32768  i
*> 198.51.101.0/24    0.0.0.0           0           32768  i

Total number of prefixes 3

vyos@rtr-border-1:~$ show ip bgp neighbors 192.0.2.1

BGP neighbor is 192.0.2.1, remote AS 64500, local AS 64512, external link
BGP state = Established, up for 63d04h11m
For address family: IPv4 Unicast
Outbound route-map: RM-TO-ISP-A
Outbound route-map applied: yes

vyos@rtr-border-1:~$ show policy route-map RM-TO-ISP-A

Route-map RM-TO-ISP-A, permit, sequence 10
Match clauses:
ip address prefix-list: PL-OUR-AGGREGATES
community: CL-PREPEND-ISP-A
Set clauses:
as-path prepend 64512 64512
Action: permit

Route-map RM-TO-ISP-A, permit, sequence 20
Match clauses:
ip address prefix-list: PL-OUR-AGGREGATES
Set clauses:
Action: permit

vyos@rtr-border-1:~$ show configuration commands | match PL-OUR-AGGREGATES

set policy prefix-list PL-OUR-AGGREGATES rule 10 action 'permit'
set policy prefix-list PL-OUR-AGGREGATES rule 10 prefix '198.51.100.0/22'
set policy prefix-list PL-OUR-AGGREGATES rule 20 action 'permit'
set policy prefix-list PL-OUR-AGGREGATES rule 20 prefix '198.51.100.0/24'
set policy prefix-list PL-OUR-AGGREGATES rule 30 action 'permit'
set policy prefix-list PL-OUR-AGGREGATES rule 30 prefix '198.51.101.0/24'

vyos@rtr-border-1:~$ show system commit

0   2026-07-28 14:06:52 by vyos via cli   add 203.0.113.0/24 for new service
1   2026-07-11 09:31:20 by vyos via cli   ntp: replace decommissioned server
2   2026-06-30 16:44:08 by vyos via cli   syslog: add secondary collector

Work the evidence before reading on

The advertisement diagnostic has four boundaries and they are ordered for a reason: the RIB, the BGP table, what we advertise, and what the peer accepts. Walk them.

1. Boundaries one and two are already answered by the first two blocks. Say out loud which boundary the failure is at before reading further, because that decision eliminates three quarters of the possible causes.
2. show ip bgp 203.0.113.0/24 lists one peer under “Advertised to non peer-group peers” and this router has two peers. That single line is the whole diagnosis if you read it as an exhaustive list rather than as a fact about the one peer it names.
3. Read RM-TO-ISP-A and ask what happens to a route that matches neither rule. Then ask what evidence that outcome leaves behind.
4. Compare the prefix-list entries against the set of prefixes this AS originates. The counting takes five seconds.
5. Now the harder question, and the one that turns a one-line fix into a real incident review: the prefix has been unadvertised to ISP-A for three weeks. Why did nobody notice, and what does the answer imply about the ISP-B session?

Before continuing: nothing in this configuration denies the prefix. Explain how it is being filtered anyway, and explain why the service worked yesterday.

Root cause

1. A route-map filters by omission, and omission leaves no trace

RM-TO-ISP-A has two rules and both of them gate on membership of PL-OUR-AGGREGATES. A route that is not in that list matches neither rule, falls off the end of the route-map, and is filtered by the implicit deny.

This is the semantic that separates a route-map from a prefix-list in the reader’s head, and it is worth stating plainly: a route-map that does not match a route denies it. There is no “permit by default” at the bottom. So the search for the deny rule that is blocking 203.0.113.0/24 fails, because there is no deny rule. The prefix is not denied. It is unmentioned, and unmentioned is denied.

The consequence for diagnosis matters more than the semantic: a deny rule leaves a hit count, a diff and something to grep for. An omission leaves nothing. Nothing is logged, no counter moves, and show ip bgp 203.0.113.0/24 reports a healthy, best, locally sourced route — because from the Loc-RIB’s point of view it is healthy. The only evidence available is a negative one: the peer is absent from the “Advertised to” list.

2. The provisioning change updated three places out of four

The commit three weeks ago did real work. It added the network statement so the prefix entered the Loc-RIB, adjusted the inbound policy, and the same change ticket covered publishing the ROA and the IRR route object. Every one of those is correct and every one of them was verified.

PL-OUR-AGGREGATES was not touched, because the provisioning runbook for a new public prefix does not mention it. The list is maintained by hand, is referenced from a route-map rather than bound to a neighbour directly, and is therefore two hops away from the place where anyone thinks about origination.

3. The other transit hid it, and hid it perfectly

Run the same show ip bgp neighbors against 192.0.2.5 and there is no outbound policy at all. The ISP-B session advertises the whole Loc-RIB.

That is why the service passed acceptance testing, why external monitoring was green for three weeks, and why nobody saw a problem: 203.0.113.0/24 was globally reachable the entire time, through exactly one upstream. It was single-homed from the day it was built, and nothing in the estate was capable of saying so.

The drain did not break the service. The drain removed the only path the service had ever had.

Resolution

1. Capture the before state into the ticket: show ip bgp neighbors 192.0.2.1 advertised-routes including the total count, and show ip bgp 203.0.113.0/24. The count is the number you will diff against, and a fix that moves it by more than one did something you did not intend.
2. Add the prefix to the list, not a rule to the route-map: set policy prefix-list PL-OUR-AGGREGATES rule 40 action permit and set policy prefix-list PL-OUR-AGGREGATES rule 40 prefix 203.0.113.0/24. The list is the inventory of what this AS originates, and keeping it as the single gate is the property that makes the policy reviewable.
3. Do not add a catch-all permit to the end of RM-TO-ISP-A. It would fix this prefix and simultaneously offer ISP-A everything in the Loc-RIB, including the full table learned from ISP-B. That is not a wider filter, it is a route leak, and unlike this incident it damages other people.
4. Read the diff with compare, then commit-confirm 10. This is outbound policy on a live transit session; the timer is what recovers you if the change is wrong in the direction you did not anticipate.
5. Push the corrected policy to the peer with clear ip bgp 192.0.2.1 soft out. That is a route refresh and leaves the session up. A hard clear ip bgp 192.0.2.1 would withdraw every prefix from a working transit to publish one that was never there.
6. Confirm and save once advertised-routes contains the prefix and the total has risen by exactly one.
7. Treat the missing outbound policy on ISP-B as a separate change with its own window, its own review and its own owner. Building an export filter for a session that has never had one is not a same-night task: if the list behind it is incomplete you withdraw everything, and the list must be built from what the estate intends to originate rather than from what today happens to be in advertised-routes.
8. If that window does not exist tonight, hold, explicitly. Leaving ISP-B unfiltered until a scheduled change, with a named owner and a fixed end time, is a defensible decision. Applying an untested export filter to a live transit at 02:00 is not, and the incident does not oblige you to do it.

The change itself, in the 1.5 configuration tree, is two lines:

set policy prefix-list PL-OUR-AGGREGATES rule 40 action 'permit'
set policy prefix-list PL-OUR-AGGREGATES rule 40 prefix '203.0.113.0/24'

The binding that is already in place, and that you should read back rather than assume, is the neighbour’s export route-map:

set protocols bgp neighbor 192.0.2.1 address-family ipv4-unicast route-map export 'RM-TO-ISP-A'

Verification

1. The prefix is advertised. show ip bgp neighbors 192.0.2.1 advertised-routes contains 203.0.113.0/24, and the total has gone from three to four. Exactly one more.
2. The right rule fired. Run clear route-map, refresh outbound, then read the hit counts with vtysh. A prefix that is meant to be prepended towards ISP-A and is instead being permitted by the general rule is a different bug wearing the same fix, and the hit counts are the only place it shows.
3. The peer accepted it. advertised-routes states what this router sent, not what ISP-A installed. Confirm from the other side - ask the peer, or read the prefix on a public looking glass in a network that does not transit ISP-B.
4. The AS path is the intended one. Read the path as seen externally and check it against the traffic-engineering policy for this prefix rather than assuming the prepend behaved.
5. The drain test passes. Drain ISP-B again, in a window, and confirm 203.0.113.0/24 stays reachable from outside. This is the only check that closes the incident, and it is the one that gets skipped because the service is working by then.
6. Nothing else is single-homed. Reconcile the advertised prefixes on both sessions against the list this AS intends to originate, and treat a difference in either direction as a finding.

Prevention

• One origination set, four consumers. The network statements, the outbound policy list, the IRR route objects and the RPKI ROAs are four copies of one fact. Generate them from a single source or reconcile them on a schedule. A change that updates three of four is the expected output of a human checklist, not an unusual lapse.
• Reconcile advertised-routes against intent, per session, on a timer. The missing advertisement produced no alert and would have produced none for years. A scheduled diff in both directions catches the prefix you failed to advertise and the prefix you advertised by accident with the same job.
• Make “no outbound policy” a monitored state. An empty filter is invisible in every configuration review, because there is nothing on the page to review. The ISP-B session was simultaneously the reason the service worked and the largest leak risk in the estate.
• Put the drain in the calendar, not in the incident. A redundant path that has never been tested alone has never been tested. This entire incident is a scheduled exercise that nobody had scheduled.
• Distrust “it works via the other path”. Redundancy does not remove defects; it converts loud ones into latent ones, timed to surface at the exact moment you are relying on it.
• When a new prefix goes live, prove both paths on day one. The acceptance test for a new public prefix should include reading it from a looking glass behind each upstream separately. That test takes five minutes and would have caught this three weeks ago.