BGP Prefix Count Grows Unexpectedly · Break/Fix · VyOS

Reported symptoms

Three tickets, three teams, three weeks of not talking to each other.

On Thursday the monitoring system alerted that free memory on rtr-edge-1 had crossed the low-water mark. The on-call engineer looked at a five-day graph, saw a smooth upward slope with no step in it, wrote “gradual growth, no single event, keep watching” and closed it. Smooth slopes look like leaks, and leaks are next sprint’s problem.

On Friday the network team reported that a planned bounce of the ISP-B link — routine carrier maintenance, done quarterly — took six minutes and eleven seconds to reconverge instead of the usual few seconds. Traffic on that path was black for the duration and no error counter moved.

On Monday finance reported that the ISP-A circuit, a regional peering link bought for regional traffic, had tripled its 95th-percentile egress over the weekend. ISP-B’s had fallen by roughly the same volume. No change ticket explains a traffic-engineering decision that weekend, because nobody made one.

Underneath all three, the things that did not happen matter more:

Both BGP sessions have been Established for eleven days. No flap, no NOTIFICATION, no adjacency event.
The maximum-prefix limit configured on the ISP-A neighbour has not fired. The on-call engineer read that as evidence that BGP was not involved and went looking at the hypervisor.
ISP-A, asked directly, confirmed in writing that they have not changed their advertisement policy in months. They are telling the truth.

The estate

rtr-edge-1 is a VyOS 1.5 LTS router in AS 64512 with two upstreams and a deliberately small routing table:

ISP-A, peer 192.0.2.1, AS 64500. A regional peering arrangement. The contract describes about 2,000 prefixes.
ISP-B, peer 198.51.100.1, AS 64600. Partial transit plus a default route, about 4,500 prefixes.

The design figure for the whole table is around 7,000 prefixes and the router is sized for it: a 4 GB virtual machine. That is an ordinary shape for a regional edge router, and it is why this failure is severe. A box built for a million routes absorbs another million; a box built for seven thousand does not.

Inbound policy on each session is a prefix-list. PL-ISP-A-IN enumerates the ranges the contract covers and relies on the implicit deny at the end of the list for everything else — the normal and correct idiom, and also the part of the design that is invisible in the configuration, because an implicit deny is not written anywhere.

Evidence provided

Read-only / Safethe design figure for 192.0.2.1 is about 2000

vyos@rtr-edge-1:~$ show ip bgp summary

Neighbor        V    AS   MsgRcvd  MsgSent  Up/Down   State/PfxRcd
192.0.2.1       4  64500    418277     1902  11d02h32m    968412
198.51.100.1    4  64600      9114     1898  11d02h31m      4517

Illustrative output

Read-only / Safethe estate baseline is about 7000

vyos@rtr-edge-1:~$ show ip route summary

Route Source         Routes               FIB
connected            14                   14
static               6                    6
ospf                 41                   41
bgp                  972918               972918
------
Totals               972979               972979

Illustrative output

Read-only / Safethe limit is real, and it is five hundred times the expected value

vyos@rtr-edge-1:~$ show ip bgp neighbors 192.0.2.1

BGP neighbor is 192.0.2.1, remote AS 64500, local AS 64512, external link
BGP state = Established, up for 11d02h32m
For address family: IPv4 Unicast
Inbound path policy configured
Incoming update prefix filter list is PL-ISP-A-IN
Maximum prefixes allowed 1000000
Threshold for warning message 75%
968412 accepted prefixes

Illustrative output

Read-only / Safeone line, five days old, in a log with no alert attached

vyos@rtr-edge-1:~$ show log frr | grep -i maximum-prefix

%BGP-4-MAXPFX: 192.0.2.1 (64500): 750001 prefixes, 75% of maximum 1000000

Illustrative output

Read-only / Safethe head and tail of the list - read rule 100

vyos@rtr-edge-1:~$ show configuration commands | match PL-ISP-A-IN

set policy prefix-list PL-ISP-A-IN rule 10 action 'permit'
set policy prefix-list PL-ISP-A-IN rule 10 prefix '198.51.100.0/24'
set policy prefix-list PL-ISP-A-IN rule 20 action 'permit'
set policy prefix-list PL-ISP-A-IN rule 20 prefix '198.51.100.0/22'
set policy prefix-list PL-ISP-A-IN rule 20 le '24'
[... rules 30 to 90 enumerate the remaining contracted ranges ...]
set policy prefix-list PL-ISP-A-IN rule 100 action 'permit'
set policy prefix-list PL-ISP-A-IN rule 100 prefix '0.0.0.0/0'
set policy prefix-list PL-ISP-A-IN rule 100 le '32'

Illustrative output

Read-only / Saferevision 1 is five days old

vyos@rtr-edge-1:~$ show system commit

0   2026-08-15 09:41:02 by vyos via cli   firewall: add NTP source rule
1   2026-08-10 16:22:47 by vyos via cli   fix partner prefix visibility ISP-A
2   2026-07-29 11:08:13 by vyos via cli   ospf: passive default on WAN
3   2026-07-14 08:55:31 by vyos via cli   monitoring: add snmp community

Illustrative output

Work the evidence before reading on

Everything above is consistent with a healthy BGP process. That is the difficulty. Nothing is broken; something is permitted that was not permitted before.

Compare the two State/PfxRcd figures against the two contracts. One is off by a factor of five hundred. Which of the three tickets does that number explain, and which does it not?
The limit is 1,000,000 and the count is 968,412. Ask what that limit was protecting against, and whether any misconfiguration ISP-A is capable of would have crossed it.
show system commit names a change five days old. Read its comment as what the engineer intended, then read rule 100 as what they did.
Put the commit, the MAXPFX warning, the memory alert and the traffic shift on one time axis and see which came first.

Before continuing: the upstream did not change what it sends. What changed about what this router is willing to accept, and where in the configuration is the thing that used to say no?

Root cause

1. The boundary lived in the local prefix-list, not at the peer

ISP-A has always offered its full view on that port. The 2,000-prefix figure in the contract describes what this router is entitled to use, not what arrives on the wire, and no mechanism at ISP-A’s end was enforcing it. The enforcement was entirely local: PL-ISP-A-IN permitted the contracted ranges, and everything else hit the implicit deny at the end of the list and was discarded.

That is the correct idiom, and it is structurally fragile in one specific way: the rule doing the work is not written down. A reviewer reading PL-ISP-A-IN sees a list of permits. The deny that turns those permits into a boundary exists only in the semantics of prefix-lists — a list with a single permit rule accepts only what that rule matches, and everything else is rejected by the implicit deny.

2. One rule removed the boundary, while fixing a real ticket

The commit five days ago was a response to a genuine fault: the partner network 203.0.113.0/24 was unreachable through ISP-A. The diagnosis was right — PL-ISP-A-IN had no rule matching it, so the route was arriving from ISP-A and being discarded before it reached the Loc-RIB.

The remediation was one rule too broad. Instead of adding a rule for 203.0.113.0/24, the engineer appended a catch-all:

set policy prefix-list PL-ISP-A-IN rule 100 action 'permit'
set policy prefix-list PL-ISP-A-IN rule 100 prefix '0.0.0.0/0'
set policy prefix-list PL-ISP-A-IN rule 100 le '32'

0.0.0.0/0 le 32 matches every IPv4 prefix of every length. The partner route appeared, the ticket was verified as fixed and closed, and the implicit deny that had been the boundary for years stopped being reachable. From that commit onward, everything ISP-A offers is accepted.

3. Every symptom is a consequence of table size

Once the table is a hundred and forty times larger than the design, the three tickets stop looking unrelated:

Memory. BGP memory tracks the number of paths held. The slope is smooth because the table filled as the peer’s updates arrived and were re-advertised through the update groups, not because anything leaked.
Convergence. zebra is single-threaded: every RIB update flows through one thread regardless of how many cores the router has. A reconvergence that walks nine hundred thousand routes takes minutes where one that walks seven thousand takes seconds. Adding vCPUs would not have helped, which is why the capacity theory produced no fix.
Traffic and the bill. Before the commit most destinations had exactly one candidate path, via ISP-B. Afterwards they have two, and the best-path algorithm chooses between them: where ISP-A offers a shorter AS path it wins at step 4 and the egress moves. Nobody made a traffic-engineering change; the candidate set changed.

A limit far above any plausible value is not a limit

maximum-prefix 1000000 looks like protection in a configuration review. It is the reason nobody was paged.

The limit came from a device template written for a router that took full tables, and was never re-derived when this router was built for a 2,000-prefix peering. Ask what misconfiguration on ISP-A’s side would have crossed a million: their entire table does not reach it. The limit could not fire for any fault this peer is capable of producing, which makes it decoration.

A limit of a few thousand, derived from the contracted count plus a growth margin, would have shut the session down within seconds of the bad commit — at 16:22 on a Monday, with an engineer still at the keyboard and the change fresh in the commit log. That is a better outcome than five days of drift and three misfiled tickets.

The warning threshold did fire, exactly as designed, into a log file nobody reads. An unmonitored warning is the same as no warning.

Resolution

Decide the order and the owner before typing. A tight maximum-prefix applied while the peer is still sending nearly a million prefixes shuts the session down on the next update - acceptable in a maintenance window, an outage in business hours. Fix the filter first, the limit second.
Capture the before state into the ticket: show ip bgp summary, show ip route summary, and show ip bgp 203.0.113.0/24. The last one is the prefix the original ticket was about, and you must not re-break it.
Replace the catch-all with the rule the original ticket needed: delete policy prefix-list PL-ISP-A-IN rule 100, then set policy prefix-list PL-ISP-A-IN rule 60 action permit and set policy prefix-list PL-ISP-A-IN rule 60 prefix 203.0.113.0/24.
Read the diff with compare, then commit-confirm 10. This is inbound policy on a transit session; if the new list is wrong in the other direction you lose routes rather than gain them, and the timer is what gets them back.
Re-apply the corrected filter with clear ip bgp 192.0.2.1 soft in. That uses route-refresh and does not reset the session. A hard clear ip bgp 192.0.2.1 would withdraw every prefix from this peer and reconverge the estate for no reason.
Watch the count fall in show ip bgp summary, and give the FIB time to drain - the BGP table shrinks first and the kernel follows. Confirm and save once the count is at the contracted order of magnitude and 203.0.113.0/24 is still present.
Only now re-derive the limit, from the contracted count plus a growth margin rather than from an imagined ceiling, and commit it as a separate change with its own comment so it is reviewable on its own.
Treat the traffic shift as a separate work item. Confirm from the monitoring system that egress has returned to the intended upstream; do not infer it from the size of the routing table.

The configuration form for the limit follows the current VyOS tree, where the local ASN sits under system-as and per-peer settings sit under neighbor:

set protocols bgp system-as 64512
set protocols bgp neighbor 192.0.2.1 address-family ipv4-unicast maximum-prefix 4000

Verification

The prefix count is back at the contracted order of magnitude. show ip bgp summary shows a figure near 2000 for 192.0.2.1. A count of 40000 is still a failure - "lower than it was" is not the check.
The reduction reached the FIB. show ip route summary is back at the estate baseline. The BGP table and the kernel FIB are different tables and only one of them forwards packets.
The original ticket is still fixed. show ip bgp 203.0.113.0/24 shows the prefix and show ip route 203.0.113.0/24 shows it installed. A fix that quietly reverts the ticket it descends from will be undone by whoever reads that ticket next.
The new limit is capable of firing. Read it back with show ip bgp neighbors 192.0.2.1 and compare it against the observed steady-state count; satisfy yourself that the gap is a growth margin and not a hole.
The host recovered: free memory and bgpd CPU are back in their pre-incident bands, measured after the table has settled rather than during the drain.
Convergence is fast again. Re-run the link bounce in a maintenance window and time it. This is the only check that closes the second ticket, and it is the one most likely to be skipped.
The egress split matches the design, read from the monitoring system over a full business day rather than from the router.
The MAXPFX warning now reaches a person. Confirm the delivery path end to end rather than confirming that the log line exists.

Prevention

Make the boundary visible. A prefix-list whose last rule is 0.0.0.0/0 le 32 is not a filter. The diff that adds one is four lines long and reads like a fix, so it needs to fail review mechanically rather than depend on a reviewer noticing.
Derive the limit from the contract, not from fear. Record the contracted prefix count per peer alongside the inbound policy name and the configured maximum-prefix. That inventory is what makes a limit reviewable; without it any number looks defensible.
Alert on the prefix count as a series. The count per peer is the highest-value BGP number to monitor, and a band around the expected value catches growth days before it becomes memory pressure.
Give the threshold warning a destination. The 75% warning fired correctly here and changed nothing, because it went to a file.
Fix the ticket you have. When one prefix is missing, the remediation is a rule for that prefix. “So this cannot happen again” is the sentence that precedes most catch-alls, and it is worth treating as a prompt to slow down rather than to widen.
Size the router for the table you intend to hold, then hold it there. A 4 GB edge router with a 7,000-prefix design is sound. It stops being sound the moment nothing enforces the 7,000.

What if the engineer had added the catch-all to the outbound list instead?

Then this would have been a shorter incident with a far worse blast radius, and it would not have been yours to diagnose — your upstream would have reported it to you.

An inbound catch-all admits the Internet into your router. An outbound catch-all advertises your router to the Internet: every prefix in the Loc-RIB, including everything learned from ISP-B, offered to ISP-A with your AS in the path. That is the shape of a route leak, and unlike this incident it damages other people’s traffic. It also tends to be caught within minutes, because transit providers apply their own inbound filters and prefix limits and one of them usually shuts the session.

The asymmetry is worth internalising. An inbound filter mistake costs you memory, convergence and money, and can run for days unnoticed. An outbound one costs other people reachability and is noticed immediately. Both come from the same four-line diff, distinguished only by the direction the list is bound in — which is a good argument for knowing, without looking it up, which of your prefix-lists is bound which way.

Why this matters

The prefix count is the health metric no single team owns. Memory alerts belong to the platform team, convergence time to the network team, and the transit bill to finance, so one cause presented itself three times to three audiences, each with a locally plausible theory and none with the number that unified them.

The transferable habit is to hold a per-peer expected prefix count as a first-class figure — written in the inventory, enforced by a maximum-prefix derived from it, monitored as a series — so that “how many prefixes is this peer sending” has a known answer rather than being something you measure after three tickets have been misfiled.

The second habit is narrower and just as valuable: when a specific thing is missing, fix the specific thing. Widening a filter so a class of ticket becomes impossible is how boundaries stop existing, and the resulting configuration passes every review because nothing in it looks wrong.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

The estate

Evidence provided

Work the evidence before reading on

Root cause

1. The boundary lived in the local prefix-list, not at the peer

2. One rule removed the boundary, while fixing a real ticket

3. Every symptom is a consequence of table size

Resolution

Verification

Prevention