Skip to main content
RunBook Academy

← All break/fix scenarios in VyOS

advancedvyos-bgp-leak~30 min

Internal Prefixes Advertised Externally

Reported symptoms

  • ●The eBGP session to the first transit provider comes up, runs for about a minute, drops, and returns roughly fifteen minutes later; it has done this continuously for three weeks and on-call has cleared it by hand eleven times
  • ●That provider has been asked to investigate twice and reports their side healthy, with no errors on the circuit and no configuration changes
  • ●All outbound traffic now leaves through the second transit provider, whose circuit runs close to its committed rate all day, and the month-to-date 95th percentile is on track to roughly double the bill
  • ●A templated route-leak notice from the second provider arrived in the shared peering mailbox nine days ago and has not been read by anybody
  • ●A forty-minute maintenance window on the second provider last Tuesday produced a complete internet outage for the company; the post-incident report attributes it to a concurrent fault on the first provider and recommends escalating with them
  • ●Nothing inside the company is broken. No user has reported anything. Every internal service, every site-to-site path and every application behaves exactly as it did a month ago
  • ●One change touched the border router in three months: three weeks ago a redistribution statement was added so a new interconnect subnet would be visible to the other sites over iBGP. It worked, it was verified at the other sites, and the change was closed

Evidence

  • · `show ip bgp summary` shows the first transit neighbour in Active or Idle with an uptime of a few minutes, and the second Established for three weeks with a PfxSnt column reading in the hundreds
  • · `show ip bgp neighbors 198.51.100.2` reports the last reset as a notification received from the peer, Cease with the maximum-prefix subcode
  • · `show ip bgp neighbors 198.51.100.6 advertised-routes` lists RFC 1918 space - 10.0.0.0/8 aggregates, a set of 10.x site subnets, a 172.16.x management range - alongside both transit point-to-point subnets and the peering subnets themselves
  • · The company originates exactly one prefix by design, and `show ip bgp summary` reports it advertising several hundred
  • · In `show ip bgp`, the leaked prefixes carry the incomplete origin code rather than the IGP code the single authorised prefix carries
  • · `show configuration commands | match redistribute` returns a redistribute statement on the BGP process with no route-map attached to it
  • · Neither eBGP neighbour has an outbound route-map or export filter configured; `show configuration commands | match neighbor` shows peer addresses, remote AS numbers and passwords, and nothing that constrains what is sent
  • · `show system commit` shows one commit in three months, timestamped three weeks ago, matching the date the first transit session began flapping
Diagnosis and resolutionclick to reveal

Root cause

Two independent gates were left open and the incident needed both. A redistribution statement was added to the BGP process three weeks ago, with no route-map attached, so every connected subnet on the border router entered the BGP table - the internal ranges, the management network, the point-to-point links to both transit providers and the peering subnets. That alone would have been survivable, because what a neighbour receives is decided by that neighbour's outbound policy rather than by what is in the table. But neither eBGP session has an outbound policy at all, and a BGP speaker with no export filter advertises everything it has. So a change intended to make one interconnect visible over iBGP made several hundred internal prefixes visible to the public internet. The symptom pattern is the two providers behaving differently in response to the same advertisement. The first has a maximum-prefix limit sized for the single prefix the company is registered to originate; the flood exceeds it, that provider sends a Cease notification and tears the session down, its restart timer brings it back about fifteen minutes later, the flood repeats, and the cycle has been running ever since. The second has no such limit, keeps the session up, accepts the advertisement and sends a notice about it. Everything else follows: with one transit down, all egress moved to the surviving provider and its bill with it, and when that provider took a maintenance window the company had no second path and went dark.

Remediation

Resist the instinctive fix. Adding a prefix-list that denies RFC 1918 stops the prefixes you thought of and leaves the transit point-to-point subnets, the peering subnets, and next year's acquisition space free to leak in exactly the same way. The correct shape is default-deny with an explicit permit of the prefixes you are authorised to originate, applied outbound on every eBGP session. Sequence the work deliberately, because the order changes the blast radius. Apply the outbound filter first: it stops the external damage immediately, it has no internal impact whatsoever, and it lets the flapping session come back up and stay up. Apply it with an outbound soft clear rather than a hard reset so the working session is not torn down to fix a problem it does not have. Only then, as a second change with its own testing and its own window, attach a route-map to the redistribution so the internal prefixes stop entering the table at all - and check before you do that the prefixes the original change was made for are still carried, because fixing a leak by breaking the thing the leak came from is a fresh incident. Both gates are worth closing even though either one would have prevented this, since a router that depends on a single filter is one review miss away from repeating the whole event. Finish with the communications, which are not optional here: reply to the leak notice, tell the first provider that the session teardowns were caused by your announcements rather than by their circuit so they stop spending engineer time on a fault that does not exist, and correct the outage report that currently blames them.

Verification

Verify what you advertise from outside your own router. `show ip bgp neighbors ADDR advertised-routes` on each session should list exactly the authorised prefixes and nothing else, and the count in the PfxSnt column of `show ip bgp summary` should equal the number you originate by design rather than merely looking smaller than before. Then confirm it independently - a public looking glass, or the provider reporting what they received - because your own router showing what it intends to send is a claim, and the peer showing what arrived is a measurement. Check the recovered session twice, more than fifteen minutes apart: the flap cycle is roughly that long, so a session inspected at the wrong moment in the cycle has always looked healthy. Read the origin codes in `show ip bgp`; after the redistribution route-map is in place the authorised prefix should still carry the IGP origin and the incomplete-origin clutter should be gone from the table entirely, which is a stronger statement than the outbound filter alone can make. Confirm the internal side has not regressed: the interconnect subnet the original change was made for must still be visible to the other sites. Watch egress rebalance across the two circuits over a full day rather than declaring success on the first minute of traffic. And close the external loop by confirming with the first provider that their maximum-prefix counter has stopped tripping, which is the only evidence that comes from outside your own configuration.

Prevention

A route leak is an externality: the damage lands on other networks, nothing inside your own breaks, and the only feedback available is a peer choosing to protect itself or to complain. That makes structural controls the whole answer and detection a distant second. Every eBGP session gets a default-deny outbound policy at the moment it is created, before it is ever brought up, and a session without one should not survive review. Treat a bare redistribute statement into BGP as an error in the same way, because the process-wide scope of redistribution is the part that surprises people: it decides what enters the table, not what leaves toward any particular neighbour, and those two questions have separate answers that must both be written down. Alert on advertised prefix count per neighbour against an exact expected number rather than a threshold - you know precisely how many prefixes you originate, so any deviation at all is an incident, and this alarm would have fired within seconds of the commit. Register what you originate and ask your transits to filter you strictly; being filtered by an upstream is a service they perform for you, and the provider in this incident that kept tearing the session down was the one doing its job. Give the peering mailbox a named owner and a response time, since the leak notice was the fastest detection anyone had and it sat unread for nine days. And add a rule to the incident playbook: when a peer's session drops repeatedly, look at what you are sending before raising a fault against them.

Reported symptoms

Nobody inside the company has noticed anything.

That is the first thing to hold on to, because it shapes every decision made over the last three weeks. Every application works. Every site-to-site path works. No user has raised a single ticket. The problems are all in places where the network team touches the outside world.

The eBGP session to the first transit provider will not stay up. It establishes, runs for around a minute, drops, and comes back roughly fifteen minutes later. It has done this without interruption for three weeks. On-call has cleared it by hand eleven times. The provider has been asked to investigate twice and has reported, both times, that their side is clean: circuit up, no errors, no changes.

Because that session is effectively down, all outbound traffic now leaves through the second provider. Their circuit runs close to its committed rate through the whole working day, and the month-to-date 95th percentile is on course to roughly double the bill. That is a separate ticket, with the person who signs off the invoices, who has been told it is a temporary consequence of a carrier fault.

Nine days ago the second provider’s NOC sent a route-leak notice to the company’s shared peering mailbox. It is a templated message. The mailbox has no owner. It has not been read.

Last Tuesday the second provider took a scheduled forty-minute maintenance window and the company lost all internet access for the duration. The post-incident report describes this as being single-homed during a maintenance window because of a concurrent fault on the first provider, and recommends escalating with them. Nobody in the review asked why the first provider had been down for two weeks.

One change touched the border router in three months. Three weeks ago a redistribution statement was added so a new interconnect subnet would appear in BGP and be visible to the other sites over iBGP. It worked. Somebody at each of the other two sites confirmed they could see the prefix. The change was closed the same afternoon.

The estate

rtr-border-1 is the VyOS 1.5 LTS border router. It runs one BGP process in AS 64496, and that one process carries three kinds of session:

  • eBGP to the first transit provider, AS 64497, over 198.51.100.0/30.
  • eBGP to the second transit provider, AS 64498, over 198.51.100.4/30.
  • iBGP to the border routers at the two other company sites.

The addresses and AS numbers above are the ranges reserved for documentation, which is worth saying out loud in a file about leaking address space.

The company originates exactly one prefix: a single /24 of its own public space, advertised with a network statement. That number — one — is written in the design document and has been true for six years.

The border router also carries, on its interfaces, the things every border router carries: the two transit point-to-point subnets, a management range, the internal transit to the campus core, and the new interconnect subnet from three weeks ago. Its routing table holds the internal estate learned from the IGP.

Evidence provided

Read-only / Safethe design says this router originates one prefix
vyos@rtr-border-1:~$ show ip bgp summary
Neighbor        V   AS  MsgRcvd MsgSent  Up/Down State/PfxRcd  PfxSnt
198.51.100.2    4 64497     117     964 00:00:47      Active        0
198.51.100.6    4 64498  914322  118755 3w0d18h      918341      617

Illustrative output

Read-only / Safethe peer is telling you, in plain words, what you did
vyos@rtr-border-1:~$ show ip bgp neighbors 198.51.100.2
BGP neighbor is 198.51.100.2, remote AS 64497, local AS 64496, external link
BGP state = Active
Last reset 00:00:47, due to NOTIFICATION received from peer:
Cease/Maximum Number of Prefixes Reached

Illustrative output

Read-only / Safesix of several hundred - note the origin code on the last line
vyos@rtr-border-1:~$ show ip bgp neighbors 198.51.100.6 advertised-routes
   Network          Next Hop     Metric LocPrf Weight Path
*> 10.0.0.0/8       0.0.0.0           0         32768 ?
*> 10.14.0.0/16     0.0.0.0           0         32768 ?
*> 172.16.9.0/24    0.0.0.0           0         32768 ?
*> 198.51.100.0/30  0.0.0.0           0         32768 ?
*> 198.51.100.4/30  0.0.0.0           0         32768 ?
*> 203.0.113.0/24   0.0.0.0           0         32768 i

Illustrative output

show configuration commands | match redistribute returns a redistribution statement on the BGP process with no route-map attached. show configuration commands | match neighbor returns peer addresses, remote AS numbers and session passwords, and nothing at all that constrains what is sent.

Work the evidence before reading on

The router is healthy. Every interface is up, the routing daemons are running, and there is no error anywhere in the logs except the notifications from a peer.

  1. Compare the PfxSnt column with the design document. What is the expected number, and what would you have needed in place for the difference to have been noticed three weeks ago?
  2. Read the origin codes in the advertised-routes output. One line differs from the others. What does that difference tell you about how each prefix got into the table?
  3. Two providers received the same advertisement and behaved completely differently. Before deciding which one has a problem, work out what each of them actually did and whether it was correct.
  4. Three weeks of a flapping session, a doubled bill, an unread notice and a total outage — and not one internal symptom. What does that combination say about who is being harmed, and therefore about which controls could ever have caught this?
  5. There are two separate decisions a BGP speaker makes here: what gets into its table, and what it sends to a given neighbour. Which of those two is misconfigured?

Before continuing: the session that keeps going down is the one behaving correctly.

Root cause

1. Two gates, both open

BGP asks two independent questions, and this router answers both of them wrongly.

The first is what enters the table. The redistribution statement added three weeks ago has no route-map on it, so every connected subnet on the border router became a BGP route: the internal ranges, the management network, both transit point-to-point links, and the peering subnets themselves. The origin codes in the advertised-routes output record this precisely — a prefix originated by a network statement carries the IGP origin, and a redistributed one carries incomplete. One line in that output says i and the rest say ?, and that single column separates the prefix the company is entitled to announce from the several hundred it is not.

The second question is what leaves toward a particular neighbour, and it is answered by that neighbour’s outbound policy. Neither eBGP session has one. A BGP speaker with no export filter advertises everything in its table, so the answer to the second question was: all of it.

Either gate alone would have contained this. A filtered redistribution would have kept the internal prefixes out of the table. A default-deny export policy would have kept them off the wire regardless of what was in the table. The change three weeks ago was a mistake; the reason it became an incident is that the router had no second line.

2. Two providers, two policies, and the wrong one got blamed

The same announcement reached both transits and they responded in opposite ways.

The first provider has a maximum-prefix limit on the session, sized for the single prefix the company is registered to originate. The flood exceeded it. That provider sent a Cease notification with the maximum-prefix subcode and tore the session down — which is the exact string sitting in show ip bgp neighbors and has been sitting there for three weeks. Their restart timer brings the session back around fifteen minutes later, the router immediately re-sends the same flood, and the cycle repeats.

The second provider has no such limit on this session. It kept the session up, accepted the advertisement, and sent a notice about it to a mailbox nobody reads.

3. Everything else is downstream of one session being gone

With the first transit effectively down, all egress moved to the second, and its circuit has been running at commit rate ever since. That is the bill.

With only one usable transit, the company was single-homed without knowing it. When the second provider took its maintenance window on Tuesday, there was nothing to fail over to, and the outage report blamed a concurrent fault on the first provider — a fault which does not exist, has never existed, and which somebody is now scheduled to escalate.

Resolution

  1. Establish exactly what you are authorised to originate, from the registry and the design document, not from the router. This is a short list and it is the input to everything below.
  2. Do not write a filter that denies RFC 1918. It stops the prefixes you thought of and leaves the transit point-to-point subnets, the peering subnets and any future address space free to leak in exactly the same way. Default-deny with an explicit permit of the authorised list is the shape that survives the next change.
  3. Apply the outbound policy first, to both eBGP sessions. This is the step that stops the external harm, and it has no internal blast radius at all - nothing inside the company depends on what these two neighbours receive.
  4. Push it with an outbound soft clear rather than a hard reset. The surviving session is carrying the entire company and does not need to be torn down to fix a problem it is not causing.
  5. Watch the first session recover. Once the advertisement is one prefix instead of several hundred, the maximum-prefix limit stops tripping and the session should establish and stay up through its next restart interval.
  6. Attach a route-map to the redistribution as a separate change, in its own window, with its own test. Before you commit it, confirm which prefixes the original interconnect change actually needed, because fixing the leak by breaking the thing that caused it is a fresh incident at the other two sites.
  7. Close both gates even though either would have sufficed. A router protected by a single filter is one review miss from repeating this in full.
  8. Reply to the leak notice with what happened and when it was fixed. It is the only detection that worked.
  9. Tell the first provider that the teardowns were caused by your announcements. They have spent three weeks of engineer time on a fault that does not exist, and your own outage report currently blames them for it. Correct the report.

Verification

  1. The advertised set is exactly the authorised set. show ip bgp neighbors ADDR advertised-routes on each session lists the authorised prefixes and nothing else, and PfxSnt in show ip bgp summary equals the number you originate by design rather than merely being smaller than it was.
  2. Confirm it from outside. A public looking glass, or the provider reporting what they received. What your router says it intends to send is a claim; what the peer received is a measurement, and this whole incident is about the gap between the two.
  3. Check the recovered session twice, more than fifteen minutes apart. The flap cycle is about that long, which is why the session has looked fine to whoever happened to check it at the right moment.
  4. Read the origin codes again once the redistribution route-map is in. The incomplete-origin clutter should be gone from the table entirely, not merely filtered on the way out - that is a stronger statement than the export filter alone can make.
  5. Confirm the internal side did not regress. The interconnect subnet the original change was made for is still visible at both other sites. Ask them, rather than inferring it from this router.
  6. Watch egress rebalance across the two circuits over a full working day, not over the first minute of traffic.
  7. Close the external loop: the first provider confirms their maximum-prefix counter has stopped tripping. That is the only piece of evidence in the whole verification that does not come from your own configuration.

Prevention

  • Give every eBGP session a default-deny outbound policy at the moment it is created, before it is brought up for the first time. A session with no export policy should not survive review. This single rule would have made the change three weeks ago harmless.
  • Treat a bare redistribute statement into BGP as an error. Redistribution is process-wide: it decides what enters the table, not what leaves toward any particular neighbour. Those are two questions with two answers and both have to be written down.
  • Alert on advertised prefix count per neighbour against an exact number. Not a threshold — you know precisely how many prefixes you originate. Any deviation at all is an incident, and this alarm fires within seconds of the commit rather than after three weeks.
  • Register what you originate and ask your transits to filter you strictly. Being filtered by an upstream is a service they perform for you. The provider that kept tearing the session down was doing its job, and a network with two such providers would have had two loud alarms instead of one.
  • Give the peering mailbox a named owner and a response time. The leak notice was the fastest detection available to anybody and it sat unread for nine days.
  • Add one line to the incident playbook: when a peer session drops repeatedly, read the reset reason and check what you are advertising before raising a fault. Eleven manual clears and two carrier tickets were spent not doing that.