critical riskservice affecting~60 min

Runbook: Respond to a BGP Route Leak

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The direction is established before anything is changed. show ip bgp neighbors PEER received-routes answers "is it arriving"; show ip bgp neighbors PEER advertised-routes answers "are we sending it". These are different incidents with different owners
· The peer contract is in hand: which prefixes this session is supposed to carry in each direction, and the expected count. Without it there is no way to say a count is wrong, only that it is large
· Whether soft-reconfiguration inbound is configured on the session: show configuration commands | match soft-reconfiguration. Without it the pre-policy view does not exist and received-routes returns nothing
· Out-of-band access is confirmed. Containment on a transit session can remove the path you are connected over
· The commit archive is readable and its newest entry is accounted for: show configuration commit. A leak that began at a commit boundary is a local change, not a peer problem
· The incident is opened with a security or NOC owner before containment, because an outbound leak has an external audience and a notification clock
· commit-confirm is the agreed apply method for every change in this runbook, and the timer length is chosen before the first commit

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Establish the DIRECTION before anything else. Run show ip bgp neighbors PEER received-routes and show ip bgp neighbors PEER advertised-routes on the session under suspicion and compare each against the contracted prefix set. Inbound and outbound leaks share a name and nothing else.
2Capture the evidence that expires, in this order: show bgp summary, show ip bgp neighbors PEER received-routes, show ip bgp neighbors PEER routes, show ip bgp neighbors PEER advertised-routes, show ip route summary. Containment destroys all of it.
3Characterise the leak with the offending prefix, not the counts: show ip bgp 198.51.100.0/24 for the paths and origin AS, show ip bgp regexp for the AS-path shape, show rpki cache-server and the validation state if RPKI is deployed, and whois -h whois.radb.net for the registered origin.
4Decide whether the leak is arriving, leaving, or transiting - a prefix that arrives from one peer and leaves to another is you providing transit you did not sell, and it is the most damaging of the three.
5Choose containment against blast radius, not against convenience: filter beats limit, limit beats shutdown, and shutdown beats deleting the peer. HOLD is a legitimate choice with an owner and an end time when the leak is inbound, bounded, and the session carries more than it costs.
6For an INBOUND leak, commit the corrective inbound prefix-list or as-path-list on the session with commit-confirm. The session stays up and the local table is bounded.
7For an OUTBOUND leak - where you are the leaker - stop the advertisement first and diagnose afterwards. Commit the outbound prefix-list restricting exports to your own allocation, or set protocols bgp neighbor PEER shutdown if the correct outbound set is not yet known.
8Refresh the session with clear ip bgp PEER soft in or soft out rather than resetting it. A hard reset re-runs the full table exchange and re-converges every prefix the session carries, including the healthy ones.
9Verify the containment on the router: the received and advertised counts are back inside the contracted band, and the specific prefix is absent from the direction it should not be in.
10Verify the containment from OUTSIDE the router. An outbound leak is only closed when the neighbouring AS stops seeing it; your own advertised-routes proves what you sent, not what propagated.
11Only after containment holds, fix the mechanism: the missing inbound filter, the missing maximum-prefix, the redistribution with no route-map, or the outbound filter that permitted more than your allocation.
12Confirm, save, and write the timeline - first advertisement, detection, containment, external confirmation - because the interval between the first two and the interval between the last two have different fixes.

4 · Verification

Confirm the procedure actually fixed the problem.

✓show bgp summary shows the received prefix count for the session inside the band the peer contract predicted, not merely lower than it was
✓show ip bgp neighbors PEER advertised-routes contains only the prefixes you are entitled to export, checked prefix by prefix against the allocation rather than by count
✓The offending prefix is absent from the direction it should not be in: show ip bgp 198.51.100.0/24 no longer shows the leaked path, or no longer shows it via this session
✓The BGP table and the FIB agree: show ip route 198.51.100.0/24 resolves through the intended path, because a route removed from BGP and left in the FIB is still carrying traffic
✓The neighbouring AS confirms it no longer sees the leaked prefixes from your ASN - external confirmation, not your own advertised-routes
✓The filter that should have prevented this exists and is bound: show configuration commands | match prefix-list shows it attached to the address family, and show policy prefix-list shows the rules
✓maximum-prefix is set on the session with a value derived from the contract, and the session is not sitting in Idle because the cap fired during containment
✓The change is confirmed and saved: confirm ran before the timer expired, save afterwards, and show log commit | last 5 shows both

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶While the commit-confirm timer is running, doing nothing is the rollback. This matters more here than in a planned change: the operator is working at speed on incomplete evidence, and the timer is the only mechanism that does not require a second correct decision
↶A filter that turned out to be too tight is reverted by correcting the prefix-list rules and re-committing, not by removing the filter. Removing it restores the leak
↶A shutdown applied for containment is removed by deleting the shutdown node and re-committing, once the corrective filter is in place. Bring the session back WITH the filter, never before it
↶To revert the whole containment: rollback 1, compare, commit-confirm 5, then confirm. Read the compare output first - during an incident the previous archive entry may be the configuration that leaked
↶POINT OF NO RETURN: an advertisement that has already propagated. Withdrawing it re-converges the path; it does not undo the traffic that took it, the caches that hold it, or any route-flap dampening penalty the neighbouring AS has now applied to your prefixes
↶POINT OF NO RETURN: traffic that transited you during an outbound leak. It was carried on a path its owner did not choose, which is a disclosure question for the security owner and not a routing one
↶POINT OF NO RETURN: delete protocols bgp neighbor PEER during an incident. It removes the session, the filters, the counters and the evidence in one commit. Use shutdown instead - it is equally effective and reversible

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the peer or provider NOC the moment the direction is established. An inbound leak is their configuration error and only they can stop it at source; an outbound leak is yours and they need to know before their upstreams tell them
· Escalate to the security owner for any outbound leak. Traffic was drawn onto a path its owner did not choose, and that is an incident with an external audience and possible notification obligations
· Escalate to network engineering before shutting down a transit or upstream session. Removing a full-table peer moves every prefix it was best for, and the recovery is a reconvergence rather than a command
· Escalate to the change owner if the newest entry in show configuration commit coincides with the start of the leak - this is a local change to be rolled back, not a peer incident to be filtered
· Escalate to the prefix owner, not the peer, when the evidence is an RPKI-invalid or IRR-mismatched origin. A wrong or stale ROA looks exactly like a hijack from your router and only the owner can tell the difference
· Escalate if the leaked prefixes belong to a customer or a third party. Their traffic is the one that moved, and the decision about disclosure is not the on-call operator to make

“Route leak” is one phrase covering two incidents that share nothing except the word, and the first minute of this runbook is spent deciding which one you are in.

A leak arriving at your router is somebody else’s configuration error landing in your table. It is loud, it is bounded by your own filters, and you can contain it alone, in minutes, without asking anyone’s permission.

A leak leaving your router is your configuration error landing in everyone else’s table. It is quiet - your router looks healthy, your sessions are up, your traffic graphs may even look better - and you cannot see the damage from the box that is causing it. That is the one that ends up on the operators’ mailing list with your ASN in the subject line.

Both are found with the same two commands, and everything after that diverges.

When to use this runbook

A maximum-prefix warning or an Idle (shutdown) state on a session.
A received prefix count materially outside the contracted band.
An upstream, a peer or a customer reports seeing your ASN in a path where it does not belong.
Traffic that used to leave one way is suddenly leaving another, with no local change to explain it.
A prefix you do not own has appeared in your table with a path that makes no commercial sense.

When not to use it

The session is down or flapping - that is vyos-rb-troubleshoot-bgp-session, and it starts from the session state.
A prefix you expect is missing - that is vyos-rb-troubleshoot-missing-route. A leak is unwanted routes present, not wanted routes absent.
You are deliberately changing what the router exports - that is vyos-rb-modify-route-policy-safely, at planned-change pace.
The last commit caused it and the archive holds the good state - that is vyos-rb-rollback-bad-config, and it is faster than anything here.

Blast radius

For an inbound leak: the local RIB, the FIB, and every flow whose best path just changed. On a router carrying a full table, a leak of more-specifics wins on longest-prefix match regardless of policy, so traffic moves before anything in the BGP table looks obviously wrong.

For an outbound leak: everything downstream of every AS that accepted your announcement. That set is not knowable from your router, does not shrink when you withdraw, and includes networks that have no relationship with you.

Step 0: direction, before anything else

Two commands. They decide who owns the incident, what containment is available, and whether there is a clock running.

Read-only / Safearriving, or leaving

show ip bgp neighbors 203.0.113.2 received-routes
show ip bgp neighbors 203.0.113.2 advertised-routes

received-routes is the pre-policy view: what the peer actually sent, before your inbound filters ran. advertised-routes is what you are sending after your outbound policy. Read each against the contracted prefix set for that session - the one written down when the peer was configured - and not against zero, and not against yesterday.

There is a third case that reads as neither and is worse than both: a prefix that arrives from one peer and appears in advertised-routes for another. That is you providing transit between two networks that did not buy it from you. It is the classic RFC 7908 leak, it attracts real traffic, and it saturates links that were never sized for it.

`received-routes` is empty unless somebody planned for this incident

show ip bgp neighbors PEER received-routes reads the pre-policy Adj-RIB-In, which FRR only retains when the session carries soft-reconfiguration inbound. Without it the command returns nothing, and the difference between “the peer never sent it” and “the peer sent it and my filter dropped it” is not answerable from this router.

Enabling it now does not recover the evidence. It triggers a route refresh, and what you then see is what the peer is sending now - which, if the peer has already noticed and fixed their side, is a clean table and no incident.

Use show ip bgp neighbors PEER routes for the post-policy view, which is always available, and record in the incident notes that the pre-policy view was not retained. Then add soft-reconfiguration inbound to the sessions that matter as a follow-up, at planned-change pace, knowing it costs memory.

Step 1: capture what containment destroys

Every containment action in this runbook removes the evidence for it. Take the capture first; it costs under a minute.

Read-only / Safethe state you will be asked to justify the containment from

show bgp summary
show ip route summary
show ip bgp neighbors 203.0.113.2 received-routes
show ip bgp neighbors 203.0.113.2 routes
show ip bgp neighbors 203.0.113.2 advertised-routes
show configuration commit

show bgp summary gives the per-neighbour prefix count, which is the number the incident will be described by. show configuration commit is the one people skip: it names the user, the source address and the timestamp of every recent commit. If the newest entry lands at the start of the leak, this is a local change and the correct runbook is the rollback one, not this one.

Read-only / Safeone session, read against its contract

show bgp summary

Neighbor        V    AS   MsgRcvd  MsgSent  Up/Down   State/PfxRcd
203.0.113.2     4  65100     41207     1893   00:04:19       418922
198.51.100.9    4  64500      2841     2790   19:22:07          312

Illustrative output

The number itself is not the finding. A customer session contracted for a single /24 and its more-specifics, showing four hundred thousand prefixes and an uptime of four minutes, is a peer who has just started re-advertising a full table to a router that never asked for one. The four-minute uptime is the second half of the evidence: the session reset, and the flood arrived with the new table.

Step 2: characterise the leak, not the count

Read-only / Safethe offending prefix, its origin and its path

show ip bgp 198.51.100.0/24
show ip bgp regexp _65100_
show rpki cache-server
show policy prefix-list

show ip bgp for the prefix shows every candidate path with its origin AS and AS-path. Compare the origin against who is entitled to announce it. show ip bgp regexp finds every prefix in the table carrying a given ASN, which is how you size an inbound leak: one prefix is an accident, ten thousand is a policy failure at the peer.

Where RPKI is deployed, the validation state separates a leak from a hijack: an invalid origin is a claim about ownership, not about policy. Where it is not, whois -h whois.radb.net against the prefix gives the registered origin from the IRR, which is weaker evidence and still better than none.

Step 3: choose containment against blast radius

Four options, in the order to prefer them. The wrong choice is not usually the weakest one - it is the strongest one, applied because it was fastest to type.

Option	Costs	Use when
Hold and observe	Nothing, except time under the leak	Inbound, bounded, and the session carries more than the leak costs
Inbound or outbound `prefix-list`	One commit; the session stays up	The correct prefix set is known
`maximum-prefix`	Bounds the count; may drop the session when hit	The correct prefix set is not known but the correct size is
`shutdown` on the neighbour	The whole session and everything it carried	Nothing else is fast enough, or you are the leaker and the export set is unknown

Step 4a: inbound containment

The corrective filter is the same object the session should have carried from the start. Commit it with the timer running.

Service impact possiblebound what this session may give you

configure
set policy prefix-list FROM-CUST-A rule 10 action 'permit'
set policy prefix-list FROM-CUST-A rule 10 prefix '203.0.113.0/24'
set policy prefix-list FROM-CUST-A rule 10 le '32'
set policy prefix-list FROM-CUST-A rule 10 description 'CUST-A cone, incident 4471'

set protocols bgp neighbor 203.0.113.2 address-family ipv4-unicast prefix-list 'FROM-CUST-A' in
set protocols bgp neighbor 203.0.113.2 address-family ipv4-unicast maximum-prefix '500'

compare
commit-confirm 5
exit

The prefix-list ends in an implicit deny, which is the property that makes it a filter rather than a preference. maximum-prefix goes in the same commit: the filter bounds which prefixes, the limit bounds how many, and a peer capable of the first mistake is capable of a second one you did not anticipate.

Then refresh the inbound stream rather than resetting the session:

Service impact possiblere-run policy over what the peer already sent

clear ip bgp 203.0.113.2 soft in

A soft refresh re-applies policy to the stored routes. A hard clear ip bgp 203.0.113.2 tears the session down and re-runs the full exchange, which re-converges every prefix the session carried - including all the healthy ones, during an incident, on a control plane that is already busy.

Step 4b: outbound containment - you are the leaker

Reverse the order of everything. Stop the announcement first; understand it afterwards.

Service impact possibleexport only what is yours

configure
set policy prefix-list TO-UPSTREAM rule 10 action 'permit'
set policy prefix-list TO-UPSTREAM rule 10 prefix '192.0.2.0/24'
set policy prefix-list TO-UPSTREAM rule 10 description 'our allocation, exact match only'

set protocols bgp neighbor 198.51.100.9 address-family ipv4-unicast prefix-list 'TO-UPSTREAM' out

compare
commit-confirm 5
exit

If the correct export set is not established with confidence - which is common, because the leak usually means the export policy was never written down - the honest containment is to take the session down and keep the configuration:

Service impact possiblestop advertising without destroying the evidence

configure
set protocols bgp neighbor 198.51.100.9 shutdown
compare
commit-confirm 5
exit

Step 5: verify from outside, not from your own router

Read-only / Safethe local half of the proof

show bgp summary
show ip bgp neighbors 198.51.100.9 advertised-routes
show ip bgp 198.51.100.0/24
show ip route 198.51.100.0/24

The FIB check is the one that gets skipped. A prefix withdrawn from the BGP table can still have a route installed, and a route in the FIB is carrying traffic no matter what the BGP table says.

The local half is not the proof. advertised-routes shows what this router is sending; it says nothing about what the neighbouring AS accepted, what it propagated, or what its own upstreams did with it. For an outbound leak, the incident closes when the neighbour confirms your prefixes are gone from their view - a phone call or a ticket to their NOC, and a public looking glass as corroboration.

Step 6: fix the mechanism that permitted it

Containment holds until somebody reverts it. The mechanism fix is what makes the second occurrence impossible, and it is nearly always one of four things:

No inbound filter on the session. The peer’s mistake became yours because nothing said no. Every session gets a prefix-list, sized from the contract.
No maximum-prefix. The filter bounds identity; the limit bounds volume. A session without one accepts as many prefixes as the peer cares to send.
Redistribution without a route-map. redistribute with no filter injects everything in the source protocol into BGP, which is how internal and customer prefixes reach an upstream without anyone typing them.
An outbound filter wider than your allocation. If advertised-routes can contain a prefix you do not own, the outbound policy is a preference and not a filter.

Commit the fix as a normal change, with review, not as a continuation of the incident. Then re-read show bgp summary a day later: a filter that silently dropped something legitimate shows up as a count that never recovered.

Common patterns

Symptom	Likely cause	Resolution
Session in `Idle (shutdown)` right after a count spike	`maximum-prefix` fired and did its job	Filter first, then clear the session. Raising the limit to bring it back re-admits the leak
`received-routes` returns nothing at all	`soft-reconfiguration inbound` is not configured	Use `routes` for the post-policy view; record the gap; add it later at planned-change pace
Count is normal but traffic moved	A more-specific won on longest-prefix match	Look for the more-specific in `show ip bgp`, not at the totals
Your ASN appears in a path you never intended	Prefixes from one peer are being re-advertised to another	You are providing transit. Fix the outbound filter on both sessions
Prefix marked RPKI-invalid, owner says it is theirs	Stale ROA or a max-length shorter than the announcement	Prefix owner fixes the ROA. Do not filter their traffic away while they do
Leak began exactly at a commit timestamp	Local change, not a peer problem	`show configuration commit`, then roll back. Filtering treats the symptom
Session cleared and the leak came straight back	Peer has not fixed their side	The filter is what holds. A reset only re-runs the same exchange
Withdrawal committed, neighbour still sees it	Propagation and caching upstream of them	Confirm with the neighbour NOC; your `advertised-routes` is not the proof

Escalation

Escalate when:

The direction is established - immediately, in both directions, to the peer or provider NOC.
The leak is outbound. That is a security incident with an external audience.
Containment requires shutting down a transit or upstream session.
show configuration commit puts a local change at the start of the leak.
The evidence is an invalid or IRR-mismatched origin - that conversation is with the prefix owner, not the peer.
Customer or third-party prefixes are involved. Disclosure is not an on-call decision.

Why this matters

The instinct under a leak is to make the number go away, and the fastest way to make a number go away is to shut the session down. That instinct is calibrated for the inbound case - where the cost lands on you, the evidence is yours, and the leak stops the moment you stop listening.

It is exactly wrong for the outbound case, which is the one that matters more. There the number on your router looks fine. Your sessions are up, your counts are unremarkable, and the only symptom is traffic arriving that you did not expect, from networks that never chose you. You cannot see the incident from the router causing it, so the direction check in Step 0 is not an administrative formality: it is the only step that tells you whether you are the victim or the cause.

The rest of the ordering follows from that. Evidence before containment, because every containment action erases the reason for it. Filter before shutdown, because the session usually carries more than the leak. And external confirmation before closing, because the one thing your own router cannot report is what the rest of the Internet is still doing with an announcement you have already withdrawn.