Skip to main content
RunBook Academy

← All break/fix scenarios in VyOS

advancedvyos-bgp-prefix~30 min

BGP Route Received but Not Selected

Reported symptoms

  • ●Five destinations became unreachable between 03:20 and 03:45, one at a time rather than together, with no common customer, country or prefix length
  • ●Three of the five came back without anyone doing anything, roughly forty minutes after they failed
  • ●One came back at 04:05 and failed again at 06:12; the other two are still down at 09:00
  • ●The BGP session to ISP-A has been Established for forty-one days and did not flap at any point
  • ●ISP-A insists it is advertising all five prefixes continuously and has been throughout
  • ●There has been no configuration change on this router in six weeks, and the prefix count in monitoring is a couple of hundred below yesterday - inside the alerting band

Evidence

  • · `show ip bgp 198.51.100.0/24` shows the path present, carrying the `d` flag, and with no `>` - the prefix is in the table and is not selected
  • · `show ip bgp neighbors 192.0.2.1 received-routes | match 198.51.100` shows the prefix arriving from the peer, so ISP-A is telling the truth
  • · `show ip bgp dampening` lists seven suppressed prefixes, of which the five reported are a subset
  • · `show ip bgp dampening flap-statistics` shows flap counts between four and nineteen per prefix, with different penalties and different reuse times
  • · `show configuration commands | match dampening` shows a route-map setting dampening parameters, bound to the ISP-A session, committed fourteen months ago
  • · `show system commit` shows nothing on this router for six weeks; the damping commit is comment "bgp: damp ISP-A after flap storm"
  • · ISP-A confirms a link inside their network was unstable between 03:10 and 03:35 and has since been taken out of service
Diagnosis and resolutionclick to reveal

Root cause

Route-flap damping was configured on the ISP-A session fourteen months ago, during a previous incident, and never revisited. Between 03:10 and 03:35 a link inside ISP-A flapped repeatedly. From this router the effect was a series of withdrawals and re-announcements of whichever prefixes traversed that link. Damping adds a penalty to a prefix on each such event and decays that penalty exponentially; once the penalty crosses the suppress threshold the prefix is held out of the best-path decision until the penalty decays below the reuse threshold. That single mechanism produces every symptom in the report. The staggered onset is prefixes crossing the suppress threshold at different moments as their individual flap counts accumulated. The self-healing after roughly forty minutes is exponential decay from a modest peak penalty. The prefix that recovered and failed again took one further flap while its penalty was still elevated, which pushed it back over the threshold from a much higher starting point. The set of affected prefixes looks arbitrary because it is determined by which prefixes crossed the flapping link, which has nothing to do with the customers who own them. And every session-level check reported health because the session genuinely was healthy: the routes flapped, not the session, and damping operates entirely below the level at which anyone was looking.

Remediation

Decide first whether ISP-A's link is stable, because that decision changes which action is correct rather than merely which is faster. If the instability is over - which here it is, the link having been taken out of service - clear the penalties and restore the prefixes. `clear ip bgp dampening 198.51.100.0/24` is surgical and restores exactly what you name, which means it silently leaves the prefixes nobody reported still suppressed; clearing damping for the whole session restores all of them and is usually what you actually want once the cause is known to be over. Use `show ip bgp dampening` to get the full suppressed list before you clear anything, so the ticket records the true blast radius rather than the five prefixes that generated calls. If the upstream link is still unstable, holding is a legitimate first-class choice and not an absence of action: clearing the penalties re-admits the churn to your RIB and to everyone downstream of you, which is the exact harm the damping was bought to prevent. A hold needs an owner, a review time and a customer message that says the prefix is suppressed and why, because from the customer's side suppression is indistinguishable from an outage. Removing damping from the session altogether is a design decision, not an incident action; it belongs to whoever owns the fourteen month old choice, in daylight.

Verification

`show ip bgp 198.51.100.0/24` must show the path without the `d` flag and carrying the best-path marker; a prefix that is merely present is not a prefix that is selected, and this scenario exists because those two states look similar in a hurry. `show ip route 198.51.100.0/24` must then show the route installed, because selection and installation are different stages and only the second one forwards packets. `show ip bgp dampening` must be empty, or must contain only entries you have deliberately decided to leave suppressed, with that decision written down. Prove reachability from a client rather than from the router. Re-read `show ip bgp dampening flap-statistics` after an hour: if prefixes are re-entering damping, the upstream is not stable and your remediation has restored a symptom rather than fixed a cause. Confirm the count of suppressed prefixes is now a monitored series with an alert attached, and prove the alert fires by checking it against the historical event rather than by assuming it would have. Finally, confirm with ISP-A in writing that the unstable link is out of service, so the timeline in the ticket rests on their statement rather than on your inference.

Prevention

A suppressed prefix is an outage that no dashboard shows, so the first control is to count them: the number of entries in `show ip bgp dampening` belongs in monitoring as a series with an alert on any non-zero value, which turns a six-hour customer-driven diagnosis into a notification with the cause already named. The second is to treat damping as a scoped, dated decision rather than a permanent setting - apply it to the specific peer whose churn justified it, never globally, and attach a review date, so that a control added during one incident cannot become the invisible cause of the next one fourteen months later. The third is to know the parameters you are running and what they imply in wall-clock time: a half-life and a suppress threshold together decide how long a customer stays dark after a flap, and if nobody on the team can state that number, the configuration is not understood well enough to be in production. The fourth is diagnostic discipline that this incident would have rewarded immediately - when a prefix is missing, ask whether the peer sent it before asking anything else, because `received-routes` separates "not received" from "received and rejected locally" in one command and sends the investigation down the correct half of the tree.

Reported symptoms

The overnight log reads like five separate faults.

At 03:22 a customer reports that they cannot reach a supplier in Portugal. At 03:31 a different customer cannot reach a payment gateway in Ireland. At 03:38, 03:41 and 03:45, three more destinations go dark, each reported by a different team, each in a different country, each a different prefix length. Nobody connects them, because there is nothing to connect them by.

By 04:05 three of the five are working again. Nobody fixed them. The tickets are closed as transient. At 06:12 one of the three fails again, which reopens the ticket with the comment “intermittent — suspect upstream”. Two of the original five have never come back.

The network on-call, arriving at 09:00, finds a router in perfect health:

  • The eBGP session to ISP-A has been Established for forty-one days. It did not flap at 03:10, at 03:35, or at any other point.
  • There has been no configuration change on this router in six weeks.
  • The prefix count is a couple of hundred below yesterday’s figure — inside the band the monitoring uses, so nothing alerted.
  • ISP-A, asked directly, insists it has been advertising all five prefixes continuously. They are right.

The estate

rtr-edge-1 is a VyOS 1.5 LTS router in AS 64512 with a single transit session to ISP-A at 192.0.2.1, AS 64500. It takes a partial table plus a default route, re-advertises to the core over iBGP, and has done so without incident for over a year.

Fourteen months ago it had a bad week. ISP-A’s network churned for several days, the local RIB churned with it, and during that incident route-flap damping was configured on the ISP-A session. The commit comment reads bgp: damp ISP-A after flap storm. It worked. The incident closed. Nobody has looked at it since, and no current member of the team was on that call.

Evidence provided

Read-only / Safethe session did not flap - only the routes did
vyos@rtr-edge-1:~$ show ip bgp summary
Neighbor        V    AS   MsgRcvd  MsgSent  Up/Down   State/PfxRcd
192.0.2.1       4  64500     91422     4118  41d06h14m      4310

Illustrative output

Read-only / Safepresent, valid, and not selected
vyos@rtr-edge-1:~$ show ip bgp 198.51.100.0/24
BGP routing table entry for 198.51.100.0/24
Paths: (1 available, no best path)
Path 1:
192.0.2.1 from 192.0.2.1 (10.255.9.1)
Origin IGP, metric 0, localpref 100, weight 0, valid, external
Dampinfo: penalty 2412, flapped 11 times in 00:24:31, reuse in 01:12:00

Illustrative output

Read-only / Safethe peer is sending it - this is the decisive command
vyos@rtr-edge-1:~$ show ip bgp neighbors 192.0.2.1 received-routes | match 198.51.100
*d 198.51.100.0/24   192.0.2.1   0   64500 65010 i

Illustrative output

Read-only / Safedifferent flap counts, different reuse times
vyos@rtr-edge-1:~$ show ip bgp dampening flap-statistics
   Network            From          Flaps Duration Reuse    Path
*d 198.51.100.0/24    192.0.2.1        11 00:24:31 01:12:00 64500 65010 i
*d 198.51.100.128/25  192.0.2.1        19 00:25:02 03:41:00 64500 65010 i
*d 203.0.113.0/24     192.0.2.1         7 00:22:18 00:19:00 64500 65021 i
*d 203.0.113.64/26    192.0.2.1         4 00:18:44 00:04:00 64500 65021 i

Illustrative output

Read-only / Safecommitted fourteen months ago and never reviewed
vyos@rtr-edge-1:~$ show configuration commands | match dampening
set policy route-map RM-ISP-A-DAMP rule 10 action 'permit'
set policy route-map RM-ISP-A-DAMP rule 10 set dampening '15 750 2000 60'

Illustrative output

Work the evidence before reading on

Nothing here is broken. Something here is doing exactly what it was configured to do, to prefixes nobody expected it to do it to.

  1. The prefix is in show ip bgp and has no best-path marker. Read the three-failure-point model: this is not “not received” and not “not installed”. What is the one thing you know for certain about where the route stopped?
  2. received-routes shows the peer sending it. That single fact eliminates half the possible causes. Which half, and what does it leave?
  3. Look at the timings in the flap statistics rather than the flap counts. Two prefixes flapped a similar number of times and have reuse times three hours apart. What does that tell you about how the penalty is calculated?
  4. Three destinations recovered on their own after about forty minutes and one relapsed. Self-healing at a characteristic interval is a strong signal. What kind of mechanism decays?
  5. The affected prefixes belong to unrelated customers in unrelated countries. Stop looking for what the customers have in common and ask what the paths have in common.

Before continuing: the session never went down, so what did?

Root cause

1. Damping penalises routes, and the session was never the subject

Route-flap damping tracks each prefix separately. Every time a prefix is withdrawn and re-announced, its penalty increases; the penalty then decays exponentially with a half-life. When the penalty crosses the suppress threshold, the prefix is held out of the best-path decision — it stays in the table, stays valid, and stops being selected — until the penalty decays below the reuse threshold.

The configured parameters here are the standard set the course describes: half-life 15 minutes, reuse 750, suppress 2000, maximum suppress time 60 minutes. Everything in the incident falls out of those four numbers.

Between 03:10 and 03:35 a link inside ISP-A’s network was unstable. This router saw that as a stream of withdrawals and re-announcements for whichever prefixes crossed that link. The session itself was never affected: BGP kept its TCP connection, kept its keepalives, and kept reporting Established, which is why every session-level check said health and why three hours went into the transport.

2. Every odd behaviour is the same mechanism

  • Staggered onset. Prefixes crossed the suppress threshold at different moments, because each accumulated penalty at its own rate depending on how many times it flapped.
  • Partial self-healing after about forty minutes. Exponential decay from a modest peak. A prefix that just crossed 2000 falls below 750 in a little over two half-lives, which is the interval the tickets recorded and closed as “transient”.
  • The one that came back and failed again. It took one further flap while its penalty was still elevated. Starting from a high base, one event was enough to cross the threshold again — and the second suppression lasts far longer than the first.
  • The arbitrary-looking set. The affected prefixes are the ones whose path crossed the flapping link. There is no property of the customers, the countries or the prefix lengths that predicts it, because none of those things chose the path.

3. The decisive command is the cheapest one

show ip bgp neighbors 192.0.2.1 received-routes is the command that collapses this investigation, and it is worth being explicit about why. It shows what the peer sent, before local policy. If the prefix is there and not in the selected table, the loss happened locally and the upstream is exonerated in one line. If it is absent, the peer is the problem and nothing on this router will fix it.

Resolution

  1. Establish whether the upstream instability is over before choosing an action. This is not a formality: if ISP-A is still flapping, clearing the penalties re-admits the churn to your RIB and to everyone downstream, which is the harm the damping was bought to prevent.
  2. Capture the true blast radius before changing anything. show ip bgp dampening lists every suppressed prefix; the five in the tickets are a subset. Put the full list in the ticket - customers who did not call are still affected.
  3. Record the flap statistics too. show ip bgp dampening flap-statistics is the evidence that the cause was upstream churn rather than anything on this router, and it will be gone once you clear.
  4. If the instability is over, clear the penalties. clear ip bgp dampening 198.51.100.0/24 restores exactly the prefix you name and silently leaves the rest suppressed; clearing damping for the session restores all of them, and once the cause is known to be over that is usually what you want.
  5. If the instability is not over, hold - and hold deliberately. Name an owner, set a review time, and send a customer message that says the prefix is suppressed and why. From the customer side, suppression and an outage are the same experience, and silence is what turns a defensible decision into a complaint.
  6. Verify selection and then installation, in that order: the d flag gone and a best-path marker present in show ip bgp, then a route in show ip route. Selection and installation are different stages.
  7. Leave the damping configuration alone tonight. Removing it is a design decision that belongs to whoever owns the fourteen-month-old choice, taken in daylight with the parameters in front of them.
  8. Re-read the flap statistics an hour later. Prefixes re-entering damping mean the upstream is not stable and you have restored a symptom rather than fixed a cause.

Verification

  1. The prefix is selected, not merely present. show ip bgp 198.51.100.0/24 shows no d flag and a best-path marker. This whole scenario exists because those two states look alike at speed.
  2. The route is installed. show ip route 198.51.100.0/24 shows it in the FIB. A selected path that is not installed forwards nothing, and it is a different failure with a different cause.
  3. The suppressed list is empty, or contains only entries you decided to leave and wrote down. show ip bgp dampening is the check, and "only the ones we knew about" is not a passing result.
  4. Reachability is proven from a client, in both directions, rather than from the router. The reported symptom was reachability.
  5. Nothing is re-entering damping. Re-run show ip bgp dampening flap-statistics after an hour, and again after a business day.
  6. The suppressed-prefix count is now a monitored series with an alert on any non-zero value. Check it against the historical event rather than assuming it would have fired.
  7. The upstream statement is in writing. The timeline in this ticket rests on ISP-A confirming that the unstable link is out of service, not on your inference from the flap counts.

Prevention

  • Count the suppressed prefixes and alert on the count. A dampened prefix is an outage that no dashboard shows. The number of entries in show ip bgp dampening is a one-line series, and any non-zero value deserves a notification with the cause already named. This single control converts a six-hour customer-driven diagnosis into a page.
  • Scope damping to the peer that justified it and give it a review date. The course is explicit that damping belongs on specific peers rather than globally, and only where the benefit of suppressing real flaps outweighs the cost of suppressing legitimate prefixes. A control added during one incident and never revisited is well placed to become the invisible cause of the next.
  • Know your parameters in wall-clock terms. Half-life 15 and suppress 2000 are not abstractions; together they decide how long a customer stays dark after a flap storm. If nobody on the team can state that duration, the configuration is not understood well enough to be in production.
  • Ask “did the peer send it” first. received-routes splits the problem in half in one command and points the next hour at the correct half. It is the cheapest question in BGP troubleshooting and it is routinely asked third.
  • Widen the monitoring band the right way. A two-hundred-prefix drop sat inside the alerting band, which is a reasonable band for noise and a poor one for suppression. Alert on the suppressed count directly rather than trying to infer it from the total.