Runbook: Investigate Route Instability

“Route instability” is a description of a symptom by someone who could not see which layer produced it. Four different faults arrive under that phrase, and they share almost no diagnostic steps: a BGP session that keeps resetting, a single prefix that keeps being withdrawn and re-advertised, a prefix that stays present but keeps changing which path it uses, and a control plane so busy processing updates that everything converges late.

Two measurements separate them, and both take about a minute. Scope — one prefix, one peer’s block, or the whole table. Cadence — the interval between transitions. Cadence in particular is the single most useful number in this investigation, because each layer produces a characteristic one: BFD tears a session down in hundreds of milliseconds, a bad optic flaps in seconds, a hold-timer expiry takes the configured hold time, and automation flaps on whatever schedule somebody wrote into cron.

The third question is the one operators skip, and it is the one that decides whether any of this is your problem: is this router the source of the instability, or a spectator to it? A router faithfully re-advertising a flap that started three hops away is working exactly as designed. Everything you can configure locally in that situation hides the flap from your downstream rather than fixing it — which is sometimes the right call, and is never the right call before you know that is what you are doing.

When to use this runbook

• A prefix keeps appearing and disappearing from show ip route.
• A BGP session’s Up/Down timer in show ip bgp summary keeps resetting.
• Traffic to a destination works intermittently with no pattern the application team can describe.
• Convergence after a topology change is taking far longer than this router’s baseline, or the table size is visibly moving.

When not to use it

• A prefix that is simply absent and stays absent. That is a missing-route problem: received, not received, or not installed. It has a different walk and a different runbook.
• A session that is down and stays down. A peer stuck in Active or Connect never reached Established and never flapped. Fix the session first; instability is a property of something that works intermittently.
• A policy change you are about to make. If you are changing a route-map or a prefix-list, use the route-policy runbook, which is built around the fact that the blast radius of a policy edit is every prefix the policy touches.
• A route leak. Prefixes appearing that this AS should never carry is not instability, it is a leak, and the urgency and the escalation path are different.

Blast radius

The investigation itself is read-only up to the first clear. After that it is not, and the two clear forms are very different animals.

clear ip bgp 203.0.113.2 soft in asks the peer to re-send what it already sent, or re-evaluates the stored copy, and leaves the TCP session and every installed route alone. It is cheap.

clear ip bgp 203.0.113.2 without soft resets the session. Every prefix learned from that peer is withdrawn, traffic re-routes to whatever is next-best or to nothing, and then the peer re-sends its whole table — which, on a transit peer, is a route burst that will itself push zebra and bgpd hard for as long as it takes to install. On a router that is already struggling with churn, a hard reset makes the immediate symptom worse before it makes anything better.

Dampening has the largest radius of all, and the longest tail: a suppressed prefix stays suppressed until its penalty decays, and during that time it is unreachable for every customer behind this router, including the ones who never noticed the original flap.

Inputs

• The affected prefix, or the peer whose block is affected.
• The time the instability started, and what was committed anywhere in the estate in the hour before it.
• This router’s normal route count, from the performance baseline.
• Whether dampening is configured anywhere in this estate.
• The peer’s owner and how to reach them.
• Whether the affected path is also your management path.

Step 1: Scope and cadence, before any theory

show ip bgp summary
show ip route summary
show ip bgp 198.51.100.0/24
show ip bgp dampening dampened-paths

Read show ip bgp summary twice, thirty seconds apart. The Up/Down column is an uptime for a session in Established; if it is smaller the second time, the session reset in between and you have your cadence. If it keeps climbing, the session is not the subject and you can stop looking at it.

show ip route summary answers the scope question in one number. Compare it against what this router normally carries. A count that is oscillating by thousands is a table-level problem — churn — and the per-prefix walk in Step 4 will waste your time. A count that is steady while one prefix misbehaves is a per-prefix problem.

Run show ip bgp dampening dampened-paths early, before you conclude anything about a missing prefix. A dampened prefix is being suppressed deliberately by a mechanism this estate configured, and the session carrying it is perfectly healthy. Chasing a session problem that does not exist is the most common way this investigation goes wrong.

Step 2: Session or prefix

These are two incidents wearing one description, and mixing them up costs the most time.

show ip bgp neighbors 203.0.113.2
show ip bgp 198.51.100.0/24
show ip bgp flap-statistics

A session that is flapping takes every prefix from that peer with it on each cycle, so the symptom is wide: a whole block appearing and disappearing together. A prefix that is flapping while the session holds is narrow: one prefix moving, everything else from the same peer untouched.

show ip bgp flap-statistics gives you the per-prefix flap count, which is the fastest way to tell “this one prefix has flapped forty times” from “everything from this peer has flapped four times”. Those lead to opposite places: the first is upstream of the peer, the second is the peer relationship itself.

Step 3: A flapping session — read the cadence to the layer

show interfaces ethernet eth1
show bfd peers
show ip bgp neighbors 203.0.113.2
journalctl -u frr --since '30 min ago' | grep 203.0.113.2

The FRR log is the piece most operators skip and it usually contains the answer outright: the state-machine transitions are logged with a reason, and “hold timer expired”, “peer closed the session” and a user-initiated reset are three different sentences.

• Interface counters moving. Carrier transitions, CRC errors, or input errors climbing on the interface the session rides means the session is a passenger. Nothing in BGP is wrong. This is a cable, an optic, or the link partner, and it escalates to whoever can touch the hardware.
• BFD down events preceding every BGP drop. BFD is doing its job too well for the link it is on. A short interval means a packet rate in both directions that a congested link or a saturated control-plane CPU cannot sustain, so BFD declares a failure that the data path never had.
• A clean link, a clean BFD, and a hold-timer expiry. Keepalives are not arriving in time. Look at control-plane load and at the path itself before touching the timers — a hold time raised to make the symptom stop is a router that now takes correspondingly longer to notice a real failure.
• A user reset in the log. Find the automation. This ends with someone deleting a cron entry, not with a routing change.

Step 4: A stable session with an unstable prefix

Read the flags first. They tell you which stage of the pipeline is dropping or moving the route, and each has a different fix.

vyos@vyos:~$ show ip bgp 198.51.100.0/24

BGP routing table entry for 198.51.100.0/24
Paths: (2 available, best #1, table default)
65001 65010
  203.0.113.2 from 203.0.113.2 (192.0.2.11)
    Origin IGP, metric 50, valid, external, best (MED)
65001 65010
  203.0.113.6 from 203.0.113.6 (192.0.2.12)
    Origin IGP, metric 100, valid, external

• d — dampened. The prefix is suppressed. The session is fine.
• h — history. The route was received but its next-hop does not resolve, so it is held rather than selected. The instability is really in whatever carries the next-hop, usually the IGP.
• *> in BGP with nothing in show ip route — the route was selected and the kernel did not take it. Check the next-hop’s own route and any table-map before assuming BGP is at fault.
• Two paths alternating > between them — this is oscillation rather than flap, and Step 4a is where it goes.

Step 4a: Oscillation between two paths

show ip bgp 198.51.100.0/24 bestpath
show ip route 203.0.113.2
show ip ospf neighbor

The best-path algorithm compares candidates in a fixed order and stops at the first step that separates them. bestpath tells you which step decided, and the deciding step names the cause:

• MED. Two paths from the same neighbouring AS whose metric keeps changing. The upstream is changing it — often their own automation reacting to link health. The local table follows faithfully.
• Cluster-id or originator-id changing each cycle. A route reflector is reflecting a route back toward its origin. This is an iBGP design property, not a fault on this box.
• IGP cost to the next-hop. The BGP path is stable and the IGP underneath it is not. Go and look at the IGP; the BGP symptom is downstream of it.

Step 5: Source or spectator

This is the decision point, and it is worth being explicit about it in the ticket.

show ip bgp neighbors 203.0.113.2 received-routes
show ip bgp 198.51.100.0/24
show ip route summary

If the instability is visible in what the peer sends you, this router is a spectator. It is receiving a flapping advertisement and re-advertising it correctly. There is no local misconfiguration to find, and the useful next action is the handover — with the cadence, the prefix, the flap count and the timestamps attached, because that is the evidence the peer’s own operators will otherwise have to recreate.

If the instability appears here and not in the received stream, this router is the source: the next-hop resolution, the IGP, the local policy, or the physical link on this box. That is the only case where a local change is the fix rather than a mask.

Step 6: Choose one intervention, and name its cost

The intervention ladder, cheapest first:

1. Nothing, plus a handover. If the instability is upstream, the table is stable in size, and traffic is completing, this is the correct action and it should be stated as a decision rather than as inaction.
2. Fix the physical layer. Cable, optic, link partner. This is the only one that removes the cause rather than the symptom, and it is the outcome for a large fraction of session flaps.
3. Loosen BFD to match the link. If BFD is declaring failures the data path is not having, a longer interval or a higher multiplier restores the intent — fast detection — without the false positives. You are trading detection time for stability, deliberately.
4. Soft-clear the peer once the cause is fixed, to re-evaluate inbound policy and rebuild a clean view without a session reset.
5. Bound what the peer can do to you. An inbound prefix-list and a maximum-prefix on the neighbour limit the blast radius of the next event. This is a change to make after the incident, in a change window, not during it.
6. Damping. Last, rarely, and with an owner.

Step 7: Soak, then close

show ip bgp summary
show ip bgp 198.51.100.0/24
show ip route summary
show ip bgp flap-statistics

Watch for at least three times the measured flap interval, or ten minutes, whichever is longer. A four-minute flap cycle looks completely resolved for three and a half minutes, and an incident closed inside that window reopens with less patience on the bridge.

The Up/Down timer climbing past the whole soak window is the single strongest piece of evidence, because it is a statement about the entire period rather than about the instant you ran the command.

Then check the control plane came back with it:

top -bn1 | head -20
mpstat -P ALL 1 5

zebra is single-threaded, so a router that spent an hour processing churn can be minutes behind on route installation even after the churn stops. A table that is still settling is not a fixed router yet.

Rollback

Removing a dampening configuration does not release a prefix that is already suppressed. The penalty lives in the BGP process, not in the configuration, and it keeps decaying on its own schedule until you clear it explicitly.

Common patterns

Escalation

Escalate when:

• The evidence shows the instability arrives from a peer. This should happen early — it is the most common correct outcome, and local investigation past that point is time spent on someone else’s fault.
• Interface counters implicate the physical layer. No routing change fixes an optic.
• The cause is MED oscillation or reflector amplification. Both are properties of a topology design, and the fix belongs to whoever owns that design.
• Anyone proposes dampening on a router where it is not already policy.
• The instability comes with prefixes this AS should not be carrying. That is a leak, and it stops being this runbook immediately.