BGP route oscillation — next-hop resolution, MED, and reflectors

A BGP route that changes best path on a repeating cycle is one of three things: the winning path keeps losing its next hop, MED keeps reordering two paths from the same neighbouring AS, or a route reflector is feeding the network a view of the table that is incomplete in a way that has no stable answer. Each has a different cadence, different evidence, and a different fix.

This lesson walks the three, the operational evidence that separates them, and the configuration that prevents them on VyOS 1.5 / FRR 10.x. The goal is to look at a route that flips and say “this is MED, and it will stop when we group by AS” or “this is the reflector, and the cluster-id is wrong” within minutes rather than within a maintenance window.

The three causes and their evidence

flowchart TD
  OSC[Best path changes<br/>path A → path B → path A → ...] --> Q1{Does the winning path's<br/>next hop resolve?}
  Q1 -->|"not always"| NH[Next-hop resolution<br/>tracking an unstable IGP]
  Q1 -->|"always"| Q2{Are the competing paths<br/>from the same neighbouring AS?}
  Q2 -->|"yes"| Q3{Is MED the deciding step?}
  Q2 -->|"no"| RR[Reflector: partial visibility<br/>check cluster-list and originator]
  Q3 -->|"yes"| MED[MED-induced oscillation]
  Q3 -->|"no"| RR2[Something upstream is flapping<br/>read the peer, not the algorithm]

  NH --> FIX1[Fix: nexthop-self on iBGP,<br/>or stabilise the IGP]
  MED --> FIX2[Fix: parameters deterministic-med]
  RR --> FIX3[Fix: explicit cluster-id per cluster,<br/>or more visibility via add-path]
  RR2 --> FIX4[Fix: the peer or the link]

Three questions, in this order. Does the next hop resolve, every time? Are the competing paths from the same neighbouring AS, which is the only case where MED is compared at all? And is MED actually the step that decides — which the router will tell you rather than leaving you to guess, because FRR prints the reason a path won in parentheses beside the best marker in show ip bgp <prefix>, and show ip bgp <prefix> bestpath narrows the output to that path alone.

Cause 1 — the next hop that does not resolve

An iBGP-learned path carries the NEXT_HOP the border router received from its eBGP peer: an address on the external peering subnet, which is not an address of any router in your AS. Before that path can be considered, FRR has to resolve that address to something in the RIB. If it cannot, the path is not valid — no * flag, never best, never installed, never advertised.

This is not an oscillation on its own; it is a route that is simply missing. It becomes an oscillation when the resolution itself is unstable: an IGP adjacency over a marginal link, a summary route appearing and withdrawing, a static route with BFD flapping. Then BGP path validity follows the IGP, and the best path changes on the IGP’s cadence rather than on any BGP timer. That cadence is the clue — sub-second to a few seconds, and correlated with IGP log lines rather than with anything a peer did.

Diagnosis

show ip bgp 203.0.113.0/24
show ip route 198.51.100.1
show ip ospf neighbor
show configuration commands | match nexthop-self

Read the first output for the next-hop line rather than for the prefix. A path whose next hop cannot be resolved is printed with the next hop marked inaccessible, and the path carries no *. Then ask the RIB directly about that address: show ip route 198.51.100.1 either returns a route or does not, and there is no third answer.

Note what the flags do not mean. In FRR’s status codes, h is history — a dampened route whose penalty has not yet decayed — and d is currently damped. Neither has anything to do with next-hop resolution. An unresolvable path is distinguished by the absence of *, not by the presence of a letter.

Fix

The usual fix is nexthop-self on the border router’s iBGP sessions. The border router rewrites NEXT_HOP to its own address before reflecting the path inward, and its own address is a loopback the IGP has always carried. The external peering subnet then does not need to be in the IGP at all, which is also the answer to “should I run OSPF on my transit link” — you should not.

set protocols bgp neighbor 10.255.0.2 address-family ipv4-unicast nexthop-self
set protocols bgp neighbor 10.255.0.3 address-family ipv4-unicast nexthop-self

If the peering subnet must be reachable internally, carry it as a passive interface in the IGP rather than forming an adjacency across it. That gives the interior routers a route to the next hop without inviting the external peer into your IGP.

And if the underlying instability is the IGP, neither of the above is a fix. A flapping adjacency produces a flapping BGP table by design, and the work is at layer 1 or 2. Turning off next-hop tracking to hide it converts a visible flap into a black hole.

Cause 2 — MED oscillation

Two paths from the same neighbouring AS oscillate between two MED values. The local router picks one based on step 6 of the best-path algorithm (lowest MED); when the upstream changes the MED, the local router picks the other path. The cycle continues as the upstream flaps between two MED values.

The canonical scenario: an upstream provider has two links to the local AS, one primary (MED 50) and one backup (MED 100). The provider’s automation changes the MED based on link health — primary link degrades, MED changes to 100; primary link recovers, MED changes back to 50. The local router’s best-path decision oscillates with the MED changes.

The cadence: matches the upstream’s MED flap rate — typically seconds to minutes. The evidence: show ip bgp 203.0.113.0/24 shows two paths from the same AS with metric 50 and metric 100, and > alternating between them.

Two different things travel under the name “MED oscillation” and it is worth separating them now. In the scenario above something outside is changing and your router is tracking it; the cure is to stop letting MED decide, either by making the comparison order-independent or by deciding earlier. The second kind is nastier: nothing outside changes and the router still oscillates, because MED is only comparable within a neighbouring AS and the algorithm therefore has no total order to settle into. That one is RFC 3345, it needs partial visibility to appear, and it is dealt with in Cause 3.

Diagnosis

show ip bgp 203.0.113.0/24
show ip bgp 203.0.113.0/24 bestpath
show ip bgp regexp _64513$
show configuration commands | match 'bgp parameters'

The first shows both paths with their metric values — MED is what FRR prints in the Metric column — and which one currently carries >. The second narrows to the winner and its reason. The third confirms the two paths really do come from the same neighbouring AS, because if they do not then MED was never compared and you are diagnosing the wrong cause. The fourth prints the parameters in effect, which is where you discover whether always-compare-med is quietly widening the comparison.

For the time dimension, watch rather than sample:

sudo journalctl -u frr --since '1 hour ago' | grep 203.0.113.0

Correlating the best-path changes with the upstream’s MED changes is what turns “it flaps” into “it flaps when they reconverge”, and that sentence is what you send to the upstream.

Fix

The two production fixes:

Option 1 — enable deterministic-med. The knob makes the router group the candidate paths by neighbouring AS before it compares MED, and pick the best of each group first. The comparison then no longer depends on the order the UPDATEs happened to arrive in, which is what made the outcome unstable. The upstream still flaps; your answer to it stops changing.

set protocols bgp parameters deterministic-med

Note the shape: on VyOS 1.4 and 1.5 the knob lives under set protocols bgp parameters, with no ASN in the path. There is no bestpath deterministic-med node — bestpath under parameters carries the as-path and med sub-options (bestpath med confed, bestpath med missing-as-worst), and deterministic-med is its own leaf beside them.

Option 2 — decide the path before MED is ever reached. MED is step 6. If your policy already has an opinion — this provider is primary, that one is backup — express it at step 2 with LOCAL_PREF on the inbound route-map, and the algorithm never gets far enough for the upstream’s MED to matter:

set policy route-map FROM-PROVIDER-A rule 10 action permit
set policy route-map FROM-PROVIDER-A rule 10 set local-preference 200
set protocols bgp neighbor 10.0.0.1 address-family ipv4-unicast route-map import FROM-PROVIDER-A

This is the stronger fix when it fits, because it removes the dependency on the upstream’s behaviour rather than making your response to it reproducible. Option 1 is the fix when you genuinely want to honour MED and only want a stable answer.

A third knob exists and is usually the wrong one: set protocols bgp parameters always-compare-med makes MED comparable across different neighbouring ASes. It does not stabilise anything — it widens the set of paths that MED can reorder, and in a network with more than one upstream it can create oscillation where there was none. Turn it on only when every upstream’s MED is set by the same policy, which in practice means never outside a single administrative domain.

Cause 3 — the reflector that shows the network only half the table

It is worth being precise about what reflectors do and do not do, because the folklore version — “the route bounces between the reflector and the client forever” — is not the failure. RFC 4456 gave reflectors two attributes specifically to stop that: a reflector stamps ORIGINATOR_ID with the router-id of the client that originated the path, and prepends its CLUSTER_ID to the CLUSTER_LIST. A speaker that sees its own router-id in ORIGINATOR_ID, or its own cluster-id in CLUSTER_LIST, discards the path. Reflection loops are closed by the protocol.

What reflectors genuinely do is reduce what each router sees. A client learns only the paths its reflector selected as best, not every path the AS received. That is the whole point — it is why reflectors scale — and it is also the ingredient that turns MED’s partial ordering into a persistent oscillation, which is what RFC 3345 is about. Different routers, deciding on different subsets, can disagree in a way that has no stable resolution.

The configuration error that produces it deliberately is two reflectors sharing a cluster-id but not sharing a client set. Each discards what the other reflected, because it recognises its own cluster-id, so paths that should have been visible to both halves of the network are visible to neither. The topology looks redundant on the diagram and is not.

The cadence: this one does not have a fixed period. It is either continuous, at whatever rate the routers can re-advertise, or it appears only when a particular path exists. That absence of a clean period is itself the signal — a flap with no rhythm is rarely a link.

Evidence: read the ORIGINATOR_ID and CLUSTER_LIST that show ip bgp <prefix> prints under each iBGP path, and compare the cluster-ids and client lists across your reflectors with show configuration commands | match cluster-id and ... | match route-reflector-client.

Fix

Option 1 — one cluster-id per cluster, and every reflector in a cluster serves every client in it. The cluster-id names a cluster, not a router: reflectors that back each other up for the same clients share it, and reflectors serving different client sets must not. Two reflectors with a shared cluster-id and different clients is the misconfiguration; two reflectors with distinct cluster-ids and the same clients is merely redundant, which is what you usually want.

set protocols bgp neighbor 192.0.2.2 address-family ipv4-unicast route-reflector-client
set protocols bgp parameters cluster-id 10.255.0.1

Two details of the 1.5 tree matter here. Reflector-client status is per address family — address-family ipv4-unicast route-reflector-client — because a peer can be a client for IPv4 and not for IPv6, and a configuration that sets it at the peer level is pre-1.4 and will not commit. And cluster-id is a parameters leaf that applies to the whole speaker; there is no per-client cluster-id.

Option 2 — reduce how much the reflection hides. The structural fix for MED oscillation under reflection is more visibility: fewer levels of reflection, reflectors placed so that their best path is also the network’s best path, and — where the platform offers it — advertising more than one path per prefix.

Here VyOS is worth being honest about: FRR implements BGP additional-paths, and VyOS 1.5 does not expose it in the configuration tree. There is no addpath node to set. If your design needs multiple paths advertised per prefix, that is an argument about topology on VyOS, not a knob: flatten the reflection hierarchy, or accept a full mesh in the core where the router count allows it.

Option 3 — take MED out of the decision, as in Cause 2. An oscillation that only exists because MED is partially ordered stops existing when LOCAL_PREF decides first.

Diagnosis for a reflector-related oscillation

show ip bgp 203.0.113.0/24
show ip bgp 203.0.113.0/24 bestpath
show ip bgp neighbors 10.255.0.2
show configuration commands | match cluster-id
show configuration commands | match route-reflector-client

The first prints each path with its ORIGINATOR_ID and CLUSTER_LIST — that is where reflection becomes visible, and it is the only place it does. The second names the deciding step, which tells you whether you are looking at a MED problem wearing a reflector costume. The third confirms the session and its policy. The last two are run on every reflector and compared: the fault is almost always a difference between two routers rather than something wrong on one.

There is no show ip bgp clusterids command in FRR. The cluster-id is read from the configuration and from the CLUSTER_LIST on the paths themselves.

How the result is validated

show ip bgp 203.0.113.0/24
show ip bgp 203.0.113.0/24 bestpath
show ip route 198.51.100.1
show ip ospf neighbor
show configuration commands | match 'bgp parameters'

The first shows the candidate paths, their next hops, their MEDs and their CLUSTER_LIST. The second names the deciding step, which is the single most useful line in the whole investigation. The third asks the RIB whether the next hop resolves. The fourth says whether the IGP that answers that question is itself stable. The fifth prints the parameters actually in effect, which is how you find the always-compare-med somebody enabled in 2021.

Run in that order, they separate the three causes before you have changed anything.

How it fails

The production failure modes the engineer must recognise:

• A path that never carries *. The next hop does not resolve. Fix: nexthop-self on the iBGP sessions, or carry the peering subnet in the IGP. Not a knob — a dependency.
• Best path tracking an IGP adjacency. The resolution is working and unstable. Fix the link; BGP is reporting accurately.
• Two paths from one neighbouring AS trading places. MED is deciding and the order of arrival is deciding MED. Fix: set protocols bgp parameters deterministic-med, or decide earlier with LOCAL_PREF.
• An oscillation with no external trigger, in a reflected topology. RFC 3345 territory: partial visibility plus MED’s partial ordering. Fix: take MED out of the decision, or flatten the reflection.
• Two reflectors sharing a cluster-id with different client sets. Each discards the other’s reflected paths. Fix: distinct cluster-ids, or identical client sets.
• always-compare-med enabled across multiple upstreams. MEDs set by different policies are now being compared as though they meant the same thing. Fix: remove it unless every upstream is under one administration.

Rollback

Oscillation fixes are configuration changes. The standard rollback paths:

• rollback N to revert a configuration change — remembering that on VyOS this reboots the router, so during an incident the targeted delete below is usually the better move.
• reset bgp 10.0.0.1 soft in to re-evaluate what a peer has already sent, after a policy change. VyOS 1.4 renamed the clear family to reset; clear ip bgp ... is not a command on 1.5.
• reset bgp 10.0.0.1 soft out to re-advertise after an outbound policy change.

The targeted reverts:

delete protocols bgp parameters deterministic-med
delete protocols bgp parameters cluster-id
delete protocols bgp neighbor 10.255.0.2 address-family ipv4-unicast nexthop-self

A soft reset is not free — it re-runs policy over the peer’s whole table — but it does not tear down the session, and on a peer carrying a full table that difference is the difference between a blip and an outage. reset bgp without soft does tear it down. Type the word.

Production discipline

Cross-course references

The Linux course’s XIX-Linux-NetFoundations covers the kernel FIB. The OPNsense course’s XXX-OPNsense-DynamicRouting covers the equivalent FRR BGP behaviour on the firewall side. The BGP lessons vyos-xxvi-04-med, vyos-xxvii-01-best-path-algorithm, and vyos-xxx-02-rr-config cover the underlying mechanisms. The lesson vyos-xxxi-03-flapping-session covers the related flapping-session causes.

Quiz