BGP ECMP — maximum-paths, eBGP/iBGP, multipath-relax, AS_PATH length requirement

Advanced⏱ ~24 minshow bgp ipv4show bgp ipv4 summaryshow ip route bgpconfigurecomparecommitcommit-confirmsavevtyshpingtraceroute

Verified against VyOS 1.5.x LTS (circinus) · VyOS 1.4.x (sagitta) — legacy · FRRouting 10.x (VyOS 1.5) · Linux kernel 6.6 LTS (VyOS 1.5 base) · strongSwan 5.9.x (IPsec) · WireGuard 1.0.x (kernel module + userspace tooling)

BGP ECMP requires the paths to be “equal” by the best-path algorithm. The best-path algorithm has 11 tie-breakers, and ECMP requires the paths to be equal up to and including the IGP cost tie-break. If any earlier tie-break separates the paths (different weight, local-preference, AS-path length, origin, MED, or eBGP/iBGP), only one path is installed.

This lesson is the operator’s reference for BGP ECMP: the maximum-paths configuration, the multipath-relax knob, the per-peer next-hop tracking, and the production failure modes that arise when the best-path algorithm picks one path despite the operator expecting ECMP.

The BGP best-path algorithm in one minute

FRR 10.x runs its route selection as an ordered list, and the multipath decision sits inside that list rather than beside it:

Lowest administrative distance.
Highest weight.
Highest local-preference.
Locally originated (network statement, aggregate, redistribute).
Shortest AS_PATH length.
Lowest origin (IGP < EGP < incomplete).
Lowest MED — compared only between paths from the same neighbouring AS, unless always-compare-med is set.
eBGP preferred over other peer types.
Lowest IGP cost to the next-hop.
Multipath check — the paths that have tied through step 9 become the multipath set.
Already-selected external route (stability).
Lowest router-ID.
Shortest cluster-list.
Peer address, as a last resort.

Read step 10 carefully, because it is the whole lesson. Paths become an ECMP set by surviving the earlier steps together. Anything that separates two paths at step 5, 7 or 9 has already eliminated one of them before multipath is considered. ECMP is not a mode you turn on; it is what happens when nothing broke the tie.

maximum-paths configuration

On VyOS 1.5 maximum-paths lives under the address family, and it always carries a peer-type qualifier:

configure
set protocols bgp system-as 64512
set protocols bgp address-family ipv4-unicast maximum-paths ebgp 2
set protocols bgp address-family ipv4-unicast maximum-paths ibgp 4
commit
save

Two things to be precise about, because the 1.3-era shape of this command encouraged a wrong mental model:

There is no unqualified maximum-paths. The node is maximum-paths <ebgp|ibgp> <1-256>; the qualifier is mandatory. A bare set protocols bgp maximum-paths 4 is not a command on this tree, and it never meant “eBGP and iBGP combined” even where it parsed.
It is per address family. IPv4 and IPv6 unicast each carry their own cap. Setting it for ipv4-unicast and expecting dual-stack ECMP is a common and completely silent omission — the IPv6 side simply keeps installing one path.

The defaults are 1 for both, so ECMP is opt-in. The example above reads as: two eBGP peers, four iBGP paths. Size the numbers to the topology rather than setting them high “just in case” — the cap is also the ceiling on how much churn a single prefix’s next-hop set can produce in the FIB.

multipath-relax — same length, different ASes

This knob is routinely described wrongly, including in a lot of vendor-adjacent blog writing, so it is worth quoting FRR’s own definition:

This command specifies that BGP decision process should consider paths of equal AS_PATH length candidates for multipath computation. Without the knob, the entire AS_PATH must match for multipath computation.

Read that twice. The default is not “equal length”; the default is identical AS_PATH. Two paths of length 2 reading 64513 64515 and 64514 64515 are not multipath candidates by default, even though they tie at every best-path step, because the paths are not the same sequence.

multipath-relax loosens that comparison from “identical” to “same length”:

configure
set protocols bgp parameters bestpath as-path multipath-relax
commit
save

multipath-relax does not equalise different AS_PATH lengths

The thing this knob is most often expected to do is the one thing it does not do. AS_PATH length is step 5 of the selection algorithm — a real tie-break, resolved long before the multipath check at step 10. A path with AS_PATH length 1 beats a path with length 3 and is chosen as best; the longer path is eliminated and never reaches the multipath comparison. multipath-relax changes nothing about that.

If you have two transits and one consistently presents a shorter AS_PATH, the levers that actually equalise them are the ones that act at or before step 5:

AS-path prepend on the shorter side’s inbound, with a route-map on filter-list/route-map import, to make the lengths match.
Accept that they are not equal and use the second transit as a backup, which is frequently the honest answer.

multipath-relax is for the far more common case where the lengths already match and only the AS numbers differ — which is exactly what two independent transits to the same destination usually look like.

flowchart TB
  subgraph EQ["Equal length, different ASes — multipath-relax applies"]
    A1["path A: 64513 64515"]
    A2["path B: 64514 64515"]
    A3["default: not multipath (paths differ)"]
    A4["with multipath-relax: both installed"]
  end
  subgraph NE["Different lengths — multipath-relax does NOT apply"]
    B1["path A: 64513"]
    B2["path B: 64514 64516 64517"]
    B3["step 5 eliminates B before the multipath check"]
    B4["with multipath-relax: still only A"]
  end

The eBGP/iBGP separation

Step 8 prefers eBGP over other peer types, so an eBGP path and an iBGP path for the same prefix are separated before the multipath check and only the eBGP one is installed. Raising maximum-paths ebgp or maximum-paths ibgp does not change this: those are caps applied within a peer type, not a way to merge across types.

The knob that does merge them is a bestpath parameter of its own:

set protocols bgp parameters bestpath peer-type multipath-relax

FRR describes it as making the decision process “consider paths from all peers for multipath computation… paths learned from any of eBGP, iBGP, or confederation neighbors will be multipath if they are otherwise considered equal cost.”

Note the qualifier at the end. It relaxes step 8 only. Every other step still applies, so an iBGP path with a different AS_PATH still needs as-path multipath-relax as well, and an iBGP path whose next-hop has a higher IGP cost is still eliminated at step 9.

MED and the always-compare-med knob

MED (Multi-Exit Discriminator) is step 7 of the selection algorithm, and by default it is compared only between paths from the same neighbouring AS. Two transits in different ASes therefore never have their MEDs compared at all — which means a MED difference between them cannot be the reason your ECMP is not forming, and always-compare-med cannot be the fix.

On VyOS 1.5 the knob sits directly under parameters, not under bestpath med:

configure
set protocols bgp parameters bestpath as-path multipath-relax
set protocols bgp parameters always-compare-med
commit
save

The bestpath med node exists but holds different options — confed and missing-as-worst. always-compare-med is its own leaf one level up. This is the kind of near-miss that autocompletes into the wrong place if you are typing from memory of another vendor’s CLI.

always-compare-med affects best-path, not just ECMP

always-compare-med is a global selection setting. It changes which path is chosen as best across the whole table, not just which paths become multipath candidates — so it will move traffic for prefixes you were not thinking about when you enabled it.

There is a reason the default is what it is. MED is set by your neighbour, and comparing MEDs across different ASes means letting two unrelated networks, who have never coordinated, jointly determine your routing by choosing numbers independently. FRR’s own documentation offers the counter- argument — that always comparing “should eliminate MED induced oscillations” — and both statements are true: the default is safer, and the default is also the configuration in which MED oscillation is possible.

The discipline: enable it deliberately, network-wide rather than on one router, and write down why. Enabling it on a single router in an AS is how you get two routers that disagree about the best path for the same prefix.

IGP cost tie-break and ECMP

Step 9 is “lowest IGP cost to the next-hop”, and it is the last step before the multipath check. For an ECMP set to form, the IGP costs to every candidate next-hop must be equal.

Two eBGP peers, ISP-A at 192.0.2.1 and ISP-B at 192.0.2.2, with IGP cost 5 to the first and 10 to the second: BGP selects ISP-A and never reaches the multipath check with ISP-B still in play. This is the failure that survives both knobs, because neither maximum-paths nor multipath-relax touches step 9 — which is why an operator who has configured both and still sees one path should look here next.

The fix is to align the costs on the links to each peer:

configure
set protocols ospf interface eth0 cost 5
set protocols ospf interface eth1 cost 5
commit
save

Two cautions before you do that. The OSPF interface cost is not private to this decision — every SPF calculation in the area uses it, so equalising two links to make BGP multipath form also changes how transit traffic moves through the router. And if these are directly connected eBGP peers, they may not be in OSPF at all; the “IGP cost” FRR compares is then the cost of whatever route resolves the next-hop, most often a connected route at cost 0 for both, in which case they already tie and step 9 is not your problem.

Check before adjusting:

show ip route 192.0.2.1
show ip route 192.0.2.2

Compare the metric on the two resolving routes. If they already match, the separation is happening at an earlier step and raising or lowering an interface cost will change traffic patterns without fixing anything.

Validation

# 1. The candidate paths exist in the BGP table, and which of
#    them BGP marked as multipath
show bgp ipv4 10.0.0.0/24

# 2. The knobs are actually in the configuration
show configuration commands | match maximum-paths
show configuration commands | match multipath-relax

# 3. The kernel has both next-hops. This is the one that
#    matters: a prefix can be multipath in BGP and single-path
#    in the FIB if something below BGP dropped it
show ip route 10.0.0.0/24

# 4. The IGP cost to each next-hop, for comparison
show ip route 192.0.2.1
show ip route 192.0.2.2

Step 1 is the important reading and it is worth being specific about what to look for rather than trusting a flag. Compare the two paths attribute by attribute in that output: local-pref, AS_PATH, origin, MED, and whether each is marked internal. The first attribute that differs is the step that eliminated the second path, and it tells you which of the fixes in this lesson is the relevant one — or that none of them is.

A clean result is three agreeing observations: BGP shows more than one path contributing, show ip route shows more than one next-hop for the prefix, and the resolving routes for those next-hops have equal metrics. If BGP shows multipath and the kernel shows one next-hop, the problem is below BGP — usually a next-hop that does not resolve — and no amount of BGP tuning will move it.

Hashing, and why “the traffic did not split evenly” is often not a bug

ECMP in the Linux kernel is per-flow, not per-packet: a hash over the packet headers picks the next-hop, and every packet of a flow hashes the same way so TCP does not reorder. By default the IPv4 hash covers layer 3 only. VyOS exposes the layer-4 variant:

set system ip multipath layer4-hashing

With layer-3 hashing, a single source talking to a single destination lands entirely on one next-hop no matter how much traffic it sends — one flow, one hash, one path. Adding the ports to the hash lets separate connections between the same pair of hosts spread across the paths.

This is why a two-path ECMP set tested with one large transfer looks broken and is not. Test with many distinct flows, and judge the result by the interface counters over minutes rather than by a single measurement.

Production failure modes

Only one path despite the knobs being set

The single most common report, and it has one diagnostic:

show bgp ipv4 10.0.0.0/24

Put the two paths side by side and walk the algorithm in order — local-pref, locally originated, AS_PATH, origin, MED, peer type, IGP cost. The first attribute that differs is the answer, and everything after it is irrelevant. Working from the list in order is what turns this from guessing into a two-minute check.

Mapped to the fix:

First difference	Step	Fix
local-preference	3	Align it. This is policy you wrote; find the route-map.
AS_PATH length	5	Not fixable with `multipath-relax`. Prepend on the shorter side, or accept the asymmetry.
AS_PATH content, same length	5	`bestpath as-path multipath-relax`. This is the case the knob exists for.
origin	6	Usually a redistribution somewhere producing `incomplete`. Fix it at the source.
MED, same neighbour AS	7	Align the MEDs, or ask the neighbour to.
MED, different neighbour AS	7	Not being compared at all. If this looks like the difference, it is not — look further down.
internal vs external	8	`bestpath peer-type multipath-relax`, and read the design caveat above first.
IGP cost	9	Align the resolving routes’ metrics.
Nothing differs	10	The paths are tied; the cap is the limit. Check `maximum-paths` is set for this address family.

That last row is worth its own sentence. maximum-paths defaults to 1, and its absence looks exactly like a tie-break problem: two identical paths, one installed. Check the configuration before you go looking for an attribute difference that is not there.

multipath-relax configured, and it changed nothing

The operator has two transits, sees one path, sets bestpath as-path multipath-relax, commits, and nothing moves.

Diagnostic: compare the AS_PATH lengths, not just their contents. If they differ, this knob was never going to help: length is decided at step 5 and the longer path is eliminated before the multipath check. The knob only relaxes the requirement that equal-length paths be identical.

Fix: prepend on the shorter path’s inbound policy to equalise the lengths, or treat the shorter path as primary and the other as backup — which, for two transits of genuinely different topological distance, is usually the correct outcome anyway.

An iBGP path is never in the set with an eBGP path

Two paths for the same prefix, one learned over eBGP and one over iBGP, and only the eBGP one is installed.

Diagnostic: show bgp ipv4 10.0.0.0/24 marks the iBGP path internal. Step 8 prefers eBGP, so it is eliminated before the multipath check.

Fix: this is the default and usually the right behaviour — prefer your own exit over hairpinning through another router. If you genuinely want both, set protocols bgp parameters bestpath peer-type multipath-relax is the knob, and the internal path needs enough capacity to carry its share.

The set forms and then churns

The ECMP set forms, but one member’s session or next-hop is unstable. Every appearance and disappearance changes the size of the next-hop set, which changes the hash, which moves flows that were not on the failing path at all. Users on the healthy path see resets.

Diagnostic:

show bgp ipv4 summary — the Up/Down column on each contributing peer. A member whose uptime keeps resetting is the source.
journalctl -u frr for the session’s reset reasons.

Fix: stabilise the member before tuning anything else. Failing that, remove the unstable path from the set deliberately rather than letting it join and leave — a stable two-path set beats a three-path set that is really two paths plus a metronome. BFD shortens the detection window on a genuinely failing path; it does nothing for one that is flapping between working states.

Rollback

Capture the configuration before the change, from configuration mode, so there is a file to diff against afterwards:

configure
save /config/pre-ecmp-change.boot

Then make the change under commit-confirm, so a mistake reverts itself rather than waiting for you to notice:

commit-confirm 5

If it looks right, confirm within the window; if it does not, do nothing and the router restores the previous configuration on its own.

To undo a committed ECMP change, delete the specific nodes:

configure
delete protocols bgp address-family ipv4-unicast maximum-paths ebgp
delete protocols bgp parameters bestpath as-path multipath-relax
commit
save

Traffic collapses back onto the single best path immediately — this is a data-plane change taking effect at commit, not a gradual reconvergence — so the remaining link carries the whole load from that moment. Size that before rolling back during business hours.

Note that rollback <N> is a different mechanism: it reverts the entire configuration to a stored revision and currently requires a reboot on VyOS. For undoing two BGP nodes, deleting them is faster and does not take the router down.

Production discipline

BGP ECMP checklist

Before committing BGP ECMP in production:

maximum-paths <ebgp|ibgp> is set under every address family you care about. IPv4 and IPv6 are separate; setting one and assuming the other is the quietest failure here.
bestpath as-path multipath-relax is set when the candidate paths have equal AS_PATH length but different ASes — and you have confirmed the lengths really are equal, because the knob does nothing when they are not.
The resolving routes for the candidate next-hops have equal metrics, checked with show ip route <next-hop> rather than assumed.
MED is understood for this pair of paths: same neighbouring AS means it is compared, different means it is not, and always-compare-med is a network-wide decision rather than a per-router one.
The hash is understood. Layer-3 hashing pins a single flow to a single path; set system ip multipath layer4-hashing adds the ports to the hash. Either way it is per-flow, so a one-stream throughput test proves nothing.
BFD is configured on the contributing sessions, so a failed member leaves the set in milliseconds rather than at the BGP hold time.
The change is made under commit-confirm, and the rollback is written down as the specific delete commands.

The discipline: BGP ECMP is not a feature you enable. It is what remains when nothing in the selection algorithm separated the paths, and maximum-paths is only the cap on how many survivors may be installed. Diagnose it by walking the steps in order, not by adding knobs.

Cross-course references

Part XXVI-02 (XXVI-VyOS-BGPAttributes / AS-path) covers the AS-path attribute and the tie-break.
Part XXVII-05 (XXVII-VyOS-BGPBestPath / IGP cost tiebreak) covers the best-path algorithm in detail.
Part XXXVI-01 (XXXVI-VyOS-ECMP / concept) covers the conceptual model.
Part XXXVI-02 (XXXVI-VyOS-ECMP / config) covers the configuration of ECMP including maximum-paths.
Part XXXII (XXXII-VyOS-BFD) covers BFD for fast failure detection.

Quiz

Knowledge check · 4 questions

Q1. R1 has two eBGP peers advertising 10.0.0.0/24 with AS_PATH lengths 1 and 3. `maximum-paths ebgp 4` is configured but only one path is installed. Which statement is correct?

`bestpath as-path multipath-relax` will make both paths multipath candidates Raising `maximum-paths ebgp` to 8 will admit the second path The AS_PATH length difference is resolved before the multipath check, so neither knob helps `always-compare-med` will equalise the paths because MED is compared before AS_PATH
Q2. On VyOS 1.5, an unqualified `set protocols bgp address-family ipv4-unicast maximum-paths 4` configures a single multipath budget shared between eBGP and iBGP paths.

True False
Q3. R1 has two eBGP peers, ISP-A and ISP-B, both advertising 10.0.0.0/24 with equal-length AS_PATHs. `maximum-paths ebgp 4` and `bestpath as-path multipath-relax` are both configured, and the routing table still shows one next-hop. Walk the diagnosis and name the fix, including what the fix costs elsewhere.

R1 reaches ISP-A via next-hop 192.0.2.1 and ISP-B via 192.0.2.2. Both next-hops are resolved through OSPF, and the route to 192.0.2.1 has metric 5 while the route to 192.0.2.2 has metric 10. Both peers advertise 10.0.0.0/24 with AS_PATH length 2. `show bgp ipv4 10.0.0.0/24` lists both paths; `show ip route 10.0.0.0/24` shows only 192.0.2.1.
Q4. R1 has an eBGP transit session and an iBGP path via R2 for the same prefix. An engineer proposes `set protocols bgp address-family ipv4-unicast maximum-paths 8` to load-share across both. Explain why that command does not do what they think, what the correct knob is, and why you might still decline to use it.

R1 in AS 64512 peers with ISP-A (AS 64513) over eBGP and with R2 (AS 64512) over iBGP. Both offer 10.0.0.0/24 with the same AS_PATH; R2 learned it from its own transit session. R1 installs only the eBGP path. R1 and R2 are connected by a single 1 Gbit/s link; each of their transit circuits is 10 Gbit/s.

Passing score: 75%. Answers are checked in this browser.