ECMP configuration — multiple next-hops, max-paths, load-balancing algorithm

ECMP configuration on VyOS 1.5 LTS / FRR 10.x differs by protocol, and the differences are not cosmetic. Static routes take multiple next-hops as separate next-hop arguments. OSPF and BGP both use maximum-paths, but the node sits in a different place in each tree and the defaults are opposite ways round. The hashing that decides which next-hop a packet takes is not in the routing configuration at all — it is a kernel sysctl.

This lesson is the operator’s reference for the commands, for the two places where the CLI does not offer what people expect it to, and for the production failure modes that arise when the configuration is incomplete.

Static routes with multiple next-hops

The simplest ECMP configuration: a static route with multiple next-hops.

set protocols static route 10.0.0.0/24 next-hop 192.0.2.1
set protocols static route 10.0.0.0/24 next-hop 192.0.2.2
set protocols static route 10.0.0.0/24 next-hop 192.0.2.3
commit
save

The route has three next-hops. The router keeps all three in the kernel’s FIB and distributes traffic across them via the hashing algorithm.

vyos@r1:~$ show ip route 10.0.0.0/24

Routing entry for 10.0.0.0/24
Known via "static", distance 1, metric 0, best
Last update 00:02:14 ago
* 192.0.2.1, via eth0, weight 1
* 192.0.2.2, via eth0, weight 1
* 192.0.2.3, via eth0, weight 1

Three * lines means three next-hops are selected, not one selected and two standing by. Confirm it in the kernel, because that is where forwarding actually happens:

vyos@r1:~$ ip route show 10.0.0.0/24

10.0.0.0/24 proto static
      nexthop via 192.0.2.1 dev eth0 weight 1
      nexthop via 192.0.2.2 dev eth0 weight 1
      nexthop via 192.0.2.3 dev eth0 weight 1

Both blocks above are illustrative of the shape rather than captures — read them for the structure (one nexthop line per path, each with a weight) and not for the exact spacing, which varies with the FRR and iproute2 versions in a given release.

The weight 1 on each line is the point of interest. It is not a knob you were given: the documented sub-nodes under a static route’s next-hop are disable, distance and bfd (with bfd profile and bfd multi-hop source-address beneath it). There is no per-next-hop weight, so VyOS static ECMP is equal-cost only, and every path gets an equal share of the hash space.

If you need weighted distribution across static next-hops, this is not the tool. The options are to move the decision into a dynamic protocol where the metric expresses the preference, or to accept equal sharing and size the links to match.

OSPF maximum-paths

OSPF installs up to maximum-paths equal-cost paths in the routing table. Two things to unlearn here: the node sits directly under protocols ospf, not under parameters, and the default is 64, not 1.

set protocols ospf parameters router-id 1.1.1.1
set protocols ospf area 0 network 192.0.2.0/24
set protocols ospf maximum-paths 4
commit
save

Note the asymmetry in that block: router-id is under parameters and maximum-paths is not. It is not a typo, and putting maximum-paths under parameters will not commit.

Because the default is 64 (VyOS documents the range as 1-64 and the default as MULTIPATH_NUM, which is 64), maximum-paths on OSPF is a limit you impose, not a feature you switch on. If OSPF ECMP is not happening, the cause is almost never this knob — it is that the paths are not actually equal cost. Setting maximum-paths 4 here caps the router at four paths where it would otherwise install everything it finds; that is a deliberate ceiling, usually chosen to match what the hardware or the monitoring can reason about.

The validation:

vyos@r1:~$ show ip route 10.0.0.0/24

Routing entry for 10.0.0.0/24
Known via "ospf", distance 110, metric 20, best
Last update 00:05:41 ago
* 192.0.2.1, via eth0, weight 1
* 192.0.2.2, via eth0, weight 1
* 192.0.2.3, via eth0, weight 1
* 192.0.2.4, via eth0, weight 1

Read the metric field as well as the next-hop count. All four paths are installed because they arrived at the same metric of 20 — OSPF ECMP requires the costs to be equal, with no relaxation available. There is no OSPF equivalent of BGP’s multipath-relax.

So when OSPF ECMP does not happen, the fix is on the cost side: align the interface costs (set protocols ospf interface <if> cost <n>), or check for an intermediate link whose bandwidth gives it a different auto-cost. Raising maximum-paths will not help, because it was already 64.

BGP maximum-paths and multipath-relax

BGP installs up to maximum-paths equal-cost paths, and unlike OSPF the default really is single-path. The 1.5 command shape differs from older material in three ways at once — the local AS moved to system-as, the knob moved under the global address-family, and there is no bare maximum-paths <n>:

set protocols bgp system-as 64512
set protocols bgp address-family ipv4-unicast maximum-paths ebgp 4
set protocols bgp address-family ipv4-unicast maximum-paths ibgp 4
commit
save

Two knobs, not three. The documented node is address-family <ipv4-unicast|ipv6-unicast> maximum-paths <ebgp|ibgp> <1-256>; an unqualified maximum-paths 4 has no node to land on. That is a better design than it first looks, because eBGP and iBGP multipath are genuinely different decisions: eBGP multipath spreads traffic across providers, iBGP multipath spreads it across exits inside your own AS, and an operator almost never wants the same number for both.

Note also that this is per address-family. Setting it under ipv4-unicast does nothing for IPv6; the v6 ECMP you expected will quietly be single-path until you set address-family ipv6-unicast maximum-paths ebgp 4 as well.

BGP’s maximum-paths requires the paths to be “equal” by the BGP best-path algorithm: same weight, local-preference, AS-path length, origin, MED, and IGP cost. The operator who wants ECMP across eBGP peers must align all of these.

multipath-relax loosens the AS-path requirement, and it lives under parameters bestpath as-path:

set protocols bgp address-family ipv4-unicast maximum-paths ebgp 4
set protocols bgp parameters bestpath as-path multipath-relax
commit
save

With multipath-relax, two eBGP paths are considered equal even if their AS-path lengths differ. This is the typical configuration for ECMP across two ISPs (one with a shorter AS-path to the destination, one with a longer).

flowchart LR
  AS64512["R1 in AS 64512"]
  ISP1["ISP-A in AS 64513"]
  ISP2["ISP-B in AS 64514"]
  DEST["10.0.0.0/24 in AS 64515"]
  AS64512 -- "eBGP" --> ISP1
  AS64512 -- "eBGP" --> ISP2
  ISP1 -- "AS-path: 64513" --> DEST
  ISP2 -- "AS-path: 64514 64516 64517" --> DEST
  AS64512 -. "without multipath-relax:<br/>ISP-A only (shorter AS-path)" .-> DEST
  AS64512 -. "with multipath-relax:<br/>both ISPs (relaxed)" .-> DEST

The discipline: multipath-relax is the right choice for multi-ISP ECMP; it is not appropriate when the operator specifically wants the shorter AS-path to win.

The kernel hashing algorithm

The kernel decides which next-hop a packet takes, and the knob is a sysctl. On VyOS 1.5 sysctls are set through the CLI as set system sysctl parameter <name> value <value>:

configure
set system sysctl parameter net.ipv4.fib_multipath_hash_policy value '1'
set system sysctl parameter net.ipv6.fib_multipath_hash_policy value '1'
commit
save

Read the values from the kernel documentation rather than from habit, because the naming is not what most people assume:

Policy 3 is the only one that consults fib_multipath_hash_fields, and that field is a bitmask, not an enumeration. The kernel’s list: 0x0001 source IP, 0x0002 destination IP, 0x0004 IP protocol, 0x0010 source port, 0x0020 destination port, with a parallel set from 0x0040 upwards for inner headers. The default is 0x0007 — source IP, destination IP and protocol.

configure
set system sysctl parameter net.ipv4.fib_multipath_hash_policy value '3'
set system sysctl parameter net.ipv4.fib_multipath_hash_fields value '0x0033'
commit
save

0x0033 is source IP, destination IP, source port and destination port — a 4-tuple that ignores the protocol number. Whether that is a good idea depends entirely on your traffic; the point is that the number is assembled from the bit list, not picked from a menu of levels.

BFD for ECMP failure detection

The default ECMP failure detection relies on the routing protocol’s hello/keepalive (OSPF hello, BGP keepalive). The detection time is seconds. For faster detection, enable BFD:

set protocols bgp neighbor 192.0.2.1 bfd
set protocols ospf interface eth0 bfd
commit
save

Note that the BGP path carries no ASN — VyOS 1.4 moved the local AS to set protocols bgp system-as <asn> and put peers under set protocols bgp neighbor.

BFD detects path failure in milliseconds. The operator who deploys ECMP across links with different failure modes (e.g., a fibre link and a wireless link) should enable BFD to detect the wireless link’s intermittent failures quickly.

The validation:

show bfd peers
# Should show all ECMP next-hops as BFD peers with state Up

A clean BFD configuration: every ECMP next-hop is a BFD peer; every BFD peer reports Up state; a path failure is detected in milliseconds.

What VyOS does not expose for ECMP

It is worth being explicit about a gap, because the shape of the CLI invites an assumption that is wrong.

There is no VyOS configuration node that selects the ECMP hashing algorithm. No set system load-balancing flow-based, no packet-based, and no show system load-balancing. The kernel sysctls above are the entire surface, reached through set system sysctl parameter.

set load-balancing ... does exist, but it is a different subsystem: the multi-WAN load balancer, which steers traffic with connection tracking and its own per-rule interface list. That is where per-packet-balancing lives, and it operates on WAN rules rather than on the FIB’s equal-cost next-hop set. Confusing the two is easy — the words overlap almost completely — and the symptom is a configuration that commits cleanly and does nothing to your ECMP routes.

The practical consequence for a change record: an ECMP hashing change on VyOS is a sysctl change, and it does not appear anywhere in the routing configuration. Anyone auditing show configuration commands | match protocols will not see it. Note it explicitly, or the next engineer will spend an afternoon looking for a routing knob that does not exist.

Validation

# 1. The routing daemon installed more than one next-hop
show ip route 10.0.0.0/24

# 2. The kernel FIB agrees. This is the one that decides forwarding;
#    a route with multiple next-hops in FRR and one here is a
#    zebra-to-kernel problem, not a protocol problem.
ip route show 10.0.0.0/24

# 3. The hash policy is what you think it is. Read it from the
#    kernel, not from the configuration: 0 is L3, 1 is L4.
cat /proc/sys/net/ipv4/fib_multipath_hash_policy
cat /proc/sys/net/ipv6/fib_multipath_hash_policy

# 4. The setting is in the configuration and will survive a reboot
show configuration commands | match fib_multipath

# 5. The traffic actually distributes. One flow proves nothing --
#    a single TCP connection hashes to exactly one next-hop by design.
iperf3 -c 10.0.0.2 -P 10

# 6. Failure detection is armed on every path
show bfd peers

# 7. A failure removes the path. Do this deliberately, in a window.
show ip route 10.0.0.0/24

A clean validation: multiple next-hops in the routing table; per-flow hash; BFD enabled; traffic distributes across paths; path failure is detected.

Production failure modes

Only one next-hop installed

The operator configured two next-hops but the routing table shows one. The causes:

• Unequal costs — OSPF/BGP picks the lower-cost path. Align the costs.
• Wrong protocol knob — for BGP, address-family ipv4-unicast maximum-paths ebgp|ibgp is unset, so the default of one path applies. For OSPF the default is 64, so this is almost never the cause there; suspect unequal cost first.
• Right knob, wrong address-family — the BGP node is per address-family. A v4-only maximum-paths leaves v6 single-path, and everything looks configured.
• Different administrative distance — for static routes, the next-hops must have the same distance.

Traffic clusters on one path

The operator sees throughput on one path much higher than the other. Causes:

• Hash clustering — few flows or biased flows. Verify with iperf3 with many parallel streams.
• Wrong hash policy — per-packet hashing on TCP. Set to per-flow.

Path failure not detected

A path fails but the routing table still shows it. The router continues to hash some flows to the failed path. Causes:

• BFD not enabled — the protocol’s hello/keepalive may take seconds to detect the failure. Enable BFD.
• No interface-state tracking — the interface is up but the path is unreachable. Use BFD or peer-tracking.

Asymmetric routing breaks stateful firewall

The forward and reverse paths use different ECMP next-hops; the stateful firewall in the middle drops the reverse packets. See Part XXXVI-01 for the detailed scenario.

Rollback

# Capture the running configuration
show configuration commands | save /tmp/ecmp-config-$(date +%s).txt

# Compare
compare

# Remove the second next-hop (revert to single-path)
delete protocols static route 10.0.0.0/24 next-hop 192.0.2.2
commit

# Or drop the BGP multipath knobs
delete protocols bgp address-family ipv4-unicast maximum-paths ebgp
delete protocols bgp parameters bestpath as-path multipath-relax
commit

Deleting protocols ospf maximum-paths is not a rollback to single-path: the default is 64, so removing your cap widens the ECMP set rather than closing it. If the intent is to stop OSPF ECMP, set set protocols ospf maximum-paths 1 explicitly.

The sysctl is its own rollback path and is easy to forget: delete system sysctl parameter net.ipv4.fib_multipath_hash_policy restores the kernel default of 0, and it is a separate commit from anything under protocols.

The rollback reverts to single-path routing. The traffic returns to the original next-hop.

The disciplined rollback uses commit-confirm:

commit-confirm 5
# If the ECMP change has unintended consequences, the auto-rollback
# fires after 5 minutes.

Production discipline

Cross-course references

• Part XII-03 (XII-VyOS-StaticRouting / static route options) covers the static route configuration including distance and next-hop options.
• Part XVIII-06 (XVIII-VyOS-OSPFFund / SPF and cost) covers OSPF cost and the equal-cost requirement for ECMP.
• Part XXVII-05 (XXVII-VyOS-BGPBestPath / IGP cost tiebreak) covers BGP’s equal-cost requirement and the role of IGP cost in the best-path algorithm.
• Part XXXII (XXXII-VyOS-BFD) covers the BFD configuration for fast failure detection.
• Part XXXVI-01 (XXXVI-VyOS-ECMP / concept) covers the conceptual model.
• Part XXXVI-03 (XXXVI-VyOS-ECMP / BGP) covers the BGP- specific maximum-paths and multipath-relax settings.
• Part XXXVI-05 (XXXVI-VyOS-ECMP / fwmark) covers the policy-based ECMP with fwmark.

Quiz