ECMP — equal-cost paths and the per-flow decision

Equal-Cost Multi-Path (ECMP) lets a router spread traffic for one destination across several next-hops of equal cost. Two ISPs, two uplinks, two fibre pairs — ECMP uses them all without any protocol negotiating who gets what. The Linux forwarding plane picks a next-hop by hashing a few header fields, and because the hash is deterministic, every packet of a conversation normally lands on the same path.

“Normally” is where the operator’s trouble starts. The hash is stateless — the kernel is not remembering a decision it made for this flow, it is recomputing the same arithmetic and getting the same answer. Change the number of next-hops and the arithmetic changes, so live conversations move paths mid-flight. Put a stateful filter downstream of that, and the moved conversations die.

This lesson is the operator’s foundation in ECMP: what it actually is on Linux, how to configure it on VyOS 1.5, and the failure modes that surface as session breakage.

What ECMP is

ECMP is the FIB’s ability to hold more than one next-hop for a destination. When zebra installs a route it can install a single next-hop (the usual case) or a set of next-hops that the routing protocol judged equal (the ECMP case).

flowchart LR
  H["Host A"] -->|"packet to 8.8.8.8"| R["VyOS router"]
  R -->|"hash -> next-hop 1"| A["Path A\nISP-1"]
  R -->|"hash -> next-hop 2"| B["Path B\nISP-2"]
  A -->|"forward"| D["Destination"]
  B -->|"forward"| D

ECMP requires that all paths are equal, and “equal” is defined by whatever installed the route. OSPF compares path cost. BGP compares the best-path attributes, which is a much longer list. Static routes compare administrative distance. If the comparison produces a winner, there is no ECMP — the loser is not installed at all.

The hash is not the 5-tuple by default

The most common wrong belief about Linux ECMP is that it hashes the 5-tuple. Out of the box it does not. The behaviour is selected by net.ipv4.fib_multipath_hash_policy, and the default is 0:

Under the default, every packet between one pair of addresses takes one path, no matter how many TCP connections are open between them. A backup job from one server to one target will not spread across two uplinks; it will fill one and leave the other idle, and the operator will conclude that ECMP “is not working” when it is working exactly as configured.

Raise it to layer 4 through the configuration tree so it survives a reboot and appears in the configuration archive:

configure
set system sysctl parameter net.ipv4.fib_multipath_hash_policy value 1
set system sysctl parameter net.ipv6.fib_multipath_hash_policy value 1
commit
save

IPv4 and IPv6 have separate knobs; setting one does not set the other.

Configuring ECMP for static routes

Give the same prefix more than one next-hop and VyOS installs them as an ECMP set:

configure
set protocols static route 10.0.0.0/24 next-hop 192.0.2.1
set protocols static route 10.0.0.0/24 next-hop 198.51.100.1
commit
save

Both next-hops go into the FIB with equal share. Nothing here checks whether either of them works — see “Failure detection” below, which is the part operators skip.

Configuring ECMP for dynamic routes

OSPF installs equal-cost paths on its own. maximum-paths caps how many it will install; the CLI accepts 1 to 64:

configure
set protocols ospf parameters router-id 192.0.2.254
set protocols ospf area 0 network 192.0.2.0/24
set protocols ospf maximum-paths 4
commit
save

BGP is the opposite: it is single-path unless you say otherwise. BGP’s job is to select one best path, and it installs exactly that one until maximum-paths gives it permission to install more. In VyOS 1.5 the knob sits under the address family, not at the top of the BGP tree:

configure
set protocols bgp system-as 65000
set protocols bgp address-family ipv4-unicast maximum-paths ebgp 2
set protocols bgp address-family ipv4-unicast maximum-paths ibgp 2
set protocols bgp parameters bestpath as-path multipath-relax
commit
save

maximum-paths ebgp and maximum-paths ibgp are separate limits and a path only counts against the one that matches its peer type.

bestpath as-path multipath-relax is needed whenever the candidate paths come from different neighbouring ASNs. Without it BGP requires the AS_PATH values to be identical, not merely the same length, so two transit providers advertising the same prefix will never be multipath candidates however high maximum-paths is set. This is the single most common reason a BGP ECMP configuration installs one route and the operator cannot see why.

Stateless hashing, not flow pinning

Because the kernel keeps no per-flow record, a change to the next-hop set rehashes everything. This is the ECMP behaviour that surprises people in production:

• Bringing up a second uplink does not only affect new conversations. Existing ones rehash across two next-hops instead of one, and roughly half of them move.
• A flapping next-hop rehashes twice per flap.
• A conversation that moves is fine if both paths are equivalent end to end, and dead if the far side keeps per-path state — a stateful firewall, a NAT device, or a TLS-terminating load balancer that pinned the session to a source address.

VyOS’s FIB has no per-packet ECMP mode. The hash is computed per packet, but over fields that do not change within a conversation, so the effect is per-conversation. Deliberate per-packet spraying is a different subsystem: set load-balancing wan rule 1 per-packet-balancing on the WAN load-balancer, which is mark-and-policy-route rather than FIB ECMP.

ECMP and stateful firewalls

Conntrack belongs to the host, not to an interface. This matters, because it decides which asymmetric-routing stories are real.

sequenceDiagram
  autonumber
  participant C as Client
  participant FW as VyOS router
  participant A as Path A
  participant B as Path B
  participant S as Server

  C->>FW: SYN
  FW->>A: SYN (hash selects path A)
  A->>S: SYN
  S-->>B: SYN-ACK (upstream returns via B)
  B->>FW: SYN-ACK arrives on eth2
  Note over FW: conntrack entry exists — it is not per-interface
  FW->>C: SYN-ACK, unless something rejects the arrival interface

If both uplinks land on the same VyOS router, the return packet still finds its conntrack entry no matter which interface it came in on. The connection works. Two things break it anyway, and they are the two to check:

Reverse-path filtering. With set firewall global-options source-validation strict, the kernel drops a packet whose source address is not reachable back out of the interface it arrived on. Under ECMP the best route back may point at the other uplink, so strict validation drops perfectly legitimate return traffic. loose mode — the source must be reachable by some route — is the usual setting on a multi-homed edge, and disable leaves it to the firewall rules.

Two separate firewalls. When the forward and return paths traverse different boxes, each has its own conntrack table. The box that only ever sees the SYN-ACK has no entry for it, classifies it INVALID, and drops it. This is the classic asymmetric-routing failure, and it is a topology problem — no amount of tuning on either firewall fixes it, because neither one has the whole conversation.

Failure detection

Nothing in the ECMP mechanism checks that a next-hop works. A static next-hop whose upstream is dead but whose ARP still answers keeps receiving its share of the hash space, and that share is blackholed.

VyOS attaches BFD to static next-hops:

configure
set protocols bfd profile FAST interval receive 300
set protocols bfd profile FAST interval transmit 300
set protocols bfd profile FAST interval multiplier 3
set protocols static route 10.0.0.0/24 next-hop 192.0.2.1 bfd profile FAST
set protocols static route 10.0.0.0/24 next-hop 198.51.100.1 bfd profile FAST
commit
save

BFD requires the far end to speak BFD. Where it does — a peering router, a datacentre fabric — it gives sub-second withdrawal, and zebra pulls the next-hop out of the ECMP set on detection.

Operational commands

show ip route 10.0.0.0/24
show ip route summary

The kernel view, from the shell:

ip route show 10.0.0.0/24
ip nexthop show
ip route get 10.0.0.5 from 192.0.2.50 ipproto tcp sport 5000 dport 443

An ECMP route in the kernel prints its next-hops beneath the prefix:

10.0.0.0/24 proto ospf metric 20
        nexthop via 192.0.2.1 dev eth1 weight 1
        nexthop via 198.51.100.1 dev eth2 weight 1

Where zebra used a kernel nexthop group, the route shows nhid followed by an identifier instead, and ip nexthop show id resolves that identifier to the same list. Both forms mean the same thing; which one you get depends on the build and the kernel.

ip route get with the full tuple is the useful one for a hash question — it asks the kernel which next-hop this traffic would take. Vary the source port and re-run it: under hash policy 0 the answer never changes, and under policy 1 it does. That single experiment settles most “is ECMP working” arguments.

Failure modes

A dead path keeps its share of the traffic

A static ECMP next-hop fails. No BFD is configured, the neighbour entry still resolves, and the kernel keeps hashing traffic to it. Roughly half of all conversations fail; the other half are fine, which is why the ticket says “intermittent”.

Diagnostic:

• show ip route 10.0.0.0/24 still lists both next-hops.
• ip neigh show 192.0.2.1 — FAILED or STALE on the dead one, REACHABLE on the live one.
• A ping from the router sourced from each next-hop’s interface: one succeeds, one does not.

Fix: attach BFD, or set fib_multipath_use_neigh where the failure is an unresponsive directly-connected next-hop.

BGP installs one path and everything looks correct

maximum-paths is configured, the peers are up, both advertise the prefix, and only one next-hop reaches the FIB.

Diagnostic:

• show ip bgp 10.0.0.0/24 — the candidate paths are all there, and only one carries multipath alongside best.
• Compare the AS_PATH values. Different upstream ASNs need bestpath as-path multipath-relax.
• Compare MED, local preference, origin and the IGP metric to each next-hop. Multipath requires equality through the best-path comparison, not just equal path length.

Fix: set protocols bgp parameters bestpath as-path multipath-relax for the AS_PATH case; align the offending attribute for the others.

Adding an uplink drops live sessions

The second uplink comes up during a change window, the ECMP set goes from one next-hop to two, and about half the established conversations rehash onto the new path. Where a downstream device kept per-path state, they break.

Diagnostic: the breakage starts at the exact commit, affects roughly half of sessions, and new connections are fine.

Fix: this is expected behaviour, so plan for it — bring ECMP paths up in a maintenance window and treat “existing sessions will rehash” as a stated impact of the change, not a surprise.

Validation

The sequence for “ECMP is not working”:

1. The paths are installed: show ip route 10.0.0.0/24 lists more than one next-hop. If not, the problem is upstream of ECMP — best-path selection, maximum-paths, or unequal cost.
2. Every next-hop is alive: ip neigh show for each, plus a probe through each one. A FAILED neighbour in an ECMP set is the incident.
3. The hash spreads the traffic you have: ip route get with two different source ports. Same answer under policy 0 is correct behaviour, not a fault.
4. Return traffic is accepted: show conntrack table ipv4 for the flow, and a capture on each uplink if it is missing.
5. End to end: a real connection establishes and transfers data, tested with enough distinct source addresses to actually exercise both paths.

Step 3 is the one people skip, and it is where most “only one path is used” reports end.

Cross-course references

• The Linux course’s XXI-Linux-NetAdvanced covers the same FIB behaviour from the host side.
• The OPNsense course covers the FreeBSD equivalent, where the state table and the path selection interact differently.
• The multi-WAN part later in this course covers load-balancing wan, which solves a related problem with a different mechanism.

Quiz

Production discipline

ECMP is arithmetic, not a session manager. It spreads traffic across equal paths, it checks nothing, it weights nothing, and it rehashes whenever the path set changes. Decide the hash policy deliberately, attach failure detection to every next-hop before you rely on the redundancy, and treat “adding a path moves existing conversations” as a planned impact. Then multi-path either works or fails in a way you can see.