ECMP concept — Equal-Cost Multi-Path, kernel flow-based or route-based hashing, throughput scaling

Equal-Cost Multi-Path (ECMP) is the routing primitive that uses multiple next-hops for the same destination. Instead of selecting one next-hop and discarding the others, the router keeps all equal-cost next-hops and distributes traffic across them.

This lesson is the conceptual reference for ECMP on VyOS 1.5 LTS / FRR 10.x: what ECMP is, how the kernel hashes the traffic, why the throughput scales, and the production failure modes that arise when the hash is wrong, the paths are asymmetric, or only one path is used.

What ECMP is

ECMP is a routing-table primitive. When multiple routes to the same destination have the same metric (or administrative distance, in the case of static routes), the router keeps all of them in the routing table and forwards traffic across the set. The selection of which next-hop to use for a given packet is the hashing function.

flowchart LR
  DEST["Destination 10.0.0.0/24"]
  NH1["Next-hop A<br/>(10.0.0.1)"]
  NH2["Next-hop B<br/>(10.0.0.2)"]
  NH3["Next-hop C<br/>(10.0.0.3)"]
  ROUTER["R1<br/>(ECMP)"]
  DEST --> ROUTER
  ROUTER -- "hash(packet) % 3 = 0" --> NH1
  ROUTER -- "hash(packet) % 3 = 1" --> NH2
  ROUTER -- "hash(packet) % 3 = 2" --> NH3

Three reasons a production network engineer reaches for ECMP:

1. Throughput scaling. Two 10G links in ECMP give 20G of forwarding capacity for the same destination (in the ideal case; the actual throughput depends on the hash distribution).
2. Redundancy. If one link fails, the others continue to forward. ECMP is a poor man’s LACP for routed paths (where LACP is not available — e.g., across multiple ISPs).
3. Cost. Two cheap 10G links may be cheaper than one expensive 40G link. ECMP lets the operator build bandwidth from commodity parts.

The kernel hashing model

The Linux kernel supports two ECMP hashing modes:

• Per-flow hashing (default). All packets belonging to the same flow (defined by source/destination IP and protocol, plus source/destination port for TCP/UDP) are forwarded via the same next-hop. A flow is a sequence of packets that share the same 5-tuple (or 3-tuple for ICMP). The hash is computed once per flow; subsequent packets of the same flow use the same next-hop.
• Per-packet hashing. Each packet is hashed independently; different packets of the same flow may use different next-hops. This mode is rare and is usually a misconfiguration.

Per-flow hashing is the production default because it preserves the flow affinity property: a TCP connection’s packets all arrive at the same destination, in order. If packets of a flow were distributed across multiple next-hops with different latencies, the receiving end would see out-of-order packets, retransmissions, and degraded throughput.

flowchart LR
  subgraph "Per-flow hashing"
    F1["Flow A<br/>(5-tuple: 10.0.0.1:80 -> 10.0.0.2:5000)"]
    F2["Flow B<br/>(5-tuple: 10.0.0.1:80 -> 10.0.0.3:5001)"]
    F3["Flow C<br/>(5-tuple: 10.0.0.1:80 -> 10.0.0.4:5002)"]
    HASH["hash(5-tuple) % N"]
    NH1["Next-hop 1"]
    NH2["Next-hop 2"]
    F1 --> HASH
    F2 --> HASH
    F3 --> HASH
    HASH -- "0" --> NH1
    HASH -- "1" --> NH2
  end

  subgraph "Per-packet hashing"
    P1["Packet 1"]
    P2["Packet 2 (same flow)"]
    P3["Packet 3 (same flow)"]
    HASH2["hash(packet) % N"]
    NH3["Next-hop 1"]
    NH4["Next-hop 2"]
    P1 --> HASH2
    P2 --> HASH2
    P3 --> HASH2
    HASH2 -- "0" --> NH3
    HASH2 -- "1" --> NH4
  end

The kernel’s hashing algorithm:

# Default algorithm (Linux 5.x and later)
# hash = ((src_ip XOR dst_ip) * 31 + src_port XOR dst_port) mod N

The hash is computed over the source/destination IP and protocol port. The same flow always hashes to the same next-hop (modulo N); different flows hash to different next-hops with reasonable uniformity.

The VyOS configuration to inspect or change the algorithm:

# Inspect the algorithm
cat /proc/sys/net/ipv4/fib_multipath_hash_policy
# 0 = per-packet (rare; do not use)
# 1 = per-flow (default; recommended)

# Set per-flow
set system sysctl net.ipv4.fib_multipath_hash_policy 1

Throughput scaling argument

In the ideal case, ECMP scales throughput linearly with the number of paths. Two 10G links give 20G; three give 30G; four give 40G. The actual throughput is the product of link speed and hash uniformity.

The hash uniformity matters. If the hash function distributes flows uniformly across the paths, throughput scales linearly. If the hash function clusters flows (e.g., a poorly-chosen hash that always returns the same value for a common 5-tuple pattern), throughput does not scale.

Common sources of hash clustering:

• Few flows. With only 10 flows and 4 paths, the hash may cluster 3-3-2-2 (acceptable) or 10-0-0-0 (one path carries all the traffic). The throughput does not scale until the number of flows exceeds the number of paths.
• Biased flows. A single elephant flow (a long-lived, high-bandwidth TCP connection) may dominate. The hash places it on one path; the other paths are idle.
• Bad hash function. A historical kernel bug (fib_multipath_hash_policy related) clustered flows on the same path. Fixed in modern kernels but worth verifying.

Asymmetric routing

A consequence of per-flow hashing: the forward and reverse paths of a flow may use different ECMP paths. The hash on each router is computed independently; a router on the forward path may hash the flow to next-hop A, while the router on the reverse path may hash it to next-hop B.

flowchart LR
  S["Source"]
  A["R-A (ECMP forward)"]
  B["R-B (ECMP reverse)"]
  H1["Next-hop 1"]
  H2["Next-hop 2"]
  S -- "packet" --> A
  A -- "hash(forward) = 0" --> H1
  H1 --> B
  B -- "hash(reverse) = 1" --> H2
  H2 --> S

The asymmetric routing is fine for connectivity — packets arrive at the destination — but creates production challenges:

• Firewall state. A stateful firewall that expects symmetric routing will drop the reverse packets if they arrive via a different path than the forward path. The firewall does not have state for the reverse flow because the forward flow took a different path through the firewall.
• QoS / traffic shaping. A shaper that applies per-flow bandwidth limits may apply the limits on the forward path but not the reverse path (or vice versa). The traffic’s bandwidth is asymmetric.
• Netflow / IPFIX. The flow records show the forward and reverse paths as different flows. The operator who correlates flow records by 5-tuple sees two flows with the same 5-tuple but different paths.

The discipline: ECMP in a network with stateful firewalls requires the firewalls to participate in the same hash. The ECMP paths must be designed so the forward and reverse paths hash to the same firewall (or the firewalls must allow asymmetric paths, which is rare).

When only one path is used

ECMP requires equal-cost paths. If one path has a higher metric, the router uses the lower-cost path and ignores the others. The operator who sees only one path used must check the metric.

For OSPF:

# R1 has two paths to 10.0.0.0/24
# Path A: cost 10 (via OSPF, 2 hops)
# Path B: cost 20 (via OSPF, 4 hops)
show ip route 10.0.0.0/24
# Only path A appears; path B is hidden because the cost is higher

For BGP:

# R1 has two paths to 10.0.0.0/24
# Path A: eBGP, AS-path 64513
# Path B: eBGP, AS-path 64514 64515
# The AS-path length differs; only path A is best
show ip bgp 10.0.0.0/24
# Only path A appears as best

For ECMP to apply, the metrics must be equal. The operator who wants ECMP must align the costs (OSPF cost, BGP attributes, or static route distance).

Validation

# 1. The ECMP routes exist in the routing table
show ip route 10.0.0.0/24
# Multiple next-hops should appear

# 2. The kernel's FIB has the ECMP entries
ip route show 10.0.0.0/24
# Should show multiple nexthops

# 3. The hash policy is per-flow
cat /proc/sys/net/ipv4/fib_multipath_hash_policy
# Should be 1

# 4. The traffic distribution
# Use a flow generator (e.g., iperf3 with many parallel streams)
# and observe the per-path utilisation
iperf3 -c 10.0.0.2 -P 10
# 10 parallel streams; observe the throughput

# 5. The path failure handling
# Disable one path (e.g., shut down the interface)
# Verify the traffic continues via the remaining path

A clean validation: the routing table has multiple next-hops; the kernel’s hash is per-flow; the traffic distribution is roughly uniform; a path failure is detected and the traffic shifts.

Production failure modes

Only one path is used

The operator configured two paths but only one is in the routing table. Cause: the paths are not equal-cost.

Diagnostic: show ip route <prefix> shows one next-hop, not two. Compare the metrics of the two paths; align them.

Traffic clusters on one path

The operator has many flows but the throughput on one path is much higher than the others. Cause: hash clustering.

Diagnostic: capture the 5-tuples of the flows and compute the hash distribution manually. If the distribution is biased, the hash function may need a different policy (though this is rarely the issue; usually it’s flow diversity).

Asymmetric routing breaks stateful firewall

The forward path uses ECMP next-hop A; the reverse path uses ECMP next-hop B; the firewall in the middle drops the reverse packets because it has no state for them.

Diagnostic: tcpdump on both ECMP paths; verify the forward and reverse packets are on different paths. conntrack -L on the firewall to see the state table.

The fix: either align the hash (so both paths use the same firewall), or use a stateless firewall, or design the ECMP to keep the firewall on a single path.

Path failure not detected

One ECMP path fails (link down, peer session down) but the traffic continues to be hashed to it. The packets are lost silently.

Diagnostic: show ip route <prefix> should show the failed path removed. If it remains, BFD or interface-state tracking is needed.

The fix: enable BFD on the ECMP paths, or rely on the protocol’s failure detection (OSPF hello, BGP keepalive).

Rollback

# Capture the running configuration
show configuration commands | save /tmp/ecmp-$(date +%s).txt

# Compare
compare

# Remove the second next-hop (revert to single-path)
delete protocols static route 10.0.0.0/24 next-hop 10.0.0.2
commit

# Or remove the ECMP configuration entirely
delete protocols ospf area 0 ...
commit

The rollback reverts to single-path routing. The traffic returns to the original next-hop; throughput drops to the single-link capacity.

Production discipline

Cross-course references

• Part II-06 (II-VyOS-RoutingFund / ECMP intro) covers the basic concept of ECMP at the routing-table level.
• Part XXXVI-02 (XXXVI-VyOS-ECMP / config) covers the configuration of ECMP for static routes and BGP/OSPF.
• Part XXXVI-03 (XXXVI-VyOS-ECMP / BGP) covers the BGP maximum-paths and multipath-relax settings.
• Part XXXVI-05 (XXXVI-VyOS-ECMP / fwmark) covers policy-based ECMP with fwmark.
• Part XXXVI-06 (XXXVI-VyOS-ECMP / troubleshoot) covers the diagnostic method for ECMP failures.

Quiz