WAN load sharing — ECMP across two circuits, and what the hash sees

WAN load sharing is the active-active multi-WAN pattern: two working circuits, both carrying traffic, with the kernel choosing one per flow. The choice is a hash, so it is stable for the life of a connection and arbitrary across connections.

This lesson covers the configuration that produces it, what the hash actually consumes, where VyOS does and does not let you weight the split, and the failure modes that decide whether active-active is a good idea on your network at all.

Two default routes, equal cost

set protocols static route 0.0.0.0/0 next-hop 203.0.113.1
set protocols static route 0.0.0.0/0 next-hop 198.51.100.1

commit

That is the whole configuration. Two next-hops under one prefix, same administrative distance, and FRR installs a multipath route that the kernel then hashes across.

Two things about that block are worth stating because configurations copied from older material get them wrong. A static route next-hop on VyOS 1.5 takes disable, distance and bfd — and nothing else. There is no interface sub-node under next-hop, so next-hop 203.0.113.1 interface eth0 is not a longer, more explicit way of saying the same thing; it does not exist. When the next hop is genuinely reached over one interface with no usable next-hop address — a point-to-point or unnumbered link — the interface form is a route in its own right:

set protocols static route 0.0.0.0/0 interface pppoe0

The route is verified in the RIB, not in the configuration:

vyos@r1:~$ show ip route 0.0.0.0/0

Routing entry for 0.0.0.0/0
Known via "static", distance 1, metric 0, best
Last update 00:04:12 ago
* 203.0.113.1, via eth0, weight 1
* 198.51.100.1, via eth1, weight 1

One asterisk means one usable path and no load sharing at all, whatever the configuration says. That is the first thing to check when a freshly built active-active router sends everything out of one circuit.

flowchart LR
  F1["Flow 1<br/>5-tuple A"]
  F2["Flow 2<br/>5-tuple B"]
  F3["Flow 3<br/>5-tuple C"]
  F4["Flow 4<br/>5-tuple D"]
  HASH["Hash function<br/>(per-flow)"]
  W1["WAN 1<br/>(eth0)"]
  W2["WAN 2<br/>(eth1)"]

  F1 --> HASH
  F2 --> HASH
  F3 --> HASH
  F4 --> HASH
  HASH -->|"hash % 2 = 0"| W1
  HASH -->|"hash % 2 = 1"| W2

The hash input is the 5-tuple (or a subset). The hash output is a number; the next-hop is selected by the output modulo the number of next-hops. For two equal-cost routes, half the flows use WAN 1, half use WAN 2.

Unequal circuits: what VyOS does not give you here

The obvious next question — WAN 1 is a gigabit and WAN 2 is 100 Mbps, how do I split the traffic ten to one? — has an answer, and it is not a static route option.

There is no weight under set protocols static route on VyOS 1.5. The weight 1 in the show ip route output above is FRR reporting the kernel nexthop weight it used, which for a plain static multipath route is 1 on every path. It is a field you can read, not a field the CLI lets you set. A configuration that carries ... next-hop 203.0.113.1 weight 10 is not doing 10:1; it never committed.

Where weighting genuinely lives on VyOS is the WAN load-balancing subsystem, which is a different mechanism with a different data path — it marks connections and distributes them itself rather than relying on the routing table:

set load-balancing wan interface-health eth0 nexthop '203.0.113.1'
set load-balancing wan interface-health eth1 nexthop '198.51.100.1'

set load-balancing wan rule 10 inbound-interface 'eth1.100'
set load-balancing wan rule 10 interface eth0 weight '10'
set load-balancing wan rule 10 interface eth1 weight '1'

That is a real 10:1 split of new connections, and it comes with health checking of each circuit as part of the same subsystem. It is covered properly in the multi-WAN lessons that deal with failover; the point here is only that if you need weighting, you have chosen the load-balancing subsystem, not ECMP, and the two are alternatives rather than layers.

Verifying the load distribution

The operator verifies the load distribution with:

show ip route 0.0.0.0/0
show interfaces
show conntrack table ipv4

The route confirms both next-hops are installed. The interface counters are the only measurement that tracks what users experience. And the conntrack table is where the per-connection decision becomes visible — with one subtlety worth knowing, because it catches people out:

a conntrack entry does not record an interface. It records two tuples, original and reply. What tells you which WAN a connection took is the translated source address in the reply tuple, because masquerade rewrites it to the address of the interface the packet left by. So the count you want is a count of translated sources:

LAN=192.168.1.
sudo conntrack -L 2>/dev/null | grep "src=$LAN" | \
  grep -o 'dst=[0-9.]* ' | sort | uniq -c | sort -rn | head

The reply-direction dst= is your own WAN address, and the counts beside them are connections per circuit. Run it twice a minute apart on a busy router and you have a distribution rather than a snapshot.

For a live view of one class of traffic, watch both circuits at once and compare:

sudo tcpdump -ni eth0 -c 20 'tcp port 443'
sudo tcpdump -ni eth1 -c 20 'tcp port 443'

Note that there is no show ip fib command on VyOS — the FIB view is show ip route, where an asterisk marks the paths FRR has installed into the kernel, and ip route show if you want the kernel’s own answer with nothing in between.

What the hash actually consumes

Here is the correction that changes how you read an uneven split. Linux IPv4 multipath is always per-flow. There is no per-packet mode to turn on or off — the per-packet multipath cache was removed from the kernel long before any release VyOS is built on, precisely because it reordered TCP.

The knob that does exist chooses which fields feed the hash, and it is net.ipv4.fib_multipath_hash_policy:

• 0 — Layer 3: source and destination address only. This is the kernel default.
• 1 — Layer 4: the 5-tuple, so source and destination ports join the hash.

VyOS exposes the second as a configuration node rather than a sysctl to be poked:

set system ip multipath layer4-hashing

The difference matters more than it sounds. Under the L3 default, every connection between the same pair of hosts takes the same WAN. A site whose traffic is dominated by one busy destination — a cloud tenancy, a VPN concentrator, a single SaaS front end — will see a split that looks broken and is working exactly as designed. Add ports to the hash and those connections spread out.

Two caveats before turning it on everywhere. Fragmented packets have no ports to hash on after the first fragment, so L4 hashing can send fragments of one datagram down different paths; if your traffic is fragment-heavy, L3 is the safer choice. And the setting is local: it changes how this router distributes, and has no influence on what the far end does with the return traffic. That asymmetry is the next section.

The asymmetric routing problem

For active-active multi-WAN with ECMP, the asymmetric routing problem is fundamental: the upstream router may hash the return traffic to a different WAN than the forward traffic.

The conntrack entry is created on the forward path; the return packet has no conntrack entry on its path. The stateful firewall drops the return packet.

For traffic your users originate, this is mostly a non-problem: the far end replies to the address it saw, masquerade rewrote that address to the WAN the connection left by, and the reply comes back the same way. Egress active-active works.

It is inbound traffic that has no such guarantee, and the patterns that fix it are all about pinning the return path:

1. Mark the connection and route on the mark. On VyOS 1.5 the policy-route rule carries set connection-mark 100 and set table 100, the policy is attached to the ingress interface with set policy route PBR-WAN1 interface eth1.10, and the alternate table is populated with set protocols static table 100 route 0.0.0.0/0 next-hop 203.0.113.1. The mark lives on the conntrack entry, so it is available on the reply direction — which is the entire reason to use set connection-mark rather than set mark.
2. Own the address space and speak BGP. Advertise the same prefix to both providers and let the Internet decide. Then there is no asymmetry to manage, because your address does not change with the circuit.
3. Publish one circuit. The unglamorous answer that most branch sites should take: inbound services live behind one WAN’s address, and the second circuit is for egress and for failover with a DNS or dynamic-DNS change. Honest, cheap, and it does not pretend.

The discipline: decide the inbound story explicitly before turning on the second circuit. Egress load sharing does not provide one, and discovering that during an incident is expensive.

Production failure modes

The five production failure modes the operator must recognise:

1. An uneven split with healthy circuits. Almost always the L3 hash meeting a concentrated destination set. Confirm by counting translated sources, then set system ip multipath layer4-hashing. Reach for weights only after the hash input is right, and remember weights mean leaving ECMP for the load-balancing subsystem.
2. Everything on one WAN. show ip route 0.0.0.0/0 shows one asterisk. The second next-hop is not installed — usually because its next-hop address does not resolve, or the interface is down and nobody noticed because traffic still flows.
3. Asymmetric routing. The far end returns traffic on the other circuit. Your own conntrack has no entry for it and the firewall drops it. This is the failure that makes plain ECMP unsuitable for anything that must be reachable inbound, and no amount of hashing tuning changes it — see the next section.
4. A flow that dies at failover. A circuit drops, the route is withdrawn, surviving connections rehash onto the other WAN — and their masqueraded source address changes with it, so the far end sees packets from a new address on an established connection and resets them. Long-lived sessions do not survive a WAN change under NAT. Design around it; do not promise otherwise.
5. Stale conntrack after a change. Entries created before a routing change still carry the old translation. sudo conntrack -F clears the table, at the cost of killing every established connection through the box — which is a maintenance-window action, not a diagnostic step.

Operational commands

The load sharing state is exercised through the standard VyOS operational commands:

show ip route 0.0.0.0/0
show interfaces
show conntrack table ipv4
show configuration commands | match 'system ip multipath'

And, when the question is what the kernel is doing rather than what VyOS was told:

ip route show default
sysctl net.ipv4.fib_multipath_hash_policy

Rollback

The load sharing changes are rolled back the same way as any VyOS configuration change:

# Show the candidate diff
compare

# Remove the second path, leaving a single default route
delete protocols static route 0.0.0.0/0 next-hop 198.51.100.1
commit
save

Deleting one next-hop is the surgical revert, and it is usually the right one during an incident: it takes effect immediately, and existing connections that were hashed to the surviving path are undisturbed.

sudo conntrack -F exists and is sometimes necessary after a change that invalidates every translation — but it drops every tracked connection through the router, which is an outage, not a cleanup step. Flush deliberately and say so in the change record; do not make it a reflex after every commit.

Production discipline

Cross-course references

• Part XXXVI-01 (XXXVI-VyOS-ECMP / ECMP concept) covers the ECMP throughput scaling.
• Part XXXVI-02 (XXXVI-VyOS-ECMP / ECMP config) covers the ECMP configuration for static routes.
• Part XXXIX-01 (XXXIX-VyOS-MultiWAN / concept) covers the multi-WAN design space.
• Part XXXIX-04 (XXXIX-VyOS-MultiWAN / policy routing) covers the policy routing pattern.

Quiz