ECMP Unexpectedly Changes Traffic Path · Break/Fix · VyOS

Reported symptoms

edge3 fronts a site with two WAN uplinks, ISP-A on eth0 and ISP-B on eth1, carrying two static default routes at equal distance — an ECMP pair. Each WAN has its own masquerade rule, so a flow’s public source address is whichever interface it leaves by. A stateful firewall sits in front of both.

A capacity project added a third circuit. The change was written up as additive and low risk: one more default route, same distance, nothing removed. It went in at 20:15.

Within the same second, a large fraction of established sessions across the entire site dropped. Not the sessions that would have used the new circuit — the helpdesk logged about a minute of “everything froze”, from users doing work that had nothing to do with ISP-C and could not have.

Three more reports followed, and each was filed as its own problem:

Two partner sites saw the source address of traffic from this site change mid-session. One partner’s allowlist rejected calls for roughly ninety seconds.
The monitoring probe that traceroutes an external target every minute has returned a different hop set on almost every run since 20:15. That reads as path instability, and it is still doing it.
ISP-C is carrying nowhere near a third of egress bytes. The capacity project appears to have failed on top of everything else.

By 20:22, when the first engineer logged in, the router looked perfect. Three next-hops installed and selected, every interface up, no drops, sessions re-established. The change had been additive, so nobody on the bridge accepted that it was responsible.

Evidence provided

Read-only / Safethree members, equal weight — this is the input that changed

vyos@edge3:~$ ip route show 0.0.0.0/0

default proto static metric 20
nexthop via 203.0.113.1 dev eth0 weight 1
nexthop via 198.51.100.1 dev eth1 weight 1
nexthop via 192.0.2.1 dev eth2 weight 1

Illustrative output

Read-only / Safethe timestamp matches the helpdesk calls to the second

vyos@edge3:~$ show system commit | head -3

0   2026-08-18 20:15:07 by admin via cli   add ISP-C default (additive, low risk)
1   2026-08-14 10:31:44 by admin via cli   dhcp pool extension for floor 3
2   2026-08-11 16:02:20 by admin via cli   snmp acl update

Illustrative output

Read-only / Safein a source-NATed entry the translated address is the reply tuple's destination— flows are split almost exactly in thirds across the three WAN addresses

vyos@edge3:~$ sudo conntrack -L --src-nat | grep -oE 'dst=(192.0.2|198.51.100|203.0.113)[0-9.]+' | sort | uniq -c

   1174 dst=192.0.2.20
 1206 dst=198.51.100.20
 1189 dst=203.0.113.20

Illustrative output

Read-only / Safesame target, same host, two different first hops, seconds apart

vyos@edge3:~$ traceroute -n 198.18.0.1; traceroute -n 198.18.0.1

traceroute to 198.18.0.1 (198.18.0.1), 30 hops max, 60 byte packets
1  192.0.2.1  1.702 ms  1.688 ms  1.694 ms
traceroute to 198.18.0.1 (198.18.0.1), 30 hops max, 60 byte packets
1  203.0.113.1  1.221 ms  1.402 ms  1.198 ms

Illustrative output

Work the evidence before reading on

There are four complaints on the ticket. Before reading on, decide how many of them are faults.

The change added a member to an ECMP group. Where on the router is the record that flow X was using ISP-A before the change? If you cannot name the place it is stored, what happens to that flow when the group changes?
Two thirds of sessions dropped, not one third. What arithmetic produces two thirds when you go from two members to three, and what does that say about which flows were affected?
A dropped session is not a routing failure by itself — a packet leaving by a different WAN still reaches the internet. What else is configured on this router that makes “different WAN” mean “different flow” to the far end?
Traceroute returns a different first hop on consecutive runs. What does traceroute vary between probes, and is that one flow or many?

Then the question that separates the fault from the noise: which of these four symptoms would still be reported if the site did no source NAT at all?

Root cause

1. Linux ECMP is stateless, so there is no flow-to-path memory

Nothing on this router remembers which path a flow was using. There is no flow table for forwarding, no per-session pin, nothing to consult. For every packet, the kernel hashes the tuple and selects a member of the group from the result.

Flow affinity — all packets of a TCP session leaving by the same WAN — is real, but it is not a stored decision. It is an emergent property of feeding identical inputs into a deterministic function and getting an identical answer. The session stays on one path because its tuple does not change.

2. The member set is one of those inputs, so membership changes remap flows

The selection is a function of the hash and the current set of members. Change the set and you change the answer for tuples that were untouched.

The scale is the part that gets underestimated. Selecting a member from a hash is a modulo-style operation over the group, so going from two members to three changes the result for about two thirds of hash values. Only by coincidence does a given flow keep its previous path. “Additive” describes the configuration diff; it does not describe the effect, and there is no additive change to the membership of an ECMP group.

This is the behaviour named in the title. The traffic path changed unexpectedly — for flows that had nothing to do with the circuit that was added, which is why the correlation with the change was rejected on the bridge.

3. Per-WAN source NAT turns a path change into a session break

A remapped flow on its own is survivable: another WAN still reaches the internet. What makes it fatal here is what the router does to the packet on the way out.

Each WAN has its own masquerade rule, so the source address is whichever interface the packet leaves by. When a flow is remapped mid-session it is translated to a different public address, the far end sees traffic from an address it has no session for, and the return traffic comes back to a provider whose firewall holds no conntrack entry for it. The state check fails, the packet is discarded, and the session resets. That is the invalid-state spike in the firewall counters, and it is why it ended after two minutes: everything had finished reconnecting on its new path.

The partner allowlist rejection is the same mechanism seen from outside. The site’s traffic arrived from an address the partner had never been told about.

4. Two of the four complaints are not faults

Traceroute varying between runs. Traceroute sends a new destination port on each probe, so every probe is a distinct flow that per-flow ECMP is entitled to hash somewhere else. The output is correct behaviour rendered by a tool that cannot express the question being asked of it. It was doing this before the change too; nobody was reading it before the change.

ISP-C carrying a small share of bytes. The hash distributes flows, not bytes. The conntrack evidence shows flows split almost exactly in thirds, which is the group working as designed. A handful of large transfers happened to hash to the other two paths, and each is pinned to its path for its lifetime, so the byte counters are lopsided while the flow counters are not.

Under the hood

Two of the diagnostic commands here answer different questions, and reaching for the wrong one is what produced two of the four complaints.

traceroute sends probes. Under its default UDP mode it increments the destination port for each one, which is how it distinguishes replies — and which makes each probe a different 5-tuple. Against a single path that is invisible. Against ECMP it means consecutive probes are separate flows that can be hashed separately, so a stable multipath group renders as a flapping path. Traceroute cannot tell you which way a session goes, because the probes are not that session.

conntrack -L reads the connection-tracking table: real flows, with their real tuples and their real translated addresses. Grouping it by translated source address gives the actual distribution of sessions across the WANs, which is the thing the hash is responsible for. If you need the path of one specific session, filter conntrack on its exact tuple, or capture on each candidate egress with that tuple as the filter.

The general rule: to observe a hash you must observe the population it acts on. A tool that manufactures a new flow per sample is measuring the hash’s distribution, not any individual flow’s path — and reporting the result as instability inverts the meaning entirely.

'Additive' is a property of the diff, not of the effect

The change record said additive, and it was accurate about the configuration: one line added, nothing removed, no existing next-hop touched. That framing put it in the low-risk category, which is why it went in at 20:15 on a weekday evening with no notification.

The category was wrong because the risk did not live in the lines that changed. It lived in the fact that a hashed group has no stable identity for its members — the group as a whole is the unit, and every flow using it is a function of the whole.

This pattern is worth recognising by shape rather than by technology. Anywhere a system distributes work by hashing across a set of members, adding a member redistributes existing work: shard maps, consistent-hash caches, load-balancer backend pools, ECMP groups. Some implementations pay to avoid it — consistent hashing and the kernel’s resilient nexthop groups exist for exactly this reason — and where that machinery is not in play, the default behaviour is a global reshuffle. The question to ask of any “we are just adding capacity” change is whether the thing being added is a member of a hashed set.

Resolution

Do not roll back reflexively. Removing the third next-hop changes the member set again, remaps most flows again, and produces a second wave of resets identical to the first. An undo here costs exactly what the original change cost.
Confirm the site has actually recovered before deciding anything: the invalid-state drop counter back at baseline and flat, and conntrack showing a stable population of established flows rather than continuous new ones. If the disruption is over, the expensive part of this incident is behind you.
Default to holding. Keep ISP-C, record that the one-off disruption has been paid, and reclassify ECMP membership as a disruptive change. Hold is a decision, so it takes a named owner and a review time — otherwise it is just nobody doing anything.
Remove the path only if the circuit itself is faulty. Then schedule it, tell service owners to expect session resets, and do it in a window, using commit-confirm so a mistake on a live path backs itself out.
Fix the coupling rather than the hash. Move the two partner integrations off ECMP into a policy-routed table with a single default, so no future WAN change can move them: set protocols static table 100 route 0.0.0.0/0 next-hop 203.0.113.1 interface eth0, then a route-map matching those destinations bound to the ingress interface with set interfaces ethernet eth1 policy route-map RM-PARTNER-PIN.
Tell the two partners what happened and confirm which address they should have allowlisted, since one of them has already tightened their list against an address the site no longer reliably uses.
Close the traceroute complaint explicitly rather than leaving it open. The monitoring probe is not detecting instability; it is detecting ECMP. Point it at a per-flow measurement or accept its output as expected.

Verification

Recovery is real. The firewall invalid-state counter is at baseline and flat across several minutes, not merely lower than it was during the spike.
The group is distributing as designed. Group sudo conntrack -L --src-nat by the reply-direction destination and confirm the three WANs hold comparable numbers of flows. Flow count is the measure; byte count is not, and treating it as one is what turned expected behaviour into a fourth ticket.
The byte imbalance has an explanation, not a fix. If flow counts are even and bytes are not, the hash is working and a few large transfers are responsible. Only skewed flow counts justify investigating the hash.
The pinned traffic is genuinely pinned. Sample conntrack entries for the partner integrations and confirm every one shows the same translated source address. A policy route that has never been observed to constrain anything has not been tested.
The partner call works from the documented address. This is the only check that does not depend on the router that caused the incident.
The decision is recorded. Whether you held or scheduled a removal, the owner and the review time exist somewhere other than the incident channel.
No verification step in this list uses traceroute. If one of yours does, it is measuring the tool rather than the network.

Prevention

Reclassify ECMP membership changes as session-resetting, alongside reboots and failovers. They get a window, an owner and a notification. “Additive” earns no reduction in category, because adding a member disturbs every flow in the group.
Pin address-dependent traffic off ECMP for good. Anything carrying a partner allowlist, a licence check or a mail reputation on its source address belongs in a policy-routed table with one default, where the hash cannot reach it. The cost is a little configuration; the benefit is that WAN changes stop being customer-visible for that traffic.
Baseline the flow distribution across the WANs while things are normal, so a future “the new link is not taking its share” is compared against a known number rather than against the assumption that flows and bytes distribute alike.
Keep an inventory of everything that depends on the estate’s public source addresses, and read it before any WAN change. That inventory is the difference between an outage and a notification sent in advance.
Where the design permits it, remove the coupling entirely: a site that announces its own address space and does not source-NAT per WAN can move a flow between providers without changing what the far end sees. That is a redesign rather than a remediation, but it is the reason larger sites do not have this failure mode.

What if the third circuit had been added as a floating backup instead?

Nothing would have moved, and the contrast is the whole lesson.

A default route at a higher administrative distance is not a member of the ECMP group. It sits in the RIB as a candidate that loses, contributes nothing to the hash, and changes no existing flow’s path. When both primary paths fail it is selected and traffic moves — disruptively, but at a moment when the traffic was already broken, so the disruption is free.

The same circuit, same cable, same provider, produces either a site-wide session reset or no observable event at all, depending on one keyword in one line of configuration. That is worth internalising: with ECMP the question “is this change disruptive” is not answered by what the change touches, but by whether it alters the membership of a hashed set.

It also implies the honest way to grow ECMP capacity. If the extra bandwidth is needed, joining the group is correct and the reshuffle is the price. Book the window, notify the owners, and pay it once — deliberately, at a time you chose, rather than at 20:15 on a Tuesday because the change record said additive.

Why this matters

Four symptoms were reported and two of them were the system working correctly. An investigation that treats all four as faults will look for a cause that explains a byte imbalance and a flapping traceroute, and no such cause exists — so it will keep looking, and the real finding, that an additive change reset most of the site, will stay unexplained because it does not fit the shape of what is being searched for.

The mechanism worth carrying away is that stateless load distribution has no memory, and anything with no memory recomputes its answer from its current inputs every time you ask. The member set is an input. That single sentence predicts the blast radius of every ECMP change you will ever make, and it generalises without modification to shard maps, cache rings and load-balancer pools.

The second half is about measurement. Traceroute looked like the right tool and was structurally incapable of answering the question, because it creates a new flow for each sample of a per-flow system. Choosing a tool whose granularity matches the mechanism is not a detail; here it decides whether the evidence you collect describes the network or describes your instrument.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. Linux ECMP is stateless, so there is no flow-to-path memory

2. The member set is one of those inputs, so membership changes remap flows

3. Per-WAN source NAT turns a path change into a session break

4. Two of the four complaints are not faults

Resolution

Verification

Prevention