ECMP troubleshooting — only one path used, asymmetric traffic, next-hop unreachable

An ECMP failure is not a request to remove a next-hop. It is a request to identify which contract failed: the costs are unequal (only one path installed), maximum-paths is missing (only one path installed), the hash clusters (one path dominates), the forward and reverse paths diverge (stateful firewall drops), or a path failure is not detected (silent blackhole).

This lesson is the operator’s troubleshooting reference for Part XXXVI: the five-layer diagnostic method, the ECMP- specific failure modes, and the rollback discipline that prevents the troubleshooting from making the outage worse.

The five-layer diagnostic method

An ECMP failure can be in any of five layers:

1. Configuration — Is maximum-paths configured? Are the next-hop statements in place? Is multipath-relax configured (BGP)? Is the cost aligned (OSPF)?
2. Path eligibility — Are the paths equal-cost? Does the best-path algorithm (BGP) or SPF (OSPF) consider them equal?
3. Kernel state — Is the hash policy per-flow? Are the routes installed in the kernel’s FIB?
4. Forwarding — Do the paths actually carry traffic? Is the next-hop reachable? Is the path failure detected?
5. Asymmetry — Do the forward and reverse paths agree? Do stateful firewalls see the same flow on the same interface?

flowchart TB
  S["Symptom<br/>(one path, asymmetry, etc.)"]
  C["Configuration check"]
  PE["Path eligibility check"]
  KS["Kernel state check"]
  F["Forwarding check"]
  AS["Asymmetry check"]
  D["Diagnosis"]
  S --> C
  C --> PE
  PE --> KS
  KS --> F
  F --> AS
  AS --> D

The operator starts with the configuration (the cheapest check) and walks through the layers. The first layer where the evidence diverges from the expected state is the failure boundary.

The first three commands

For an ECMP failure:

show ip route <prefix>
show ip fib
show configuration commands | match "maximum-paths\|next-hop\|cost\|multipath"

The three together identify whether the routes are installed, whether the kernel’s FIB has them, and whether the configuration is correct.

For a BGP ECMP failure:

show ip bgp <prefix>
show ip bgp regexp <pattern>
show configuration commands | match "bgp.*maximum-paths\|multipath"

For an OSPF ECMP failure:

show ip ospf database
show ip ospf interface
show configuration commands | match "ospf.*maximum-paths\|ospf.*cost"

For a fwmark ECMP failure:

show ip rule
ip route show table <N>
nft list ruleset | grep mark

Only one path used

The operator configured ECMP but the routing table shows only one next-hop. The five-layer diagnostic:

Layer 1: Configuration

show configuration commands | match "maximum-paths\|next-hop\|multipath"

Verify:

• maximum-paths is configured (BGP, OSPF).
• Multiple next-hop statements exist (static routes).
• multipath-relax is configured (BGP, if AS-paths differ).

Layer 2: Path eligibility

For BGP, the best-path tie-break:

show ip bgp <prefix>
# Compare: weight, local-pref, AS-path length, origin, MED, IGP cost.

For OSPF, the cost:

show ip ospf interface
# Compare the costs on each ECMP path.

Layer 3: Kernel state

ip route show <prefix>
# Should show multiple nexthop lines

cat /proc/sys/net/ipv4/fib_multipath_hash_policy
# Should be 1 (per-flow)

Layer 4: Forwarding

iperf3 -c <dest> -P 10
# 10 parallel streams; both ECMP paths should see traffic

If only one path sees traffic, the hash may be clustering (few flows) or the path may be unreachable.

Layer 5: Asymmetry

# Trace forward and reverse paths
tcpdump -ni <intf> host <src> and <dest>
# Verify both directions are present

Asymmetric traffic breaking stateful firewall

The forward path uses ECMP next-hop A; the reverse path uses ECMP next-hop B; the stateful firewall in the middle drops the reverse packets.

Diagnostic:

# 1. Capture packets on both ECMP paths
tcpdump -ni <intf-A> host <src> and <dest>
tcpdump -ni <intf-B> host <src> and <dest>

# 2. Verify the firewall state
conntrack -L | grep <flow-5-tuple>

# 3. Determine why the reverse path differs
# (asymmetric routing is normal for ECMP; the question is
# whether the firewall tolerates it)

The fix depends on the network design:

• Align the hash so forward and reverse paths use the same ECMP next-hop. This requires both routers (forward and reverse) to hash the flow to the same next-hop. The hash inputs are the same (5-tuple); the asymmetry must be due to different next-hop sets.
• Use a stateless firewall (rare; not recommended for most production networks).
• Design the ECMP so the firewall is on a single path: the firewall is the ECMP next-hop itself, and both ECMP paths converge at the firewall.

Next-hop unreachable after a failure

A path fails (link down, peer session down) but the routing table still shows it as a next-hop. The kernel continues to hash some flows to the failed path. The packets are lost silently.

Diagnostic:

# 1. Verify the path is actually down
show ip route <prefix>
# Look for the failed next-hop — it should be removed; if not,
# the failure detection is not working

# 2. Check BFD
show bfd peers
# The BFD peer for the failed path should be Down; if Up, BFD
# is not detecting the failure

# 3. Check protocol-level failure detection
show ip ospf neighbor
show ip bgp summary
# The neighbour for the failed path should be Down; if not,
# the protocol's hello/keepalive has not detected the failure

The fix:

• Enable BFD for fast failure detection: set protocols ospf interface eth0 bfd or set protocols bgp <asn> neighbor <ip> bfd.
• Verify the protocol’s hello/keepalive is configured with appropriate timers.
• Add interface-state tracking if the link failure is not propagated to the routing protocol.

Production failure modes summary

The ECMP-specific failure modes:

• Only one path used — maximum-paths not configured or costs unequal.
• Traffic clusters on one path — hash clustering due to few flows; verify with iperf3 with many parallel streams.
• Asymmetric routing breaks stateful firewall — forward and reverse paths diverge; the firewall drops the reverse packets.
• Path failure not detected — BFD not enabled; the protocol’s hello/keepalive is too slow.
• fwmark-based ECMP empty table — the routing table has no routes; packets are dropped.
• fwmark collision — two nftables rules set the same fwmark for different traffic classes.
• Asymmetric marking — fwmark is set on the forward path but not the reverse path.

Rollback discipline

The ECMP changes are typically small but the impact is broad. The rollback discipline:

• For static routes — delete protocols static route <prefix> next-hop <nh> and commit. The traffic returns to the remaining next-hops.
• For BGP — delete protocols bgp <asn> maximum-paths or delete protocols bgp <asn> bgp bestpath multipath-relax and commit. The traffic returns to the single best path.
• For OSPF — delete protocols ospf parameters maximum-paths or delete protocols ospf interface <ifname> cost and commit. The traffic returns to the lower-cost path.
• For fwmark — delete firewall ipv4 name <name> or delete policy route-table <name> and commit. The traffic returns to the main table.

For all changes, use commit-confirm:

configure
# ... make the change ...
commit-confirm 5
# If the change has unintended consequences, the auto-rollback
# fires after 5 minutes and the previous configuration is restored.

Production discipline

Cross-course references

• Part XXXVI-01 through XXXVI-05 cover the conceptual, configuration, BGP, OSPF, and fwmark primitives that this lesson assumes.
• Part XXXII (XXXII-VyOS-BFD) covers BFD for fast failure detection.
• Part LII (LII-VyOS-Troubleshoot) covers the general troubleshooting methodology that applies across all subsystems.

Quiz