Per-class ECMP — policy routes, alternate tables, and the fwmark VyOS owns

Ordinary ECMP gives every destination one set of next-hops. Per-class ECMP gives different traffic classes different sets: voice across two circuits from one provider, bulk data across two from another, each class load-sharing within its own pair.

VyOS 1.5 builds this out of two pieces you have already met separately — a policy route that selects an alternate routing table, and a table holding several next-hops for the same prefix. What ties them together in the kernel is a firewall mark, and the useful thing to understand about that mark is that it is not yours.

The shape of the configuration

flowchart LR
  P["Packet in on eth1"]
  PR["policy route PBR-CLASSES<br/>rules matched in order"]
  T1["table 100<br/>two next-hops via ISP-A"]
  T2["table 200<br/>two next-hops via ISP-B"]
  M["table main<br/>everything else"]
  H1{"multipath hash"}
  H2{"multipath hash"}
  P --> PR
  PR -- "rule 20: udp/5060 → set table 100" --> T1
  PR -- "rule 30: guest subnet → set table 200" --> T2
  PR -- "no rule matched" --> M
  T1 --> H1
  T2 --> H2
  H1 --> A1["198.51.100.1"]
  H1 --> A2["198.51.100.5"]
  H2 --> B1["203.0.113.1"]
  H2 --> B2["203.0.113.5"]

Three parts, and all three are required. Two of them are the policy route you built in Part XIII; the third is what makes this lesson about ECMP rather than about PBR.

1. The tables, each an ECMP set

set protocols static table 100 route 0.0.0.0/0 next-hop 198.51.100.1
set protocols static table 100 route 0.0.0.0/0 next-hop 198.51.100.5

set protocols static table 200 route 0.0.0.0/0 next-hop 203.0.113.1
set protocols static table 200 route 0.0.0.0/0 next-hop 203.0.113.5

Two next-hops for one prefix in one table, at the same distance, is what makes the entry multipath. The distance is the whole condition: two next-hops at distance 1 and 2 are a primary and a floating backup, not an ECMP pair, and nothing in the output shouts about the difference. Leave the distance alone unless you mean to break the sharing.

2. The policy route that selects a table

set policy route PBR-CLASSES description 'per-class ECMP sets'

set policy route PBR-CLASSES rule 10 description 'internal traffic stays on main'
set policy route PBR-CLASSES rule 10 destination group network-group 'INTERNAL-NETS'
set policy route PBR-CLASSES rule 10 set table 'main'

set policy route PBR-CLASSES rule 20 description 'SIP signalling across the voice pair'
set policy route PBR-CLASSES rule 20 protocol 'udp'
set policy route PBR-CLASSES rule 20 destination port '5060'
set policy route PBR-CLASSES rule 20 set table '100'

set policy route PBR-CLASSES rule 30 description 'guest subnet across the bulk pair'
set policy route PBR-CLASSES rule 30 source address '192.168.100.0/24'
set policy route PBR-CLASSES rule 30 set table '200'

Rules are evaluated in numeric order and the first match wins, which is why rule 10 exists: without an explicit early rule sending internal traffic to main, a packet between two internal subnets that happens to be UDP/5060 would be sent out of a WAN circuit.

3. The attachment

set policy route PBR-CLASSES interface 'eth1'
commit
save

A policy route with no interface is configuration that never runs. It commits cleanly, appears in show configuration, and matches nothing, because nothing is feeding it packets. From VyOS 1.4 onward this attachment is a property of the policy rather than of the interface; the 1.3 form set interfaces ethernet eth1 policy route PBR-CLASSES is gone.

The fwmark, and why it is not yours

Underneath, VyOS renders that policy as an nftables rule that marks the packet and an ip rule that matches the mark and looks up the table.

vyos@vyos:~$ ip rule show

0:      from all lookup local
32764:  from all fwmark 0x7fffff37 lookup 200
32765:  from all fwmark 0x7fffff9b lookup 100
32766:  from all lookup main
32767:  from all lookup default

The mark is derived from the table id — on current images by subtracting it from 0x7fffffff — and both the mark and the rule priority are chosen by VyOS. Check the shape rather than the numbers: a selector for each table you configured, all of them sitting above from all lookup main at priority 32766. A selector below that line is unreachable, because from all lookup main matches everything.

The hash inside the table

Having reached table 100, a packet meets a multipath entry with two next-hops, and the kernel picks one by hashing. Which fields it hashes is not what most operators assume.

There is no per-packet mode in the IPv4 FIB. Every setting here is a per-flow hash; they differ only in what counts as a flow. A given combination of hashed fields always maps to the same next-hop, which is what keeps a TCP connection on one path.

For per-class ECMP the default is often exactly wrong. Take the voice example: SIP signalling from one PBX to one provider SBC is a single pair of addresses, so under policy 0 every SIP packet takes one next-hop and the second circuit in the pair carries none of it. The class was separated successfully and then pinned to half of the capacity it was given.

set system sysctl parameter net.ipv4.fib_multipath_hash_policy value 1
set system sysctl parameter net.ipv6.fib_multipath_hash_policy value 1
commit
save

IPv4 and IPv6 have separate knobs. Setting the sysctl through the configuration tree rather than with sysctl -w is what makes it survive a reboot and appear in the configuration archive.

Validation

Work outward from the policy, and finish with a real lookup rather than an inference.

# 1. Is the policy attached? This is the line most reviews miss.
show configuration commands | match 'policy route PBR-CLASSES'

# 2. Did the router install the selectors, above the main rule?
ip rule show

# 3. Are the tables populated, and are the entries multipath?
ip route show table 100
ip route show table 200

# 4. What did the rules actually match? The counters do not lie.
show firewall statistics
nft list ruleset

Then prove the path for one flow of each class:

DST=203.0.113.20
for SRC in 10.10.0.5 192.168.100.5; do
  ip route get "$DST" from "$SRC" iif eth1
done

ip route get runs a real lookup with the selectors you supply and reports the next-hop the kernel would use. Two sources that should take different classes returning the same next-hop is the answer, before anyone opens a ticket.

For the hash itself, generate several flows that differ only in port and watch where they leave:

tcpdump -ni eth2 -c 20 'host 203.0.113.20'
tcpdump -ni eth3 -c 20 'host 203.0.113.20'

Both interfaces carrying traffic means the hash is spreading. One interface carrying all of it, with a layer-3 hash policy and a single pair of addresses, is the expected outcome rather than a fault.

Failure modes

The policy has no interface attachment

Everything is configured, nothing happens, and no command reports an error. The policy is not in any packet’s path.

Diagnostic: show configuration commands | match 'policy route' and look for the interface line specifically. ip rule show will also be missing the selector, because VyOS installs it as part of the attachment.

The selected table has no matching route

The rule fires, the table is consulted, nothing matches — and the packet is not dropped. The routing policy database continues to the next rule, reaches from all lookup main, and the packet leaves by the default route.

This is the failure that wastes the most time, because every individual check looks healthy: the policy exists, the rule matched, the counters increment, and the traffic still goes the wrong way. The diagnosis is ip route show table 100 returning nothing.

Populate the table in the same change as the policy, not the one after.

The traffic never enters the policy

policy route acts on traffic arriving in through the attached interface. Two classes of traffic are therefore invisible to it:

• Traffic the router originates itself — a ping or an SNMP poll from the router does not arrive on any interface. It needs set policy local-route, which is a separate rule set with its own matches.
• Traffic arriving on an interface with no policy attached — including the return direction of a flow you carefully steered outbound. If the return path also needs steering, it needs its own attachment on the interface it arrives on.

Diagnostic: ip route get with iif naming the interface the traffic really arrives on, not the one you were thinking about.

The class is separated but pinned to one next-hop

The policy works, the table has both next-hops, and one circuit is idle. This is the hash, not the policy: a layer-3 hash policy with one pair of addresses has exactly one answer.

Diagnostic: cat /proc/sys/net/ipv4/fib_multipath_hash_policy. Fix by raising it to 1 through set system sysctl parameter, and re-test with flows that differ in port.

A rule ordering surprise

A packet matches an earlier, broader rule and never reaches the specific one written for it. First match wins, and rule numbers are the only ordering.

Diagnostic: read the rules in numeric order and find the first that matches your flow, rather than the one you intended. show firewall statistics shows which rules are actually counting.

A next-hop is down but still in the table

A static next-hop is withdrawn when it becomes unreachable, but “the far end is broken” is not the same as “the next-hop is unreachable” — a live link to a dead provider keeps its next-hop resolvable, and the hash keeps sending half the class into it.

Attaching BFD to the next-hops is what converts a far-side failure into a withdrawal the router can act on. Part XXXII covers the configuration; the point here is that ECMP has no health check of its own and will happily balance into a black hole.

Rollback

# Take the policy out of the path — the fastest, least destructive step
delete policy route PBR-CLASSES interface
commit-confirm 5

# Remove one class without touching the others
delete policy route PBR-CLASSES rule 20
commit-confirm 5

# Remove the whole policy and its tables
delete policy route PBR-CLASSES
delete protocols static table 100
delete protocols static table 200
commit

# Or return to a previous revision
rollback N
commit

Deleting the interface attachment is the surgical rollback: the rules and the tables stay in the configuration for inspection, and all traffic immediately falls back to the main table. That is usually the right first move during an incident, because it restores a known behaviour without discarding the evidence.

Use commit-confirm for anything that changes which path management traffic takes. A policy route that captures your own SSH session’s return path and sends it out a circuit that cannot reach you is recoverable only by the timer.

Production discipline

Cross-course references

• Part XIII (vyos-xiii-03-pbr-rules) covers the policy route tree, its attachment, and the ip rule render in detail.
• Part XIV (vyos-xiv-02-multiple-tables, vyos-xiv-04-table-source-bind) covers routing tables and the policy database.
• Part XXXVI (vyos-xxxvi-01-ecmp-concept, vyos-xxxvi-02-ecmp-config) covers the multipath entry and the hash in the main table.
• Part XXXIX (vyos-xxxix-04-wan-policy-routing) applies the same mechanism to multi-WAN, including the router-originated traffic case.
• Part XLV (vyos-xlv-03-classification-and-matching) covers the QoS classifier, which is the consumer a set mark action actually has.

Quiz