Source-based routing — binding traffic to a table by source

Source-based routing is the most common production use of multiple routing tables. The operator has two upstream ISPs, each serving a different LAN segment. Traffic from LAN-A should leave via ISP-A; traffic from LAN-B should leave via ISP-B. Destination-based routing cannot express this — the destination does not differ between the two flows. Source-based routing can: the kernel’s RPDB walks rules in priority order, and a rule with from <src> selects a different table for traffic from that source. This lesson covers the configuration idiom, the multi- homing use case that motivates it, the return-path discipline that prevents asymmetric routing, and the validation commands that prove the binding is live.

The mechanism

flowchart LR
  A[Packet from 10.10.0.0/24] --> B[RPDB walk]
  B --> P0[Priority 0: local]
  P0 --> P1[Priority 100: from 10.10.0.0/24 lookup isp-a]
  P1 -->|hit| T1[Table isp-a]
  T1 -->|via ISP-A| F1[Egress eth0]
  B2[Packet from 10.20.0.0/24] --> B
  B --> P2[Priority 200: from 10.20.0.0/24 lookup isp-b]
  P2 -->|hit| T2[Table isp-b]
  T2 -->|via ISP-B| F2[Egress eth1]

The mechanism is two ip rule entries and two routing tables. The RPDB walks rules in priority order; for each rule, the selectors are evaluated against the packet; if the rule matches, the kernel performs a FIB lookup in the rule’s named table. The first matching rule wins.

A source-based rule has the form:

ip rule add from <src> table <name|id> priority <p>

The VyOS 1.5 LTS configuration tree equivalent:

set system ipv4 rule <priority> from <src> table <name>

The <name> is the alias declared in set system ip rt-table. The VyOS validator checks the alias exists at commit time. A rule that references an undeclared name is rejected.

The multi-homing use case

The canonical use case for source-based routing is a site with two ISPs, each serving a different LAN. The site wants traffic from LAN-A to always leave via ISP-A and traffic from LAN-B to always leave via ISP-B. The destination is the same (any host on the Internet); the source differs.

flowchart TB
  subgraph Site[Site network]
    LA[LAN-A: 10.10.0.0/24]
    LB[LAN-B: 10.20.0.0/24]
  end
  subgraph Router[VyOS router]
    ETH0[eth0 -> ISP-A, 198.51.100.2/24]
    ETH1[eth1 -> ISP-B, 203.0.113.2/24]
  end
  LA --> Router
  LB --> Router
  Router -->|ISP-A default| WAN1[ISP-A upstream]
  Router -->|ISP-B default| WAN2[ISP-B upstream]

The configuration:

[edit]
vyos@vyos# set system ip rt-table rt-table-id 100 name isp-a description "ISP-A traffic table"
vyos@vyos# set system ip rt-table rt-table-id 200 name isp-b description "ISP-B traffic table"
[edit]
vyos@vyos# set system ipv4 rule 100 from 10.10.0.0/24 table isp-a
vyos@vyos# set system ipv4 rule 200 from 10.20.0.0/24 table isp-b
[edit]
vyos@vyos# set protocols static table isp-a route 0.0.0.0/0 next-hop 198.51.100.1
vyos@vyos# set protocols static table isp-b route 0.0.0.0/0 next-hop 203.0.113.1
[edit]
vyos@vyos# commit
[edit]
vyos@vyos# save

The four pieces:

1. Table declaration. Two rt-table entries at IDs 100 and 200, with names isp-a and isp-b.
2. Rules. Two rules at priorities 100 and 200, each selecting the corresponding table for traffic from the corresponding source.
3. Routes. Two default routes, one in each table, with different next-hops (the ISP gateways).
4. Connected routes. The interfaces eth0 and eth1 are configured with addresses in 198.51.100.0/24 and 203.0.113.0/24 respectively. The connected routes go in main by default; the kernel uses them to resolve the next-hop.

The main table has no default route — all outbound traffic is sourced from a LAN, and every LAN source has a rule that selects a custom table. Traffic that does not match either rule (e.g. traffic sourced from the router itself) consults main and finds no default; the kernel returns ICMP unreachable.

Return-path discipline

Source-based routing on the egress side is only half the problem. The return path must also be deterministic, or the connection will fail because of asymmetric routing.

flowchart LR
  A[Host 10.10.0.5] -->|forward via ISP-A| B[Server 8.8.8.8]
  B -->|return via ISP-B| A
  C[Firewall on ISP-A path] -.->|sees forward, not return| X[Connection broken]

If the host sends a packet via ISP-A but the return packet arrives via ISP-B, any stateful device on either path sees only one direction of the flow. The connection breaks.

The fix is return-path symmetry. There are three common strategies:

1. BGP with the upstream. Run BGP with both ISPs. The upstream announces the site’s prefix; both ISPs have a route back. The return path is determined by the upstream’s routing, which is typically symmetric.
2. Source-address binding. The host uses a source address that the return path can predict. With NAT, the router’s source address is the outgoing interface address. The return path uses the same interface.
3. Static return routes on the upstream. The operator negotiates with the upstream ISP to route the site’s source addresses via the same ISP the site uses for egress. This is rarely possible without BGP.

For the source-based routing deployment to work, the operator must have a return-path story. The most common production deployment uses BGP with both ISPs — the upstream’s BGP table provides the symmetric return path.

Validation

The validation command set for a source-based routing deployment:

vyos@vyos:~$ ip rule show
0:	from all lookup local
100:	from 10.10.0.0/24 lookup isp-a
200:	from 10.20.0.0/24 lookup isp-b
32766:	from all lookup main
32767:	from all lookup default

vyos@vyos:~$ ip route show table isp-a
default via 198.51.100.1 dev eth0 proto static metric 1
198.51.100.0/24 dev eth0 proto kernel scope link src 198.51.100.2

vyos@vyos:~$ ip route show table isp-b
default via 203.0.113.1 dev eth1 proto kernel scope link src 203.0.113.2

vyos@vyos:~$ ip route get 8.8.8.8 from 10.10.0.5
8.8.8.8 from 10.10.0.5 via 198.51.100.1 dev eth0 table isp-a uid 0
    cache

vyos@vyos:~$ ip route get 8.8.8.8 from 10.20.0.5
8.8.8.8 from 10.20.0.5 via 203.0.113.1 dev eth1 table isp-b uid 0
    cache

The ip route get <dst> from <src> command is the canonical validation. It returns the table, the next-hop, and the egress interface for the specified source and destination. The output must match the operator’s design: traffic from LAN-A consults isp-a, traffic from LAN-B consults isp-b.

A traceroute -s <src> <dst> from a host on the corresponding LAN validates the forward path through the Internet:

host-a:~$ traceroute -s 10.10.0.5 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max
  1  198.51.100.1 (ISP-A gateway)  0.456 ms
  2  ...
  3  8.8.8.8

The first hop must be the ISP-A gateway. If the first hop is the ISP-B gateway, the source-based routing is not working — either the rule is wrong or the route is in the wrong table.

How it fails

The production failure modes the engineer must recognise:

• Rule with wrong source. A rule at priority 100 with from 10.10.0.0/24 that should have been from 10.10.0.0/16. Traffic from 10.10.1.0/24 matches; traffic from 10.10.2.0/24 does not. The operator sees inconsistent behaviour.
• Table with no routes. The rule selects table isp-a; the table has no default. The lookup misses; the walk falls through to main; main has no default either (by design). The packet is dropped with ICMP unreachable.
• Wrong next-hop. The default in isp-a points at the ISP-B gateway. Traffic from LAN-A goes to ISP-B. The operator sees traffic leaving on the wrong interface.
• Missing return-path story. Source-based egress without symmetric return. The forward path works; the return path is routed via a different ISP. Stateful devices on the path break the connection.
• Router-originated traffic not handled. BGP sessions, monitoring probes, and NTP queries originate from the router itself. They consult main; main has no default; they fail.
• Rule priority wrong. A rule at 32768 instead of 100. The rule is consulted after main. Traffic from the LAN hits main first; main has no default; the rule is never reached.

Rollback

The recovery from a source-based routing misconfiguration:

• Wrong source selector: delete system ipv4 rule <priority>; set system ipv4 rule <priority> from <correct>; commit; save.
• Wrong table: delete system ipv4 rule <priority>; set system ipv4 rule <priority> ... table <correct>; commit; save.
• Wrong route in custom table: delete protocols static table <name> route <prefix>; commit; save.
• Whole-tree rollback: rollback N; commit; save.

For emergency rollback, load <file>; commit; save replaces the candidate with a saved backup. For a partial rollback that restores only one direction, the operator must remove the broken rule and the matching route together — keeping one without the other creates a half-configured deployment.

Production discipline

Cross-course references

The Linux course’s XXI-Linux-NetAdvanced covers source-based routing from a host perspective (the same RPDB, the same rule format). The PBR course’s XIII-VyOS-PBR covers policy routing more broadly — source-based routing is one tool in the PBR toolbox. The BGP course’s XXV-BGP-Advertisement covers the return-path story: how to advertise prefixes such that the upstream’s return path matches the egress.

Quiz