Policy routing - choosing routes by source, mark, or interface

Destination routing (the default) picks a route based on the destination IP. Policy routing picks a route based on any combination of source IP, incoming interface, packet mark, or other criteria. Policy routing is the tool for multi-homing, tenant isolation, and traffic steering.

Rule selectors

ip rule add accepts many selectors:

Selectors can be combined:

ip rule add from 10.0.0.0/24 iif eth0 lookup 100 priority 100

A multi-uplink example

A host has two uplinks:

• Uplink A: eth0, 10.0.0.10/24, default route via 10.0.0.1.
• Uplink B: eth1, 192.168.1.10/24, default route via 192.168.1.1.

You want:

• Replies to traffic from Uplink A to go out Uplink A.
• Replies to traffic from Uplink B to go out Uplink B.
• New connections to use Uplink A by default.

This is the “two default routes” problem. The solution is policy routing:

echo "100 backup" >> /etc/iproute2/rt_tables

ip route add 192.168.1.0/24 dev eth1 src 192.168.1.10 table backup
ip route add default via 192.168.1.1 dev eth1 table backup

ip rule add from 192.168.1.10 lookup backup priority 100

ip route add default via 10.0.0.1 dev eth0 metric 100

Verify:

ip route get 8.8.8.8 from 10.0.0.10
ip route get 8.8.8.8 from 192.168.1.10

The two queries should pick different routes.

Mark-based routing

A packet mark is a 32-bit value attached to a packet in the netfilter hook. The mark can be set by iptables or nftables and read by ip rule. This lets the firewall decide routing.

sudo nft add table inet mark_table
sudo nft add chain inet mark_table prerouting { type filter hook prerouting priority -150; policy accept; }
sudo nft add rule inet mark_table prerouting iif eth0 tcp dport 8080 meta mark set 0x10
sudo nft add rule inet mark_table prerouting iif eth0 tcp dport 8443 meta mark set 0x20

ip rule add fwmark 0x10 lookup 100 priority 100
ip rule add fwmark 0x20 lookup 200 priority 100

Now traffic to TCP/8080 follows table 100, and traffic to TCP/8443 follows table 200. The firewall is steering traffic into different routing tables.

Use cases

Policy routing is the right tool when:

• Multi-homing: more than one uplink with different characteristics.
• Tenant isolation: routes for one tenant’s traffic should not affect another’s.
• Service-specific routing: “database traffic should always go via the storage VLAN”.
• Source-based routing for SNAT/MASQUERADE: a NAT gateway with multiple external IPs.
• Steering based on port or mark: routing web traffic differently from backup traffic.

It is not the right tool when:

• One interface, one route - use the main table.
• The decision is per-destination only - the main table is enough.
• The complexity can be avoided with a different network design.

Common pitfalls

• Asymmetric routing: incoming traffic on eth0, outgoing reply on eth1. Breaks connection tracking and many firewalls. Test with ip route get from every source.
• Reverse-path filtering: see the callout below. This is the single most common reason a correct-looking second uplink silently receives nothing.
• Existing flows keep their old path: the IPv4 route cache was removed in kernel 3.6, so there is nothing to flush - ip route flush cache is a no-op on any kernel you will meet in production. What does persist is conntrack state and already-established sockets, which keep using the path chosen when they were created. New rules apply to new flows. If a change appears not to take effect, check the rule priority order (ip rule show), confirm the target table actually holds a matching route (ip route show table 100), and check rp_filter - do not reach for a cache flush.
• Marks not surviving: a mark set in prerouting is visible to routing. A mark set in input is not visible to routing. Place marks in the right table.

Knowledge check