Multiple routing tables and the ip rule policy database

Linux is not a single-table router. The kernel maintains a Routing Policy Database (RPDB) — an ordered list of rules — and each rule selects a table. When a packet needs forwarding, the kernel walks the rules in priority order. The first rule that matches the packet’s selectors (source, destination, interface, mark, fwmark, tos) wins; the kernel then performs the FIB lookup in the table that rule names. This is the mechanism that makes multiple routing tables, source-based routing, and policy routing possible on Linux. This lesson covers how VyOS 1.5 LTS exposes this model through the configuration tree, what the operator- facing set system ipv4 rule paths look like, and the rule priority convention (32766 for main, 32767 for default) the kernel ships with.

The Routing Policy Database

flowchart TD
  A[Packet arrives] --> B[RPDB walk]
  B --> P0[Priority 0: local lookup]
  P0 -->|match| L[Local delivery]
  P0 -->|miss| P1[Priority 32766: main lookup]
  P1 -->|hit| M[Forward via main table]
  P1 -->|miss| P2[Priority 32767: default lookup]
  P2 -->|hit| D[Forward via default table]
  P2 -->|miss| X[ICMP unreachable / drop]

The RPDB is a list of rules. Each rule has a priority, a set of selectors, an action (lookup, blackhole, prohibit, unreachable), and a table ID for the lookup action. The kernel walks the list in ascending priority order. The first rule whose selectors match the packet determines what happens to the packet.

The default rule list on a stock Linux system:

The three default rules cover every packet. Rule 0 selects the local table (the kernel’s view of its own addresses — used for packets addressed to the router itself). Rule 32766 selects the main table (the operator’s standard routes). Rule 32767 selects the default table (empty by default; used by some user-space tools for fallback routes).

The priorities 0, 32766, and 32767 are kernel-reserved for the default rules. Operator-defined rules must use priorities outside this set — typically 1 to 32765 for rules that should override the main lookup, and 32768 and above for rules that should be consulted after main (rare in practice; usually handled with suppress-prefixlength).

Selecting a table for a packet

The selectors that an ip rule can match are richer than most operators realise. The Linux kernel RPDB supports:

• from <src> — source address (with optional /prefixlen)
• to <dst> — destination address (with optional /prefixlen)
• iif <iface> — input interface
• oif <iface> — output interface (less commonly used; the rule must match before the lookup)
• tos <value> or dsfield <value> — Type of Service / DSCP
• fwmark <mark> — packet mark from iptables/nftables
• uid <range> — UID range (socket-based lookup)
• ipproto <proto> — IP protocol (tcp, udp, icmp, etc.)
• sport <range> — source port
• dport <range> — destination port

The selectors combine with logical AND. A rule with from 10.10.0.0/24 to 8.8.8.0/24 fwmark 0x10 matches packets whose source is in 10.10.0.0/24, whose destination is in 8.8.8.0/24, and whose firewall mark is 0x10. Anything outside that intersection falls through to the next rule.

The action can be:

• lookup <table> — perform a FIB lookup in the named table
• blackhole — silently drop the packet
• unreachable — drop and reply with ICMP unreachable
• prohibit — drop and reply with ICMP administratively prohibited
• goto <priority> — jump to the rule at the named priority

blackhole, unreachable, and prohibit are the policy-side analogues of static blackhole routes. They are useful when an operator wants to drop traffic without affecting the routing table itself (e.g. dropping traffic from a compromised subnet without installing a route for it).

The VyOS configuration tree

VyOS 1.5 LTS exposes the RPDB through set system ipv4 rule (and set system ipv6 rule for IPv6). The full syntax:

set system ipv4 rule <priority> from <src>
set system ipv4 rule <priority> to <dst>
set system ipv4 rule <priority> iif <iface>
set system ipv4 rule <priority> table <name|id>
set system ipv4 rule <priority> suppress-prefixlength <n>

A canonical rule that selects table mgmt (ID 100) for traffic from the management subnet:

[edit]
vyos@vyos# set system ipv4 rule 100 from 10.10.0.0/24 table mgmt
[edit]
vyos@vyos# commit
[edit]
vyos@vyos# save

The table argument accepts either the numeric ID or the alias name declared in set system ip rt-table. VyOS prefers the name when it can; the alias keeps the configuration human-readable.

The alias declaration:

[edit]
vyos@vyos# set system ip rt-table rt-table-id 100 name mgmt description "management traffic table"
[edit]
vyos@vyos# commit
[edit]
vyos@vyos# save

The rt-table-id is the numeric ID. The name is the alias. The description is metadata. VyOS writes the alias into /etc/iproute2/rt_tables on commit, which makes the name available to iproute2.

The default rule list, validated

vyos@vyos:~$ ip rule show
0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default

The VyOS shell shows the same data through show ip rule:

vyos@vyos:~$ show ip rule
0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default

Once a custom rule is committed, it appears in the list. For example, after the rule above:

vyos@vyos:~$ show ip rule
0:	from all lookup local
100:	from 10.10.0.0/24 lookup mgmt
32766:	from all lookup main
32767:	from all lookup default

The walk order is priority ascending. Rule 100 (the custom rule) is consulted before rule 32766 (main). Traffic from the management subnet consults table mgmt first; if the lookup misses (no matching prefix), the walk continues to the next rule.

The suppress-prefixlength attribute is the way to add a fallback without changing the priority. A rule like:

vyos@vyos# set system ipv4 rule 200 lookup table mgmt suppress-prefixlength 0

selects table mgmt only for prefixes that are longer than /0 — i.e. all prefixes, except the default. The suppress-prefixlength says “do not use this table for prefixes shorter than N”; a value of 0 means “for everything except the default route”.

The FRR zebra view

FRR zebra is the user-space daemon that programs the kernel’s RPDB on behalf of the operator. VyOS renders the configuration tree into frr.conf.d/*.conf fragments; zebra reads its configuration and pushes the rules into the kernel.

vyos@vyos:~$ vtysh -c 'show ip rule'
0:	from all lookup local
100:	from 10.10.0.0/24 lookup mgmt
32766:	from all lookup main
32767:	from all lookup default

The FRR view and the kernel view should agree. If they do not, the cause is one of:

• A rule written outside of VyOS (a script that called ip rule add directly). FRR does not know about it; the kernel sees it.
• A FRR reload that has not propagated. The rule is in the candidate but not in zebra’s running config.
• A kernel-side rejection. Zebra tried to install the rule; the kernel returned an error.

The first of these is the most common in production. The operator must remove the script-installed rule and re-apply through VyOS.

How the result is validated

The validation command set for a multi-table deployment:

vyos@vyos:~$ show ip rule
0:	from all lookup local
100:	from 10.10.0.0/24 lookup mgmt
32766:	from all lookup main
32767:	from all lookup default

vyos@vyos:~$ ip rule show
0:	from all lookup local
100:	from 10.10.0.0/24 lookup mgmt
32766:	from all lookup main
32767:	from all lookup default

vyos@vyos:~$ ip route show table all
table local:
local 192.0.2.1 dev lo proto kernel scope host src 192.0.2.1
broadcast 192.0.2.255 dev lo proto kernel scope link src 192.0.2.1
table main:
10.20.0.0/16 via 192.0.2.2 dev eth0 proto static metric 1
192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.1
table mgmt:
10.30.0.0/24 via 192.0.2.3 dev eth0 proto static metric 1
table default:

vyos@vyos:~$ ip route get 10.30.0.5 from 10.10.0.5
10.30.0.5 from 10.10.0.5 dev eth0 table mgmt uid 0
    cache

The ip route get command is the canonical source of truth for “which table will this packet consult?”. The output explicitly names the table. If the table is wrong, the rule chain is wrong.

How it fails

The production failure modes the engineer must recognise:

• Rule with wrong priority. A rule at priority 32768 is consulted after the main table; the operator expected it to be consulted before. Traffic to the affected prefix falls through to main and uses the wrong route.
• Rule with wrong selector. A rule with from 10.10.0.0/24 that should have been from 10.10.0.0/16. Traffic from 10.10.1.0/24 matches the operator’s intent; traffic from 10.10.2.0/24 does not. The operator sees inconsistent behaviour.
• Rule references a table with no routes. The rule selects table mgmt; the table is empty. The lookup misses; the walk falls through to main. The custom rule does nothing.
• Rule added in two places. An operator script added a rule at priority 100 with from 10.10.0.0/24 lookup mgmt. The same rule is in VyOS. VyOS does not deduplicate; both rules exist; one of them wins by ip rule add order.
• Alias name not declared. The rule references table mgmt but set system ip rt-table rt-table-id 100 name mgmt is missing. The rule works at the kernel level (the kernel does not know names) but ip route show table mgmt returns nothing because iproute2 cannot resolve mgmt to 100.

Rollback

The recovery from a rule misconfiguration:

• Wrong selector: delete system ipv4 rule <priority>; set system ipv4 rule <priority> from <correct>; commit; save.
• Wrong priority: delete system ipv4 rule <priority>; set system ipv4 rule <new-priority> ...; commit; save.
• Wrong table: delete system ipv4 rule <priority>; set system ipv4 rule <priority> ... table <correct>; commit; save.
• Stray rule from a script: ip rule del priority <priority> on the live system, then audit the script that added it.
• Whole-tree rollback: rollback N; commit; save.

For emergency rollback, load <file>; commit; save replaces the candidate with a saved backup.

Production discipline

Cross-course references

The Linux course’s XXI-Linux-NetAdvanced covers the kernel- side mechanics of the RPDB and the same rule model from a host perspective. The OPNsense course’s II-OPNsense-Routing covers the FreeBSD setfib equivalent and how the firewall rule selects a FIB. The BGP course’s XXXIII-RoutePolicy covers the policy-side; the RPDB is the substrate that policy routing sits on.

Quiz