Policy-based routing — concept, FIB vs RIB, table selection, why PBR exists

A classic IP router makes one decision per packet: given the destination address, pick the most specific matching route in the routing table, forward to that next-hop. The decision is purely destination-based. Source address, ingress interface, packet size, markings, time-of-day — none of them influence the routing decision in a destination-only design.

That model is what the original IP designers wanted: stateless, destination-only, every router independent. It is also what breaks when the operator needs traffic to take different paths based on who sent it, where it entered, or what application generated it. Policy-based routing (PBR) is the production mechanism for breaking the destination-only constraint, and VyOS 1.5 LTS implements it with FRR’s route-map tree, the Linux kernel’s ip rule selector, and the multiple routing tables that FRR’s zebra populates. This lesson walks through the model, the forwarding path, and the production scenarios where PBR is the only correct answer.

Destination-based routing — what the original model does

flowchart LR
  A[Packet arrives<br/>src=10.0.0.5<br/>dst=203.0.113.7] --> B{Routing table lookup}
  B -->|longest-prefix match| C[next-hop 198.51.100.1]
  C --> D[Egress eth0]
  B -->|no match| E[ICMP unreachable]

The destination-only model is the default. The router inspects the destination IP, runs longest-prefix-match across the routing table, and forwards. The source address is not consulted. The ingress interface is not consulted. The DSCP bits are not consulted. The TCP source port is not consulted. For a simple enterprise or single-homed site, this model is correct and sufficient.

The model breaks when the operator needs:

• Source-based routing. Traffic from the voice VLAN must use a specific provider regardless of destination; traffic from the data VLAN must use a different provider.
• Application-based routing. Traffic from the database segment must exit through the dedicated database uplink; traffic from the user segment must exit through the user uplink.
• Source-address failover. Traffic from the office subnet must use the primary WAN; if the primary WAN is down, the office traffic must fail to the backup WAN without disturbing traffic from the warehouse subnet that is already on the backup.
• Multi-VRF leak policy. Traffic arriving on VRF-A destined for a known address in VRF-B must be leaked through a specific route-map; the default route in VRF-A must not be used for the leak.
• Compliance and audit. Outbound traffic from the PCI segment must be tagged with a community so the upstream can log it; traffic from the rest of the network does not need the community.

Each of those is a policy that cannot be expressed in destination-only forwarding.

The Linux FIB vs RIB split — where PBR sits

Linux separates the forwarding plane from the control plane:

• RIB (Routing Information Base) — the control-plane view of what routes exist and where they came from. FRR’s zebra maintains the RIB in memory; it is consulted by routing protocols, by show ip route, and by the route-map logic.
• FIB (Forwarding Information Base) — the data-plane view consulted on every packet. The kernel maintains the FIB; it is consulted on every forwarding decision. The FIB is optimised for fast lookup, not for rich metadata.

flowchart TB
  subgraph CP["Control plane"]
    Z[zebra<br/>FRR RIB]
    RM[route-map<br/>policy]
    Z --> RM
  end
  subgraph DP["Data plane"]
    K[Kernel FIB]
    NETLINK[netlink updates]
  end
  subgraph RS["Routing stack"]
    RULE[ip rule selector]
    TBL[table main<br/>table 100<br/>table 200]
  end
  Z -->|netlink| NETLINK
  NETLINK --> K
  CP --> RS
  DP --> RS

The kernel also maintains multiple routing tables. The default table is main (table 254). Operators can create additional tables (table 100, table 200, etc.). Each table is a separate set of routes; the kernel consults them in priority order through the ip rule selector.

The ip rule selector is the heart of PBR on Linux. Each rule says: “for packets matching this criterion, look up the route in this table.” The criterion can be source address, destination address, ingress interface, fwmark, or a combination. The first matching rule wins.

vyos@vyos:~$ ip rule show
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

This is the default. The rule at priority 0 says “all packets first consult the local table” (for locally-destined packets). The rule at priority 32766 says “if no local match, consult the main table.” The rule at priority 32767 says “if no match, consult the default table” (usually empty).

A PBR deployment adds a rule with a higher priority (lower number) than 32766. For example:

vyos@vyos:~$ set policy route-map RM-VOICE-OUT rule 10 match ip source address 10.10.0.0/24
vyos@vyos:~$ set policy route-map RM-VOICE-OUT rule 10 set ip next-hop 198.51.100.1
vyos@vyos:~$ set interfaces ethernet eth1 policy route-map RM-VOICE-OUT

After commit, the FRR configuration renders to:

ip rule 100 from 10.10.0.0/24 lookup 100
ip route 0.0.0.0/0 198.51.100.1 table 100

The kernel installs rule 100 at priority 100 (higher priority than 32766). For packets with source 10.10.0.0/24, the kernel consults table 100 instead of the main table. Table 100 has the default route via the voice provider. The result: voice traffic uses the voice provider regardless of destination.

How PBR sits in the forwarding path

flowchart TD
  A[Packet arrives on eth1] --> B[Ingress interface]
  B --> C{VRF selection}
  C --> D{ip rule selector}
  D -->|rule 100 match<br/>src 10.10.0.0/24| E[Table 100 lookup]
  D -->|rule 32766 match<br/>all| F[Main table lookup]
  E -->|next-hop 198.51.100.1| G[Egress eth2<br/>voice provider]
  F -->|next-hop 203.0.113.1| H[Egress eth0<br/>data provider]

The packet enters on eth1. The kernel’s VRF selection picks the routing instance (default if no VRF). The ip rule selector walks the rules in priority order. Rule 100 matches because the source is 10.10.0.0/24; the kernel consults table 100. Table 100 returns the voice provider next-hop. The packet exits on eth2.

If the source had been 10.20.0.0/24 (the data VLAN), rule 100 would not match. The selector would fall through to rule 32766 (main table). The main table returns the data provider next-hop. The packet exits on eth0.

Two packets, same destination, different paths. That is the essence of PBR.

Why production networks need PBR

The production scenarios the engineer must recognise:

flowchart TB
  subgraph SC["Production scenarios that need PBR"]
    S1[Multi-WAN: source-based provider selection]
    S2[Source-address failover independent of destination]
    S3[Application-based routing via fwmark]
    S4[Compliance: tag PCI segment with community]
    S5[Multi-VRF: route-map at leak boundary]
  end

Multi-WAN with source-based provider selection. A site has two providers and the voice traffic must always use provider-A (lower latency, jitter guarantee), while the data traffic can use provider-B (cheaper). Destination-based routing cannot express this; the destination is the same for both traffic classes (public Internet). PBR via a route-map on the ingress interface or via fwmark from iptables is the production answer.

Source-address failover independent of destination. The office subnet must fail to backup WAN when primary WAN is down, but the warehouse subnet is already on the backup and must not fail back when primary recovers. Destination-based routing has no notion of “the office subnet” vs “the warehouse subnet” — it sees only destination prefixes. PBR per ingress interface or per source subnet is the only way to keep the failover scopes independent.

Application-based routing via fwmark. The database traffic (TPC-C to a known set of hosts) must exit through the dedicated database uplink with a guaranteed 100 Mbps. The user traffic can exit through the user uplink. Destination-based routing picks the shortest prefix; if the database hosts have shorter prefixes than the user hosts, the database traffic still uses the user uplink. PBR via fwmark set by iptables/nftables and matched in ip rule overrides the destination-only decision.

Compliance tagging. Outbound traffic from the PCI segment must be tagged with the BGP community 65001:9999 so the upstream can apply the compliance log policy. Destination-based routing cannot tag based on source. PBR via a route-map on the egress interface matches the source and sets the community.

Multi-VRF route-leak policy. Traffic arriving on VRF-A destined for a known address in VRF-B must be leaked through a specific route-map. The default route in VRF-A must not be used for the leak. PBR via a route-map at the leak boundary enforces the policy.

The price PBR extracts

PBR is not free. The production engineer must recognise the cost:

• Asymmetric routing. PBR can easily create a forwarding path for which the return path takes a different route. The return packets bypass the firewall and the stateful inspection sees a flow it never saw go out. Stateful firewalls drop the packets.
• Hidden complexity. A route-map is harder to read than a routing table. The operator reading the configuration must trace the rule priority, the table id, and the matching criteria to understand the effective path.
• Bypass of dynamic failover. A PBR rule that hard-codes a next-hop overrides the dynamic failover that the routing protocol would otherwise provide. When the hard-coded next-hop fails, the PBR rule does not fail over.
• Performance. The kernel must walk the rule list on every packet. With a small rule list this is fast; with hundreds of rules (anti-pattern) the lookup is O(n) per packet and the data plane degrades.

The lesson vyos-xiii-06-pbr-anti-patterns covers the specific anti-patterns; the lesson vyos-xiii-05-pbr-troubleshoot covers the troubleshooting commands.

How the configuration is validated

The validation command set confirms the PBR is active:

show policy route-map
show ip route
show ip rule
ip rule show
ip route show table 100
ip route show table all

The show policy route-map lists the configured route-maps. The show ip route shows the main table. The show ip rule shows the FRR view of the rules; ip rule show shows the kernel view after zebra has installed them. The ip route show table 100 shows the contents of the PBR table.

A working PBR deployment has:

• The route-map rule in show policy route-map with the expected action, match, and set clauses.
• The ip rule show output contains the PBR rule at the configured priority.
• The PBR table (ip route show table <id>) contains the routes the route-map adds.
• The ip route get <src> <dst> resolves to the PBR next-hop for the matching source.

How it fails

The production failure modes the engineer must recognise:

• Route-map not applied. The route-map is configured but not bound to any interface. The rule never fires. ip rule show does not contain the PBR rule.
• Wrong rule priority. The PBR rule is installed at a priority that is overridden by another rule. ip rule show shows the rule; the kernel falls through to the higher-priority rule first.
• Empty PBR table. The route-map matches but the PBR table has no routes. The kernel has no next-hop; the packet is dropped. ip route show table <id> is empty.
• Hard-coded next-hop down. The PBR rule hard-codes a next-hop that has failed. The packet is forwarded to the dead next-hop. ip route get <src> <dst> returns the PBR next-hop; ping <nexthop> fails.
• Asymmetric routing. The forward path uses PBR; the return path uses the main table. Stateful firewalls on the return path drop the packets because no state exists. The flow is one-way silent.

Rollback

The recovery from a bad PBR configuration:

• Wrong route-map: delete policy route-map <name>; commit; save.
• Route-map on wrong interface: delete interfaces ethernet <ifname> policy route-map <name>; commit; save.
• Wrong rule priority: change the priority on the route-map rule; commit; save.
• Wrong next-hop: set policy route-map <name> rule <n> set ip next-hop <correct>; commit; save.
• Whole-tree rollback: rollback N; commit; save.

Production discipline

Cross-course references

The Linux course’s XIX-Linux-NetFoundations covers the kernel FIB and the routing table model in depth. The VyOS lessons vyos-xii-01-static-routes, vyos-xii-04-blackhole-routes, and vyos-xii-03-static-route-options cover the route-map use for BGP redistribution that this lesson assumes. The lessons vyos-xiv-01-routing-table-concept and vyos-xiv-02-multiple-tables cover the table model and the ip rule selector that PBR uses. The lesson vyos-xv-02-vrf-config covers the VRF binding that PBR can leverage.

Quiz