Control plane versus data plane — what FRR decides and what the Linux kernel forwards

A VyOS router is two machines glued together. One is the routing engine — the FRR suite of daemons that runs the protocols, builds the routing table, and decides where each prefix should go. The other is the forwarding engine — the Linux kernel that actually moves packets between interfaces. The two communicate through a tight contract: FRR tells the kernel which prefixes go where, and the kernel consults that information on every packet it forwards. This lesson is about the contract — what FRR does, what the kernel does, how they talk, and what goes wrong when they disagree.

The lesson is foundational because every routing-protocol lesson (OSPF, IS-IS, BGP) operates on the FRR side, and every performance lesson (interrupt affinity, packet drops, route churn) operates on the kernel side. The operator who understands the two-plane model diagnoses routing incidents in minutes; the operator who does not treats FRR and the kernel as one thing and spends hours in the wrong subsystem.

The two planes

The control plane is the decision: “destination 203.0.113.5 should go via 198.51.100.1 on eth0”. The control plane runs the routing protocols, builds the routing information base (RIB), and chooses a best path for every prefix. On a VyOS router the control plane is FRR — the zebra daemon plus the per-protocol daemons (ospfd, bgpd, isisd, ripd, staticd).

The data plane is the action: “for this packet to 203.0.113.5, look up the route, write the destination MAC, send the frame out eth0”. On a VyOS router the data plane is the Linux kernel. The kernel maintains the forwarding information base (FIB) — the structure consulted on every packet — and pushes packets out through the NIC driver.

flowchart TD
  subgraph ControlPlane["Control plane (FRR, userspace)"]
    A1[ospfd] --> Z[zebra]
    A2[bgpd] --> Z
    A3[isisd] --> Z
    A4[staticd] --> Z
    Z --> RIB[FRR RIB]
  end
  subgraph DataPlane["Data plane (Linux kernel)"]
    FIB[Kernel FIB]
    FIB --> P[Per-packet lookup]
  end
  RIB -->|netlink RTM_NEWROUTE| FIB
  P --> NIC[NIC driver]
  NIC --> W[Wire]

The two planes live on the same box, but they are distinct processes: FRR is a set of userspace daemons; the FIB is kernel state. The boundary between them is the netlink socket — the interface zebra uses to push routes into the kernel, and the kernel uses to answer.

What FRR does

FRR runs as a set of cooperating daemons. Each protocol has its own: ospfd handles OSPF, bgpd handles BGP, isisd handles IS-IS, ripd handles RIP, staticd handles static routes. Each one implements its protocol, receives routing information, runs its algorithm, and offers zebra a candidate path for every prefix.

zebra is the arbiter. It collects candidates from every protocol daemon, applies the tie-breaking rules (administrative distance first, then metric), selects one best path per prefix, and installs it into the kernel over netlink. zebra is also the next-hop oracle: when bgpd computes a path whose next hop is 198.51.100.1, it asks zebra whether that address is reachable. If zebra says no, the path is not selected and never reaches the kernel.

$ show ip route

Codes: K - kernel route, C - connected, L - local, S - static,
     R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
     T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
     F - PBR, f - OpenFabric, t - Table-Direct,
     > - selected route, * - FIB route, q - queued, r - rejected, b - backup
     t - trapped, o - offload failure

S>* 10.20.0.0/16 [1/0] via 192.0.2.2, eth0, 00:00:12
O>* 198.51.100.0/24 [110/20] via 192.0.2.2, eth0, 00:04:31
C>* 192.0.2.0/24 is directly connected, eth0, 00:11:07
L>* 192.0.2.1/32 is directly connected, eth0, 00:11:07

The > and * markers are two different statements. > is FRR’s decision: this is the best path I have for this prefix. * is the kernel’s answer: I accepted it and it is in the FIB. They usually agree. When they do not, you have found a control-plane to data-plane divergence, and you already know which direction it is in.

What the kernel does

The kernel maintains the FIB: a longest-prefix-match structure that maps a destination address to a next hop, an egress interface, and a set of attributes. The FIB is the source of truth for what the router actually does with a packet.

VyOS gives you the kernel’s view without leaving operational mode. show ip route forward runs ip route list underneath, and the tagged form show ip route forward 10.20.0.0/16 runs ip -s route list for a single prefix:

$ show ip route forward

10.20.0.0/16 via 192.0.2.2 dev eth0 proto static
198.51.100.0/24 via 192.0.2.2 dev eth0 proto ospf
192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.1

The proto field is the provenance of the route. proto kernel means the kernel created it itself because an interface came up with an address on that subnet. proto static, proto ospf, proto bgp mean zebra installed it and tagged it with the originating protocol. Routes an operator adds by hand with ip route add default to proto boot — which is a useful fingerprint when you find one nobody meant to create.

The kernel FIB is updated through netlink. zebra opens a netlink socket, sends RTM_NEWROUTE messages to add routes and RTM_DELROUTE to withdraw them, and reads the kernel’s acknowledgement or error.

How FRR and the kernel communicate

When a route changes in FRR — a new static, a withdrawn BGP prefix, an OSPF path that flipped — zebra sends a netlink message. The kernel updates the FIB and answers.

sequenceDiagram
  participant Z as zebra
  participant K as Kernel
  Z->>K: RTM_NEWROUTE (RTA_DST, RTA_GATEWAY, RTA_OIF, proto)
  K->>Z: ack, or an error such as ENETUNREACH
  Note over K: On ack, the prefix is in the FIB
  Z->>K: RTM_DELROUTE when the path is withdrawn
  K->>Z: ack

Three properties of the contract matter:

• FRR is the writer for dynamic routes. The kernel does not invent them; it accepts or rejects what zebra sends.
• The kernel is the executor. zebra cannot make the kernel forward a packet. It can only put an entry in the FIB and let the kernel decide per packet.
• The kernel can say no. A netlink route message can be rejected, and the rejection is an error on the socket rather than a missing line in a table. If nobody reads the log, the only symptom is a route with > and no *.

To watch the boundary live, use ip monitor route, which subscribes to the same netlink route group the kernel multicasts changes on. It prints each add and delete as it happens, which is the fastest way to prove whether zebra pushed a route at all:

$ ip monitor route

10.20.0.0/16 via 192.0.2.2 dev eth0 proto static
Deleted 10.20.0.0/16 via 192.0.2.2 dev eth0 proto static

The views of the routing table

A production VyOS router has several views. Each view is correct for what it shows; they can disagree, and the disagreement is the diagnosis.

Reading them in that order narrows a fault fast. If the intent is not in show ip route, the problem is in FRR or the configuration. If it is in show ip route but not in show ip route forward, the problem is at the netlink boundary or in the next-hop resolution. If both agree and traffic still does not flow, the problem is not routing at all — it is firewall, forwarding state, or the return path.

show configuration commands | match 'static route'
show ip route 10.20.0.0/16
show ip route forward 10.20.0.0/16

The single fastest check is ip route get, which asks the kernel to perform one real lookup and report the answer:

$ ip route get 203.0.113.5

203.0.113.5 via 192.0.2.2 dev eth0 src 192.0.2.1 uid 0
  cache

Forwarding state, and what VyOS does not expose

A route in the FIB is necessary but not sufficient. The kernel also has to be willing to forward, globally and per interface.

VyOS turns per-interface forwarding off with an ip node on the interface, not a top-level leaf:

configure
set interfaces ethernet eth0 ip disable-forwarding
commit
save

That renders to the kernel as /proc/sys/net/ipv4/conf/eth0/forwarding set to 0. The IPv6 equivalent is set interfaces ethernet eth0 ipv6 disable-forwarding. The failure this produces is quiet: the route is in the FIB, the interface is up, and packets are dropped anyway.

VyOS 1.5 has no show ip forwarding operational command. The show ip tree covers routes, neighbours, and the routing protocols; forwarding state is not in it. Read it from the kernel directly, which is where it lives:

cat /proc/sys/net/ipv4/ip_forward
cat /proc/sys/net/ipv4/conf/eth0/forwarding

FRR keeps its own idea of the same flag, and vtysh will report it, which is occasionally useful when you want to know whether FRR believes forwarding is on:

vtysh -c 'show ip forwarding'

How it fails

The production failure modes the engineer must recognise:

• Route in FRR with no * marker. FRR selected the path but the kernel does not have it. The dominant cause is next-hop resolution: the gateway is not reachable over any connected subnet, so the kernel would reject the install and zebra does not complete it. Fix the next hop or the interface address, not the route.
• Route in the kernel, absent from FRR. Almost always a route added by hand with ip route add, which is why the proto boot marker is worth knowing. It works until the next reboot or the next interface event, and it is invisible to show configuration, so it survives exactly long enough to confuse the next engineer.
• Route in the configuration, absent from FRR. The commit succeeded but the rendered FRR configuration did not contain the line, or FRR rejected the reload. VyOS renders the whole FRR configuration at commit and tests it before applying, so this usually shows as a failed commit; when it does not, compare show configuration commands against vtysh -c 'show running-config'.
• Forwarding disabled on the interface. The FIB says “out eth0”, the interface is up, and /proc/sys/net/ipv4/conf/eth0/forwarding is 0. Packets are dropped with no log line anywhere.
• zebra stopped, routes still in the kernel. The kernel does not age FIB entries, so a stopped zebra leaves its routes behind, pointing at whatever they pointed at when it died. The router keeps forwarding on stale information. restart zebra re-establishes the daemon and reconciles; confirm afterwards that show ip route and show ip route forward agree again.
• Netlink errors during a large churn. A burst of route changes can produce netlink errors that appear only in the FRR log. show log frr is where they surface on VyOS — not /var/log/frr/, because VyOS runs FRR under systemd and its output goes to the journal.

Rollback

• Wrong route in FRR: delete protocols static route 10.20.0.0/16 then commit and save. zebra withdraws it over netlink and both views align again.
• A route added by hand outside VyOS: ip route del 10.20.0.0/16 removes it, and nothing persists it across a reboot because it was never in the configuration. If the route is wanted, add it properly under set protocols static route so it survives and so the next engineer can find it.
• Forwarding disabled by mistake: delete interfaces ethernet eth0 ip disable-forwarding then commit and save.
• Control plane in an unknown state: restart zebra restarts the RIB manager. It is disruptive — every dynamic route is reinstalled — so treat it as an intervention, not a diagnostic step, and expect a brief forwarding gap.

For a change on the router that carries your own access path, commit-confirm 5 commits for five minutes and reverts unless you type confirm. The default revert action is a reboot to the saved configuration; set system config-management commit-confirm action reload makes it a configuration reload instead.

Production discipline

Additional discipline:

• Baseline the route counts. show ip route summary gives per protocol totals; record them after each change so a later deviation is measurable rather than a feeling.
• Ship the FRR journal to a remote collector. Netlink errors and route-install failures appear there and nowhere else, and they are exactly the class of failure that produces “the route is in the table but the packets do not flow”.
• Know the boundary tools before you need them: ip monitor route to watch installs live, ip route get to ask the kernel one question, and show log frr to read what zebra thought about it.
• Treat any route with proto boot in show ip route forward as an incident to be explained. Something outside the configuration tree wrote to your FIB.

Cross-course references

The architecture part III-VyOS-Architecture covers the FRR and kernel roles in depth. II-VyOS-RoutingFund covers the RIB and FIB model. L-VyOS-Performance covers the cost of a large FIB on the data plane. The Linux course’s XXII-Linux-NetTroubleshoot covers the host-side equivalent of the same split.

Quiz