Linux base — what VyOS is at the bottom

VyOS is a Linux distribution specialised for routing. The routing engineer who treats VyOS as a black box will misdiagnose incidents that are actually kernel-level — netfilter conntrack exhaustion, neighbour-cache overflow, socket-buffer drops, tc-queue congestion. Every one of these has a Linux-subsystem home, and the operator who knows which subsystem owns which failure can find the diagnostic faster.

This lesson is the operator’s foundation in the Linux base VyOS builds on: what the kernel provides, what userspace adds, where the boundary is, and how to read the kernel state when an incident demands it.

What the Linux kernel provides to VyOS

flowchart TB
  subgraph "Linux kernel"
    NET["Networking stack\nnet/ipv4, net/ipv6, net/core"]
    NF["Netfilter\nnftables, conntrack"]
    NEIGH["Neighbour subsystem\nARP / NDP / bridge FDB"]
    TC["Traffic control\nqdisc, class, filter"]
    ROUTE["Routing subsystem\nFIB, RIB, netlink"]
    CRYPTO["Crypto subsystem\nIPsec, WireGuard"]
    SKB["Socket buffer\nqueues, drops"]
  end

  subgraph "VyOS userspace"
    FRR["FRRouting\nBGP, OSPF, BFD, RIP"]
    WGD["WireGuard daemon"]
    STR["strongSwan"]
    NFTS["nftables CLI"]
    CMD["vyos-configd\nconfiguration daemon"]
  end

  NET --> ROUTE
  ROUTE --> FRR
  NF --> NFTS
  CRYPTO --> WGD
  CRYPTO --> STR

The kernel provides:

• Networking stack (net/ipv4, net/ipv6, net/core) — IPv4/IPv6 packet handling, TCP/UDP/ICMP/ICMPv6, sockets, routing hooks.
• Netfilter (net/netfilter) — packet filtering, NAT, connection tracking. The kernel side of the VyOS firewall.
• Neighbour subsystem (net/core/neighbour.c) — the ARP / NDP / bridge-FDB table the kernel maintains.
• Traffic control (net/sched) — qdiscs, classes, filters. The kernel side of the VyOS QoS configuration.
• Routing subsystem (net/ipv4/route.c, net/ipv6/route.c) — the kernel FIB, the routing hooks, the netlink interface FRRouting uses to install routes.
• Crypto subsystem (net/ipv4/esp4.c, crypto/) — IPsec ESP, WireGuard noise construction, kernel-keyed crypto.
• Socket buffers (include/linux/skbuff.h) — the per-packet data structure that flows through every kernel subsystem. Drops happen here.

The userspace adds:

• FRRouting — the dynamic routing protocols. Reads/writes the kernel FIB via netlink.
• WireGuard daemon (or kernel WireGuard module) — the modern VPN. Talks to the kernel via netlink and the wireguard.ko character device.
• strongSwan — the IPsec implementation. Userspace IKE daemon; kernel IPsec ESP via XFRM.
• nftables CLI — the userspace tool to install nftables rules into the kernel.
• vyos-configd — the configuration daemon that translates the VyOS configuration tree into kernel state via the various userspace CLIs.

The kernel / FRRouting boundary

The boundary is at the netlink interface. FRRouting runs as a userspace daemon; the kernel owns the routing table.

flowchart LR
  subgraph "FRRouting (userspace)"
    Z["Zebra daemon\n(Kernel interface)"]
    B["bgpd"]
    O["ospfd"]
    RP["ripd / staticd"]
  end

  subgraph "Kernel"
    K["Routing table\n(netlink)"]
  end

  Z <-->|"netlink messages"| K
  B -->|"internal RPC"| Z
  O -->|"internal RPC"| Z
  RP -->|"internal RPC"| Z

The zebra daemon is FRRouting’s interface to the kernel. Every other FRRouting daemon (bgpd, ospfd, ripd, etc.) communicates with zebra via internal RPC; zebra translates the RPC into netlink messages; the kernel updates the routing table; zebra receives the kernel’s netlink notifications.

This architecture has consequences for the operator:

• FRRouting’s view of the routing table (vtysh show ip route) includes routes zebra has installed in the kernel. The kernel’s view (ip route show) is what is actually being used.
• A change made via ip route add is visible to FRRouting via zebra’s netlink subscription — the routing engine sees the kernel’s view.
• A change made via FRRouting (e.g. vtysh ... ip route 10.0.0.0/24 ...) flows through zebra to the kernel. The kernel is the source of truth.

Reading kernel state from /proc

The kernel exposes its state through /proc and /sys. The operator who knows where to look can find the source of many routing incidents.

# Routing
cat /proc/net/route           # IPv4 routing table (legacy format)
cat /proc/net/ipv6_route      # IPv6 routing table
cat /proc/net/rt_cache_stat   # routing cache statistics

# Conntrack
cat /proc/net/ip_conntrack    # legacy conntrack (alias for nf_conntrack)
cat /proc/sys/net/netfilter/nf_conntrack_max

# Interfaces
cat /proc/net/dev             # interface statistics
ip -s link show               # modern equivalent

# Netfilter
cat /proc/net/netfilter/nfnetlink_log
nft list ruleset               # modern equivalent

# Routing protocol sockets
ss -ulnp | grep zebra          # zebra's netlink socket

The /proc/net/route output is the legacy hex format. ip route show is the modern equivalent. Both should agree.

Subsystem-by-subsystem: where routing failures live

The operator who knows which subsystem owns a symptom can find the right diagnostic in seconds. The operator who does not spends hours clicking through dashboards.

What VyOS adds on top of the Linux kernel

VyOS is a Linux distribution that provides:

• A declarative configuration interface (set, commit, save).
• A configuration daemon (vyos-configd) that translates the configuration tree into the kernel’s underlying CLIs (ip, nft, vtysh, etc.).
• A boot image based on Debian that includes the FRRouting package, WireGuard tools, strongSwan, nftables, and the VyOS-specific scripts.
• A package management layer (add system image, set system image default-boot) that supports upgrading the OS image with automatic rollback.
• Documentation, examples, and the operational CLI helpers (show, monitor).

The routing engineer uses the VyOS CLI for configuration but the Linux CLI (ip, ss, tcpdump, conntrack) for diagnostics. The VyOS CLI writes through the Linux CLI to the kernel; the kernel is the source of truth.

Failure modes

Kernel FIB and FRRouting RIB disagree

The kernel sees a route FRRouting did not install, or FRRouting sees a route the kernel did not accept.

Causes:

• Zebra daemon crashed; FRRouting has routes but kernel does not.
• Netlink message rejected; FRRouting tried to install but the kernel refused.
• Manual ip route add overrode a FRRouting route.

Diagnostic:

• Compare ip route show and vtysh show ip route.
• systemctl status frr or the equivalent to check the zebra daemon.

Conntrack table exhausted

The kernel stops tracking new connections. New flows are dropped.

Diagnostic:

• cat /proc/sys/net/netfilter/nf_conntrack_max for the limit.
• conntrack -L | wc -l for the current count.

Fix: raise the limit, or tune the timeouts to expire old entries faster.

Softirq saturation under traffic

The kernel softirq handler cannot keep up with packet arrival rate. Packets are dropped at the NIC or in the driver queue.

Diagnostic:

• top shows the si (softirq) percentage climbing.
• mpstat shows per-CPU softirq time.
• sar -n DEV shows packets dropped on the interface.

Fix: increase NIC ring buffer, pin softirqs to specific CPUs, or upgrade the NIC.

Kernel module missing

A subsystem the routing protocol needs is not loaded.

Diagnostic:

• lsmod lists loaded modules.
• dmesg | grep -i 'unknown symbol' for missing symbols.
• The routing daemon logs to syslog when it cannot use a feature.

Fix: load the module with modprobe or include it in the boot initramfs.

Validation

The validation sequence for “the kernel state is wrong”:

1. The kernel version: uname -r and cat /proc/version.
2. The loaded modules: lsmod | head.
3. The interface state: ip -br addr and ip -s link show.
4. The routing table: ip route show vs vtysh show ip route.
5. The neighbour table: ip neigh show.
6. The conntrack table: conntrack -L | wc -l and nf_conntrack_max.
7. The kernel logs: dmesg | tail -20.

If any subsystem reports unexpected state, the issue is at the kernel level. VyOS CLI commands cannot diagnose kernel state; they only show what they configured.

Cross-course references

• The Linux course’s V-Linux-NetConfig, XX-Linux-NetConfig, XXI-Linux-NetAdvanced, and XXII-Linux-NetTroubleshoot cover every kernel subsystem in depth.
• The OPNsense course covers the FreeBSD-specific equivalents for netfilter, pf, and the FreeBSD kernel.
• The lesson on FRRouting and the routing daemons in this part covers the userspace side in more depth.

Quiz

Production discipline

VyOS is a Linux distribution. The Linux kernel is the source of truth for routing, forwarding, firewalling, and connection tracking. The VyOS CLI is a wrapper. The Linux CLI is the ground truth.

Plan the kernel state once. Document the connection tracking sizing, the routing table validation, the netlink diagnostics. Then either routing works or the operator finds the kernel state that is wrong.