Ethernet, MAC and ARP — the Layer 2 the routing engineer must read

A VyOS router is, at the bottom of the stack, a Linux kernel forwarding packets between Ethernet interfaces. The routing engine (FRR) decides which interface the packet should leave through. The kernel actually moves the bytes. Between those two, the neighbour subsystem turns the next-hop IP into the next-hop MAC address the wire actually requires.

This lesson is about the half of the network most routing engineers skip until something breaks. A BGP session that “looks fine” but shows no traffic, an OSPF adjacency that hangs in EXSTART, a floating static route that never activates — every one of those symptoms is, half of the time, a Layer 2 problem dressed up as a Layer 3 problem. The routing engineer who cannot read what is on the wire has no way to tell the two apart.

What is on the wire

Every packet a VyOS router forwards rides on an Ethernet frame. A frame is what actually crosses the NIC; the IPv4 or IPv6 packet is the payload. Even on a routed path the packet is wrapped in a fresh frame at every Layer 3 hop, because the Layer 2 segment ends at the router’s interface.

flowchart LR
  subgraph "Frame on eth0"
    A["Preamble\n7 + 1 bytes"] --> B["Dest MAC\n6 bytes"]
    B --> C["Src MAC\n6 bytes"]
    C --> D["Ethertype\n2 bytes"]
    D --> E["Payload\n46 - 1500 bytes"]
    E --> F["FCS\n4 bytes"]
  end

The fields that matter to the routing engineer:

• Destination MAC: the MAC the frame is being delivered to. For unicast traffic this is the next-hop’s MAC, not the final destination’s MAC — this is the single most important fact about Layer 2 forwarding.
• Source MAC: the MAC of the sending interface. On VyOS, this is the MAC of the egress interface, set by the kernel.
• Ethertype: 0x0800 is IPv4, 0x86DD is IPv6, 0x0806 is ARP, 0x8100 is an 802.1Q VLAN tag wrapping a second ethertype.
• Payload: the IP packet (or ARP message).
• FCS: the Ethernet frame check sequence. Stripped by the NIC before the kernel ever sees the bytes.

The MAC address is 48 bits, written aa:bb:cc:11:22:33. The first three bytes are the OUI assigned to the manufacturer; the last three are device-unique. VyOS assigns MACs to every interface at boot (driven by the NIC firmware for physical ports, deterministically derived for sub-interfaces), and the operator should know how to read them — they are the only way to tell which of two physical interfaces the kernel has actually bound a sub-interface to.

Three kinds of Layer 2 traffic

flowchart TB
  classDef uni fill:#dbeafe,stroke:#1e3a8a
  classDef broad fill:#fef3c7,stroke:#854d0e
  classDef multi fill:#dcfce7,stroke:#166534

  UNI["Unicast\none MAC\none IP"]:::uni
  BROAD["Broadcast\nff:ff:ff:ff:ff:ff\nsubnet or 255.255.255.255"]:::broad
  MULTI["Multicast\n01:00:5e... / 33:33...\n224.0.0.0/4 or ff00::/8"]:::multi

  UNI --> E["Reaches one host"]
  BROAD --> F["Reaches every host on the segment"]
  MULTI --> G["Reaches hosts that joined the group"]

Most production traffic is unicast. ARP requests are broadcast. VRRP advertisements, OSPF hello packets, and PIM joins are multicast. IPv6 ND and Router Advertisements are multicast too. CARP on OPNsense is multicast (one of the few places a network operator has to remember that CARP is the BSD firewall’s variant of VRRP, not the same protocol as VRRP on VyOS).

How ARP and the Linux neighbour subsystem work

ARP is the protocol that turns “the next-hop IP is 192.0.2.1” into “the next-hop MAC is aa:bb:cc:11:22:33”. On a VyOS router, this is handled by the Linux neighbour subsystem — the kernel data structure that maps IP to MAC.

When the kernel needs to send a packet to 192.0.2.1:

1. The routing lookup produces the next-hop 192.0.2.1 and the egress interface eth0.
2. The neighbour subsystem looks up 192.0.2.1 in its table.
3. If the entry is REACHABLE or STALE, the kernel uses the cached MAC immediately.
4. If the entry is missing or INCOMPLETE, the kernel queues the packet and broadcasts an ARP request (who has 192.0.2.1? tell 192.0.2.50).
5. The target replies with an ARP reply carrying its MAC. The kernel installs the entry as REACHABLE.
6. The queued packet is forwarded.

The state machine is the Linux kernel neighbour state machine, not a VyOS-specific subsystem. VyOS exposes it through the ip neigh command. Reading it well is half of routing incident diagnostics.

How VyOS exposes Layer 2 to the operator

The VyOS operational CLI exposes the Linux state machine through the standard ip commands. The VyOS configuration tree does not directly model the neighbour table — that is a kernel-level decision — but the ARP/NDP timeout, the proxy-ARP flag, and the static ARP entry are all settable from configure mode.

configure
# Global neighbour-table parameters (kernel sysctl equivalents)
set system ipv6 disable-ipv6
set system ip neighbor ...
# Per-interface parameters
set interfaces ethernet eth0 ip enable-arp
set interfaces ethernet eth0 ip arp-cache-timeout 1800
set interfaces ethernet eth0 ip proxy-arp
commit
save
exit

The most useful operational commands are not VyOS-specific:

show interfaces ethernet eth0
ip -s link show dev eth0
ip -s neigh show dev eth0
bridge fdb show
tcpdump -nei eth0 -c 50 arp
tcpdump -nei eth0 -c 50 'arp or icmp'

ip -s neigh shows both the entry state and the counters (REACHABLE, STALE, FAILED plus the number of probes sent and replies received). bridge fdb show is the bridge-level forwarding database — relevant if the router has bridge interfaces.

How a routing incident surfaces at Layer 2

sequenceDiagram
  autonumber
  participant App as Application
  participant K as VyOS kernel
  participant N as Neighbour table
  participant E as Ethernet (eth0)
  participant NH as Next-hop host

  App->>K: sendto(8.8.8.8)
  K->>K: routing lookup → next-hop 192.0.2.1, dev eth0
  K->>N: lookup 192.0.2.1
  alt entry FAILED or missing
    K->>E: ARP request broadcast
    E-->>K: no reply
    K-->>App: EHOSTUNREACH
  else entry REACHABLE
    K->>E: frame, dst MAC aa:bb:cc:11:22:33
    E->>NH: deliver
    NH-->>E: reply
    E-->>K: reply frame
    K-->>App: deliver
  end

The routing engineer who sees “no route to host” or “connect: network is unreachable” must be able to say whether the failure is at Layer 3 (no route in the kernel FIB) or at Layer 2 (route is fine, neighbour cannot be resolved). The ip route get <dst> and ip neigh show commands together answer that question.

Validation: proving the Layer 2 path is healthy

The diagnostic sequence when traffic is not flowing where the routing table says it should:

# 1. Confirm the route exists and points the right way
show ip route 8.8.8.8
ip route get 8.8.8.8

# 2. Confirm the neighbour entry state for the next-hop
ip -s neigh show 192.0.2.1 dev eth0

# 3. Confirm the interface is up and the link is OK
show interfaces ethernet eth0
ethtool eth0

# 4. Capture the wire to prove frames are actually leaving
tcpdump -nei eth0 -c 20 'host 192.0.2.1'

# 5. Capture ARP specifically to prove the resolution works
tcpdump -nei eth0 -c 20 arp

If the route is correct and the neighbour is FAILED, the problem is Layer 2 and the routing table is a red herring. If the neighbour is REACHABLE but no traffic flows, the problem is either upstream of the next-hop or on the wire itself (cable, SFP, port).

Cross-course references

• The Linux course’s V-Linux-NetConfig, XXI-Linux-NetAdvanced, and XXII-Linux-NetTroubleshoot parts cover the same iproute2 primitives from the host perspective.
• The OPNsense course covers the equivalent Layer 2 picture on FreeBSD, including CARP (a BSD-flavour VRRP variant that uses multicast on a sync segment) and how OPNsense uses arpwatch and ifconfig to expose the same data.
• The Proxmox course covers the host-side network configuration that runs under a VyOS VM, including VLAN trunking through vmbr bridges and the MAC-address randomisation behaviour of KVM.

Quiz

Production discipline

The lesson above is the foundation for several production-grade skills taught later in the course:

• BGP session Established but no traffic flows → check the neighbour state.
• OSPF adjacency stuck in EXSTART/EXCHANGE → check the MTU and the neighbour state.
• VRRP failover does not occur when expected → check the multicast path between the peers.
• Floating static route never activates → check that the tracked object’s reachability is being tested at Layer 2.

Every one of those incidents is a Layer 2 problem on a Layer 3 ticket. The routing engineer who cannot read what is on the wire opens the wrong ticket every time.