Ethernet, MAC and ARP — the Layer 2 the firewall operator must read

Every firewall is, at the end of the day, a packet forwarder. The packet crosses a wire (or a fibre, or a virtual wire) as an Ethernet frame with a source MAC and a destination MAC. The firewall operator who can read that frame — or at least understand what is in it — diagnoses problems the operator who only knows the IP layer cannot.

This lesson covers what is on the wire, what MAC addresses are, how ARP maps an IP to a MAC, and why the ARP table on the firewall is sometimes the only place a Layer 2 incident is visible.

What is in an Ethernet frame?

An Ethernet frame carries a payload (almost always an IPv4 or IPv6 packet) between two MAC addresses on the same Layer 2 segment.

+----------+----------+----------+----------+----------+
|  Preamble| Dest MAC | Src MAC  | Ethertype| Payload  | FCS
+----------+----------+----------+----------+----------+
| 8 bytes  | 6 bytes  | 6 bytes  | 2 bytes  | 46-1500  | 4

The ethertype tells the receiving host what is inside the frame. 0x0800 is IPv4, 0x86DD is IPv6, 0x0806 is ARP, 0x8100 is 802.1Q VLAN tagging.

The MAC address is 48 bits, usually written as six pairs of hex digits separated by colons (aa:bb:cc:11:22:33). The first three bytes identify the manufacturer (the OUI); the last three are the device-unique portion. OPNsense’s interfaces each have a MAC, assigned by the driver; the firewall also has MAC addresses for its VLAN interfaces, CARP VIPs, and any bridges it owns.

Three kinds of Layer 2 traffic

Most of what crosses a firewall segment is unicast. ARP requests are broadcast. CARP advertisements and OSPF hello packets are multicast. ND (Neighbor Discovery) and IPv6 router advertisements are multicast too — a fact that has HA consequences the HA part of the course covers.

How ARP maps IP to MAC

ARP turns “next-hop IP” into “next-hop MAC”. On a Linux/FreeBSD host:

192.0.2.50 → 192.0.2.1 (next hop) → aa:bb:cc:11:22:33 (real MAC)

If the host does not have an entry for 192.0.2.1 in its ARP table, it broadcasts an ARP request:

"Who has 192.0.2.1? Tell 192.0.2.50."

The host that owns 192.0.2.1 replies:

"192.0.2.1 is at aa:bb:cc:11:22:33."

The requester stores the mapping in its ARP cache and uses it for every subsequent frame sent to 192.0.2.1. Entries age out — on FreeBSD the default is 20 minutes for a complete entry.

$ arp -an

? (192.0.2.1) at aa:bb:cc:11:22:33 on igb0 expires in 1185 [ethernet]
? (192.0.2.50) at 66:77:88:99:aa:bb on igb0 expires in 1198 [ethernet]
? (192.0.2.255) at ff:ff:ff:ff:ff:ff on igb0 expires in 0 [ethernet]

Why the firewall cares about ARP

OPNsense itself maintains an ARP table for every directly-attached network. Three things break that table in production:

1. An IP moves. A host’s IP is reassigned to a different host, or DHCP gives the same address to a different MAC, and the firewall continues to send frames to the old MAC until the entry ages out.
2. ARP spoofing. A malicious host sends gratuitous ARP replies claiming to own someone else’s IP. Until the cache expires, the firewall will hand traffic to the attacker. (This is exactly what Dynamic ARP Inspection on a managed switch is designed to stop.)
3. Asymmetric reachability. The firewall can ARP a host’s IP but cannot actually deliver frames to it because the host lives behind a different path than the firewall expects. The ARP entry will be present but packets to that host will silently fail.

In each case, the firewall’s PF state table will show traffic as “established” while the actual frame delivery fails. The firewall operator has to suspect the ARP table — not the firewall rule — to find the cause.

Reading a capture at Layer 2

A packet capture is the most direct way to see Layer 2.

$ tcpdump -nei igb0 arp

12:34:56.789012 aa:bb:cc:11:22:33 > ff:ff:ff:ff:ff:ff, ARP, Request who-has 192.0.2.1 tell 192.0.2.50, length 28
12:34:56.789345 66:77:88:99:aa:bb > aa:bb:cc:11:22:33, ARP, Reply 192.0.2.1 is-at 66:77:88:99:aa:bb, length 28

The first frame is a broadcast ARP request; every host on the segment sees it. The second is the unicast reply. If the reply never arrives, the firewall cannot send frames to that IP — and that looks, from above, like the firewall is dropping traffic.

Summary

• Frames carry MAC addresses, not IPs. The firewall kernel does the Layer 2 work; PF sees the resulting IP packet.
• ARP maps IP to MAC and ages out. Stale ARP is one of the most common silent failure modes.
• Multicast matters for HA: CARP and OSPF need a Layer 2 segment that carries multicast between peers.
• The firewall operator reads arp -an and tcpdump -nei to see Layer 2 evidence.