Lab: Read an Ethernet Frame and an ARP Table

Objective

By the end of this lab you will have watched a neighbour entry be born, mature, decay and be deliberately poisoned, and you will be able to answer the question that decides which team owns a routing incident: is the routing table wrong, or is the routing table right and the frame going to the wrong MAC?

Everything here is evidence-based. You will not conclude that ARP resolved because a ping succeeded; you will read the request, the reply, the resulting table entry, and the destination MAC in the frames that follow.

Architecture

Two VyOS routers on one Layer 2 segment, plus a management path that this lab never touches.

   management network (hypervisor NAT or lab LAN, DHCP)
        |                                   |
      eth0                                eth0
   +--------+                          +--------+
   |   r1   | eth1 ============== eth1 |   r2   |
   +--------+   lab-l2 (isolated)      +--------+
                                            |
                                          dum0
                                     198.51.100.1/32

eth1 on both routers must be on a bridge, vSwitch or virtual network that carries no other traffic and runs no DHCP server. A noisy segment is not fatal, but a quiet one makes every capture in this lab readable, which is the whole point.

The dum0 interface on r2 exists for one reason: without a destination that is not on the segment, you cannot demonstrate the single most important fact about Layer 2 forwarding, which is that the destination MAC and the destination IP of a forwarded packet refer to two different machines.

Requirements

• Two VyOS 1.5.x (circinus) virtual machines, r1 and r2, each with two network interfaces. 512 MB RAM and 4 GB disk each is ample; nothing in this lab is heavy.
• Both routers reachable over eth0 by SSH, or console access to both. The lab never changes eth0, so you should not be able to lock yourself out — but Task 1 makes you prove which interface carries your session before you configure anything, because “I thought eth1 was the lab interface” is the way this lab bites people.
• Two terminal sessions on r1. Several tasks run a capture in one session and generate traffic from the other. Two SSH windows, or tmux, or a console plus an SSH session — any pair works.
• tcpdump on both routers. It is present on a stock VyOS image: VyOS’s own monitor traffic interface ethN filter '...' operational command is a wrapper around it. This lab calls tcpdump directly because the -e flag, which prints the Ethernet header, is the entire subject of the first half and the wrapper’s job is to hide exactly that layer.
• No internet access is needed. No credentials beyond your own login. No production system is involved at any point.

Scenario

You are on call. A colleague reports that traffic from a customer segment to 198.51.100.1 stopped about twenty minutes ago. They have already checked the obvious thing and are certain the problem is not routing: the static route is present, show ip route agrees with the design document, the next hop is the correct address, and the interface is up. They want the transit provider paged.

Before you page anybody, you need to be able to distinguish two states that look identical from the routing table:

1. The router does not know where to send the packet. Layer 3.
2. The router knows exactly where to send the packet, and is addressing the frame to a MAC address that nothing on the segment owns. Layer 2.

This lab builds the second state on purpose, so that you have seen it once before you have to recognise it under pressure.

Tasks

Task 1: Record the starting state, and prove which interface is yours

Run this on both routers. It writes nothing and changes nothing; it exists so that Cleanup has something to restore to and so that you know which interface not to touch.

mkdir -p ~/arp-lab

ip -br addr show           > ~/arp-lab/pre-addr.txt
ip route show              > ~/arp-lab/pre-route.txt
ip neigh show              > ~/arp-lab/pre-neigh.txt

cat ~/arp-lab/pre-addr.txt

$ ip -br addr show

lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0             UP             198.51.100.61/24 fe80::5054:ff:fe3a:1101/64
eth1             UP             fe80::5054:ff:fe3a:1102/64

Now the part that matters. Find the interface your own session is arriving on, and write it in your journal as the interface you will not touch:

# The address you SSH'd to. Substitute yours:
MGMT_ADDR=198.51.100.61

ip -br addr show | grep -F "$MGMT_ADDR"

If that returns eth1, stop and re-cable the lab. Everything below assumes eth1 is the isolated segment and eth0 is your way in.

Task 2: Build the segment

On r1:

configure
set interfaces ethernet eth1 description 'LAB-L2'
set interfaces ethernet eth1 address '192.0.2.1/24'
set interfaces ethernet eth1 address '2001:db8:0:1::1/64'
compare
commit
exit

On r2:

configure
set interfaces ethernet eth1 description 'LAB-L2'
set interfaces ethernet eth1 address '192.0.2.2/24'
set interfaces ethernet eth1 address '2001:db8:0:1::2/64'
set interfaces dummy dum0 address '198.51.100.1/32'
commit
exit

Back on r1, give the router a reason to forward:

configure
set protocols static route 198.51.100.0/24 next-hop '192.0.2.2'
compare
commit
exit

Note what you did not type: save. The change is live and it is not persistent. Lesson vyos-v-02-candidate-active-saved calls this the source of the most common post-incident surprise on a VyOS box; here it is a feature, because it means a reboot is a guaranteed rollback.

Confirm r2 will actually forward for you. VyOS enables IPv4 forwarding by default, and the point of checking is that dum0 is reachable from r1 before you start reasoning about frames:

$ ping 198.51.100.1 count 3

PING 198.51.100.1 (198.51.100.1) 56(84) bytes of data.
64 bytes from 198.51.100.1: icmp_seq=1 ttl=64 time=0.582 ms
64 bytes from 198.51.100.1: icmp_seq=2 ttl=64 time=0.401 ms
64 bytes from 198.51.100.1: icmp_seq=3 ttl=64 time=0.418 ms

--- 198.51.100.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2031ms

If that fails, fix it now. Every remaining task assumes a working path, and a lab that starts broken teaches you nothing about the fault you are about to inject deliberately.

Task 3: Watch ARP resolve from a cold table

This is the observation the rest of the lab is built on. You are going to delete the neighbour entry, capture the segment, send exactly one packet, and read what the kernel had to do before that packet could leave.

In terminal A on r1, start the capture. It stops on its own:

SEG_IF=eth1

sudo timeout 30 tcpdump -nei "$SEG_IF" -c 6 arp

In terminal B on r1, clear the entry and send one packet:

SEG_IF=eth1
PEER=192.0.2.2

sudo ip neigh del "$PEER" dev "$SEG_IF" 2>/dev/null
ip neigh show "$PEER" dev "$SEG_IF"

ping "$PEER" count 1

Terminal A should show something with this shape:

$ sudo timeout 30 tcpdump -nei eth1 -c 6 arp

tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
11:04:22.118431 52:54:00:3a:11:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.2 tell 192.0.2.1, length 28
11:04:22.118902 52:54:00:7c:44:02 > 52:54:00:3a:11:02, ethertype ARP (0x0806), length 42: Reply 192.0.2.2 is-at 52:54:00:7c:44:02, length 28

Read it field by field, because every one of them appears again later:

• 52:54:00:3a:11:02 > ff:ff:ff:ff:ff:ff — source MAC and destination MAC. The request goes to the broadcast address because r1 does not yet know who to ask, so it asks the entire segment.
• ethertype ARP (0x0806) — ARP rides directly on Ethernet. There is no IP header in this frame at all, which is why an ARP request cannot cross a routed boundary: there is nothing for a router to route.
• Request who-has 192.0.2.2 tell 192.0.2.1 — the target IP and the sender IP, carried inside the ARP payload.
• The reply is unicast, 52:54:00:7c:44:02 > 52:54:00:3a:11:02. r2 learned r1’s MAC from the request it just received, so it has no reason to broadcast the answer.

Now read what the exchange produced:

$ ip -s neigh show 192.0.2.2 dev eth1

192.0.2.2 dev eth1 lladdr 52:54:00:7c:44:02 used 1/1/1 probes 0 REACHABLE

The MAC in the table is the MAC from the reply. Confirm that by eye against your capture — this is the first and simplest form of the correlation the whole lab is teaching.

Task 4: Watch the entry decay, and understand why it does not vanish

Leave the segment alone for a minute, then look again:

SEG_IF=eth1
PEER=192.0.2.2

for i in 1 2 3 4 5 6; do
  date +%T
  ip neigh show "$PEER" dev "$SEG_IF"
  sleep 15
done

You will see the state move from REACHABLE to STALE. It does not move to nothing, and it does not move to FAILED.

Record in your journal: the state at each sample, and roughly how long the entry stayed REACHABLE. You are building the intuition that a STALE entry is normal and healthy, so that when you find one during an incident you do not waste ten minutes treating it as the cause.

Task 5: Prove the destination MAC belongs to the next hop

This is the task that changes how you read a capture forever.

Terminal A on r1 — capture ICMP to the address behind r2, with link-layer headers:

SEG_IF=eth1
BEHIND=198.51.100.1

sudo timeout 30 tcpdump -nei "$SEG_IF" -c 4 "icmp and host $BEHIND"

Terminal B on r1:

BEHIND=198.51.100.1

ping "$BEHIND" count 2

$ sudo timeout 30 tcpdump -nei eth1 -c 4 'icmp and host 198.51.100.1'

11:09:03.441027 52:54:00:3a:11:02 > 52:54:00:7c:44:02, ethertype IPv4 (0x0800), length 98: 192.0.2.1 > 198.51.100.1: ICMP echo request, id 4521, seq 1, length 64
11:09:03.441598 52:54:00:7c:44:02 > 52:54:00:3a:11:02, ethertype IPv4 (0x0800), length 98: 198.51.100.1 > 192.0.2.1: ICMP echo reply, id 4521, seq 1, length 64

Line up the two halves of the first line:

In this lab those two happen to be the same physical box, which is convenient for checking your work: ip neigh show 192.0.2.2 dev eth1 on r1 gives you the MAC, and you can confirm it against ip -br link show eth1 on r2. Extend the topology by one hop and they stop being the same box, and the rule still holds — the destination MAC is always the next hop’s, and it is rewritten at every Layer 3 hop along the path.

Task 6: Break it — the routing table stays right, the traffic stops

Take a “healthy” snapshot first. You will compare against it in Task 7, and producing a before/after pair is the habit this lab is really teaching.

BEHIND=198.51.100.1
PEER=192.0.2.2
SEG_IF=eth1

{
  echo "=== healthy $(date -Is)"
  ip route get "$BEHIND"
  ip neigh show "$PEER" dev "$SEG_IF"
} > ~/arp-lab/healthy.txt

cat ~/arp-lab/healthy.txt

Now poison the mapping. 02:00:00:00:00:99 is a locally administered address that nothing on the segment owns, so this is exactly the state a router lands in when a peer’s NIC is swapped and the cache has not caught up:

$ sudo ip neigh replace 192.0.2.2 lladdr 02:00:00:00:00:99 dev eth1 nud permanent

nud permanent is what makes this a lab and not a race: a permanent entry is never probed and never ages out, so the fault stays put while you investigate it rather than healing itself thirty seconds in.

Now investigate, in the order a good on-call engineer actually would — cheapest and most-likely-to-exonerate first.

Is there a route?

$ ip route get 198.51.100.1

198.51.100.1 via 192.0.2.2 dev eth1 src 192.0.2.1 uid 1000
  cache

Yes. Identical to the healthy snapshot. The static route is present, the next hop is right, the egress interface is right. This is precisely the output your colleague in the Scenario used to conclude that routing was fine and the provider was at fault — and they were right about routing.

Is the link up, and is the interface passing anything?

SEG_IF=eth1

ip -br link show "$SEG_IF"
ip -s link show "$SEG_IF" | tail -n 4

Up, and the error and drop counters are not moving. A black hole at this layer is silent: the router is not dropping these packets, it is transmitting them perfectly to a machine that does not exist.

Does traffic actually flow?

BEHIND=198.51.100.1

ping "$BEHIND" count 3

It does not. Note the failure text: on a modern kernel a permanent entry pointing at an unused MAC produces plain timeouts, not Destination Host Unreachable, because from the kernel’s point of view nothing has gone wrong — it had a MAC, it used it, the frames left. That absence of an error message is itself the clue.

Now look at Layer 2.

$ ip -s neigh show 192.0.2.2 dev eth1

192.0.2.2 dev eth1 lladdr 02:00:00:00:00:99 PERMANENT

There it is. Compare the lladdr against ~/arp-lab/healthy.txt — one field differs across the entire investigation, and it is not in the routing table.

Confirm it on the wire, because a table is a claim and a capture is evidence. Terminal A:

SEG_IF=eth1
BEHIND=198.51.100.1

sudo timeout 20 tcpdump -nei "$SEG_IF" -c 4 "icmp and host $BEHIND"

Terminal B: ping 198.51.100.1 count 2.

You will see echo requests leaving with destination MAC 02:00:00:00:00:99, and no replies at all. Requests out, nothing back, and the outbound destination MAC is not your next hop’s: that triplet is the signature of this fault and it takes about fifteen seconds to establish.

Task 7: Fix it, and prove the fix

Remove the poisoned entry and let the protocol do its job:

$ sudo ip neigh del 192.0.2.2 dev eth1

Then re-run the exact commands from the healthy snapshot and diff them:

BEHIND=198.51.100.1
PEER=192.0.2.2
SEG_IF=eth1

ping "$BEHIND" count 2

{
  echo "=== repaired $(date -Is)"
  ip route get "$BEHIND"
  ip neigh show "$PEER" dev "$SEG_IF"
} > ~/arp-lab/repaired.txt

diff -u ~/arp-lab/healthy.txt ~/arp-lab/repaired.txt

The diff should show a changed timestamp line and nothing else of substance: the same route, the same MAC, and a state of REACHABLE or STALE depending on how recently the ping ran. That diff is your evidence that the fault was Layer 2 and is now gone — and it is a far better artefact for a ticket than “I flushed ARP and it started working.”

Task 8: The IPv6 half — compute the answer, then confirm it

ARP and NDP do the same job, and NDP’s version of “ask the whole segment” is deliberately narrower. Before you capture anything, work out what you expect to see.

The peer is 2001:db8:0:1::2. Its solicited-node multicast address is ff02::1:ff followed by the last 24 bits of that address. The last 24 bits of ...::2 are 00:00:02, so:

solicited-node multicast address:  ff02::1:ff00:2

The Ethernet multicast MAC for any IPv6 multicast address is 33:33 followed by the last 32 bits of the address. The last 32 bits of ff02::1:ff00:0002 are ff:00:00:02, so:

destination MAC on the NS frame:   33:33:ff:00:00:02

Write both in your journal before running the capture. Now check your arithmetic against the wire. Terminal A on r1:

SEG_IF=eth1

sudo timeout 30 tcpdump -nei "$SEG_IF" -c 4 icmp6

Terminal B on r1:

SEG_IF=eth1
PEER6=2001:db8:0:1::2

sudo ip -6 neigh del "$PEER6" dev "$SEG_IF" 2>/dev/null

ping "$PEER6" count 1

$ sudo timeout 30 tcpdump -nei eth1 -c 4 icmp6

11:21:40.882014 52:54:00:3a:11:02 > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: 2001:db8:0:1::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has 2001:db8:0:1::2, length 32
11:21:40.882515 52:54:00:7c:44:02 > 52:54:00:3a:11:02, ethertype IPv6 (0x86dd), length 86: 2001:db8:0:1::2 > 2001:db8:0:1::1: ICMP6, neighbor advertisement, tgt is 2001:db8:0:1::2, length 32

Three things to take from this, contrasted against the ARP capture in Task 3:

• The destination MAC is not the broadcast address. Only hosts that joined that solicited-node group process the frame; on a segment with a thousand hosts, ARP wakes all thousand and NDP wakes one.
• The frame is an ordinary IPv6 packet carrying ICMPv6. Unlike ARP, NDP has an IP header — which is why it can be filtered by an IPv6 firewall, and why an over-enthusiastic “drop all ICMPv6” rule breaks IPv6 in a way that looks nothing like a firewall problem.
• Both families land in the same table. Confirm it:

SEG_IF=eth1

ip neigh show dev "$SEG_IF"

One table, IPv4 and IPv6 entries side by side, the same state machine driving both.

Validation

Work through each of these and confirm the stated result before you tear the lab down.

• Your Task 3 capture contains an ARP request with destination MAC ff:ff:ff:ff:ff:ff and an ARP reply with a unicast destination MAC. If the reply is also broadcast, something on your segment is proxying — investigate before continuing.
• The lladdr in ip neigh show 192.0.2.2 dev eth1 on r1 equals the link/ether of eth1 on r2. Check it with ip -br link show eth1 on r2. They must match exactly.
• In your Task 5 capture, the destination MAC of the echo request is r2’s eth1 MAC while the destination IP is 198.51.100.1. You can state which machine each of those two fields refers to.
• diff -u ~/arp-lab/healthy.txt ~/arp-lab/repaired.txt shows no difference in the ip route get output. The routing table was correct throughout the fault, and you have the file to prove it.
• You can name, without looking it up, the one command whose output changed between the healthy and broken states. (ip neigh show.)
• Your hand-computed ff02::1:ff00:2 and 33:33:ff:00:00:02 match the addresses in the Task 8 capture. If they do not, redo the arithmetic before reading the answer off the capture — the arithmetic is the skill, the capture is the marking scheme.

Expected Outcome

At the end of the lab:

• r1 has eth1 addressed on 192.0.2.0/24 and 2001:db8:0:1::/64, and a static route for 198.51.100.0/24 via 192.0.2.2. r2 has the matching addresses and a dum0 carrying 198.51.100.1/32.
• Neither router has been saved, so a reboot of either one returns it to its pre-lab configuration. Lab 02 shows how to prove that claim rather than assert it.
• ~/arp-lab/ on r1 contains five files: the three pre-lab captures, plus healthy.txt and repaired.txt.
• Nothing on eth0 on either router has changed. diff your pre-addr.txt against a fresh ip -br addr show and the only differences are on eth1 and dum0.

Troubleshooting

The ARP request appears in the capture but no reply does. The request is leaving r1 and nothing is answering. Check that r2’s eth1 is actually on the same bridge or virtual network — this is by far the most common cause, and it is a hypervisor problem, not a VyOS one. Confirm with a capture on r2’s eth1: if r2 does not even see the request, the two eth1s are not on the same segment.

tcpdump prints You don't have permission to capture on that device. Run it under sudo. Packet capture needs privilege; the reason VyOS ships monitor traffic as an operational command is precisely so that the common case does not require the operator to think about this.

ping rejects count 3 as an invalid command. You are in configuration mode, not operational mode. Type exit first — the # prompt runs a different command set from the $ prompt, and that distinction is lesson vyos-v-01-configure-mode’s whole subject.

The neighbour entry never leaves REACHABLE. Something is still talking to the peer and refreshing the confirmation. Stop any running ping before you start the Task 4 sampling loop, and remember that the capture itself does not refresh anything — tcpdump is passive.

In Task 6, the ping starts working again on its own. You omitted nud permanent, so the kernel probed the entry, got no answer, and re-resolved it correctly. That is the system healing itself, which is good news in production and inconvenient here. Re-run the ip neigh replace with nud permanent.

commit fails with an address-in-use or overlap error. Something else on the router already holds an address in 192.0.2.0/24 — check ~/arp-lab/pre-addr.txt. Pick a different documentation prefix (203.0.113.0/24 is reserved for exactly this) and adjust the plan consistently on both routers.

The IPv6 ping fails with a destination-unreachable rather than timing out. Duplicate address detection may still be running, or it failed. Check ip -br addr show eth1 on both routers: an address shown as tentative or dadfailed is not usable, and dadfailed means two devices on the segment claim the same address.

Cleanup

Cleanup here has to restore, not merely delete — the standing rule for every lab in this course. Take the configuration back on r1:

configure
delete protocols static route 198.51.100.0/24
delete interfaces ethernet eth1 address '192.0.2.1/24'
delete interfaces ethernet eth1 address '2001:db8:0:1::1/64'
delete interfaces ethernet eth1 description
compare
commit
exit

And on r2:

configure
delete interfaces dummy dum0
delete interfaces ethernet eth1 address '192.0.2.2/24'
delete interfaces ethernet eth1 address '2001:db8:0:1::2/64'
delete interfaces ethernet eth1 description
compare
commit
exit

Read the compare output before each commit. If it removes anything you did not add — an address that was in pre-addr.txt, a description you did not write — stop and put it back before committing.

Then clear the runtime state the configuration tree does not own, on both routers:

SEG_IF=eth1

sudo ip neigh flush dev "$SEG_IF"
ip neigh show dev "$SEG_IF"

ip neigh entries are kernel state, not configuration. A delete in configure mode removes the address; it does not necessarily clear every neighbour entry learned through that interface, and a permanent entry certainly survives. This is worth internalising: anything you did with ip rather than with set is invisible to show configuration, invisible to compare, and will not be undone by a rollback.

Finally, confirm you are back where you started:

ip -br addr show > ~/arp-lab/post-addr.txt
ip route show    > ~/arp-lab/post-route.txt

diff -u ~/arp-lab/pre-addr.txt  ~/arp-lab/post-addr.txt  && echo 'ADDRESSES RESTORED'
diff -u ~/arp-lab/pre-route.txt ~/arp-lab/post-route.txt && echo 'ROUTES RESTORED'

Keep ~/arp-lab/ — it is small, and healthy.txt and repaired.txt are the deliverables. If you want the routers clean, reboot them: nothing was saved, so the saved configuration is still the one you started with.

Production notes

Map this lab onto a real change window and almost nothing carries over unchanged, which is the point of reading it twice.

The read-only half needs no window at all. ip neigh show, ip route get, ip -s link show and a bounded tcpdump are safe on a production router at any hour, and they are the first four things to run when someone claims a route is broken. There is no honest excuse for guessing when this evidence is fifteen seconds away.

The write half needs a window and an out-of-band path. ip neigh del, ip neigh flush and any ip neigh replace are service-impacting. Scope them to one address on one interface, log them in the incident channel as you run them, and never leave a nud permanent entry behind.

A tcpdump on a loaded router is not free. A capture on a busy interface with no filter costs CPU on the control plane at exactly the moment you can least afford it. Always filter to the conversation you care about, always bound it with -c or timeout, and prefer VyOS’s monitor traffic interface ethN filter '...' when you do not specifically need the link-layer view.

The evidence pair is the deliverable. healthy.txt and repaired.txt took thirty seconds to produce and they are what turns “I flushed ARP” into a defensible change record. Build the habit here, where the stakes are zero.

What You Learned

• ARP resolution is two frames, and you have read both. A broadcast request with ethertype 0x0806 and no IP header, and a unicast reply. You watched the entry that resulted appear in the neighbour table with the MAC from the reply.
• STALE is health, not sickness. The kernel keeps a mapping past its confirmation lifetime and revalidates lazily, and it randomises the lifetime so that a segment full of neighbours does not expire in unison. You will not misdiagnose a STALE entry as the cause of an outage.
• The destination MAC is the next hop’s; the destination IP is the destination’s. You proved it from a single capture line where the two fields referred to two different interfaces, and that one line is the model for every routed hop on every network you will ever debug.
• A correct routing table is not evidence that packets are being delivered. You produced a fault in which ip route get was right throughout, ip -s link showed no errors, and every packet was black-holed — and you separated Layer 2 from Layer 3 with one command and one capture.
• Runtime state and configuration are different things. ip neigh replace left no trace in show configuration, would survive a rollback, and had to be undone explicitly. Anything you change with ip on a VyOS box is invisible to the configuration model.
• NDP is ARP with a narrower blast radius. You computed the solicited-node multicast address and its Ethernet MAC from the peer’s address before capturing them, and confirmed both on the wire.