ARP and IPv6 neighbour discovery

IP packets travel inside Ethernet frames. To send an IP packet to a host on the same subnet, the sender needs the destination MAC address. The mapping from IP to MAC is held in a cache called the neighbour cache (IPv4 calls this ARP, IPv6 calls it ND).

ARP (IPv4)

When host A (10.0.0.1) wants to talk to host B (10.0.0.5):

1. A checks the ARP cache (/proc/net/arp or ip neigh).
2. If B is not cached, A broadcasts an ARP request: “who has 10.0.0.5? Tell 10.0.0.1” - source MAC A, broadcast destination.
3. B replies with its MAC: “10.0.0.5 is at aa:bb:cc:dd:ee:ff”.
4. A caches the binding and proceeds with the IP packet.

The cache is dynamic: entries age out after a few minutes and are re-learned. That re-learning is a feature, not overhead: it is how a host follows a MAC address that has moved.

Inspect the ARP / neighbour cache

ip neigh show
ip -4 neigh show
ip -6 neigh show
arp -a                  # legacy command, kept for compat
cat /proc/net/arp

Output looks like:

10.0.0.1 dev eth0 lladdr aa:bb:cc:00:00:01 REACHABLE
10.0.0.5 dev eth0 lladdr aa:bb:cc:dd:ee:ff STALE
fe80::1 dev eth0 lladdr aa:bb:cc:00:00:ff router REACHABLE

The state matters:

A FAILED or persistent INCOMPLETE entry means the host cannot reach the gateway - the next layer up (IP routing) cannot function.

Static ARP entries

# 10.0.0.50 is an appliance, not the gateway - see the warning below
ip neigh add 10.0.0.50 lladdr aa:bb:cc:00:00:0a dev eth0 nud permanent
ip neigh replace 10.0.0.50 lladdr aa:bb:cc:00:00:0b dev eth0 nud permanent
ip neigh del 10.0.0.50 dev eth0

Use replace rather than change to update an entry: change fails if the entry does not already exist, replace adds or updates it.

nud permanent means exactly what it says. Per man 8 ip-neighbour, the entry is valid forever and can only be removed administratively. The kernel never probes it, never ages it out, and never updates it from a gratuitous ARP.

That last point is the whole risk. A permanent entry pins one MAC address for the lifetime of the host. If the real MAC changes, the host keeps sending frames to a MAC that no longer answers, and nothing in the system notices: no FAILED state, no log line, no metric. Recovery is manual, on every host, over a network path that no longer works.

So a permanent entry is only defensible for a device whose MAC is fixed by contract and which has no failover partner - a serial console server, a lights-out board, a lab appliance. Record every such entry in the host inventory, because the next person to replace that hardware will not find it by accident.

Entries added by hand do not survive a reboot. If one is genuinely justified, declare it in the network configuration (a NetworkManager dispatcher script, a systemd unit ordered after the interface, or the distribution’s own hook) so that it is version-controlled and visible.

ARP announcements and gratuitous ARP

When a host’s IP changes (e.g. failover), it sends a gratuitous ARP: an ARP reply that was not asked for, with both source and target IP set to the new IP. This forces every other host on the LAN to update its cache immediately, without waiting for the stale entry to expire.

Keepalived and many load balancers use gratuitous ARP for VIP failover. If your VIP fails to migrate, check that gratuitous ARP is enabled and not blocked by a switch feature such as “port security” or “MAC limiting”.

The same rule applies to IPv6. A nud permanent neighbour entry ignores unsolicited neighbour advertisements exactly as it ignores gratuitous ARP.

IPv6 neighbour discovery (ND)

IPv6 replaces ARP with the Neighbour Discovery Protocol (NDP, RFC 4861):

• NDP uses ICMPv6 messages rather than a separate protocol.
• Router Solicitation (RS) - hosts ask “is there a router here?”
• Router Advertisement (RA) - routers announce prefixes and themselves.
• Neighbour Solicitation (NS) - the IPv6 equivalent of an ARP request, sent to the solicited-node multicast address.
• Neighbour Advertisement (NA) - the response.

The ip -6 neigh show command shows the same cache as for IPv4, with the same states. Diagnose IPv6 connectivity at layer 2 the same way you would diagnose IPv4.

ip -6 neigh show
ping6 fe80::1%eth0    # ping link-local with zone identifier

The %eth0 (zone identifier) is required when pinging a link-local address because the same address may exist on multiple interfaces.

Diagnose “no route to host”

No route to host is EHOSTUNREACH, and despite its wording it is usually not a routing failure. It has three common causes, in order of likelihood:

1. Neighbour resolution failed for the destination on a directly connected subnet. ip neigh show <dst> reports FAILED, or sits in INCOMPLETE. This is the layer-2 case and the one this lesson is about.
2. A firewall rejected the packet with icmp type host-unreachable (nftables reject with, iptables --reject-with icmp-host-unreachable). The neighbour cache is healthy; the answer is in the ruleset at one end or the other.
3. The route itself is wrong or absent. This is the rarest of the three, and it is the only one that is genuinely layer 3. A completely missing route usually surfaces as ENETUNREACH - “Network is unreachable” - not this error.

Two commands separate them, in this order:

DST=10.0.0.20
ip route get "$DST"     # route present, right interface, right src?
ip neigh show "$DST"     # FAILED or INCOMPLETE -> cause 1

If ip route get answers cleanly and ip neigh shows FAILED, the route is fine and layer 2 is the problem: wrong VLAN, switch port in the wrong VLAN, ARP blocked by a port security feature, or the peer’s NIC down. If both look healthy, you are in cause 2 - read the rulesets, not the caches.

One state is worth checking explicitly here. A gateway entry shown as PERMANENT while off-subnet traffic is dropped is the signature of a pinned MAC that no longer exists. Confirm with arping -I eth0 10.0.0.1 and compare the MAC that answers against the pinned one. If they differ, replace the entry with the live MAC and hand it back to the kernel:

ip neigh replace 10.0.0.1 lladdr aa:bb:cc:00:00:02 dev eth0 nud reachable

Knowledge check