Static Route Exists but Traffic Fails · Break/Fix · VyOS

Reported symptoms

edge1 is the DC router that faces the payments zone. It has one static route that matters here — 10.90.0.0/16 via 192.0.2.20, the inside address of the payments firewall pair — and 192.0.2.20 sits on the transit VLAN that edge1 reaches through eth1. There is no recursion involved: the next hop is on a directly connected subnet.

At the weekend the firewall team replaced that chassis. Same addresses, same cabling, same policy, new hardware. They validated their side on Sunday afternoon and closed the window.

At 08:05 on Monday every batch job that writes into the payments zone began timing out. Timing out, not being refused — the sessions hang until the client gives up, which is what a far-end problem looks like from a log file.

Three facts arrived over the next hour, and none of them pointed at edge1:

Traffic in the other direction is perfect. The 08:15 file drop from the payments zone landed on time, and a monitoring poll originating inside 10.90.0.0/16 has not missed a single sample all morning. Whatever is broken, it is not the link and not the firewall’s policy.
The firewall team’s new chassis reports zero packets received from edge1. Not denied, not dropped by a rule, none at all — while its counters for the other devices on the same VLAN increment normally.
Nothing changed on edge1. show system commit goes back eleven weeks, and the routing is verifiably the same routing that worked on Friday.

On the router itself, the first two checks both came back healthy. The route is in the table with its markers. The kernel agrees with the route. By 09:00 the working theory was that the replacement chassis had a policy defect, because the local routing had been confirmed twice by two people.

Evidence provided

Read-only / Safeselected, installed, and eleven weeks old — this route is not the fault

vyos@edge1:~$ show ip route 10.90.0.0/16

Codes: K - kernel route, C - connected, S - static, R - RIP,
     O - OSPF, B - BGP, > - selected route, * - FIB route

S>* 10.90.0.0/16 [1/0] via 192.0.2.20, eth1, 11w2d

Illustrative output

Read-only / Safethe thing that actually decides, agreeing with the thing that was configured

vyos@edge1:~$ ip route get 10.90.4.11

10.90.4.11 via 192.0.2.20 dev eth1 src 192.0.2.2 uid 1000
  cache

Illustrative output

Read-only / Safeloss with no error line — read what is missing here, not what is present

vyos@edge1:~$ ping -c 3 192.0.2.20

PING 192.0.2.20 (192.0.2.20) 56(84) bytes of data.

--- 192.0.2.20 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2043ms

Illustrative output

Read-only / Safetwo neighbours the kernel maintains, and one it has been told

vyos@edge1:~$ ip neigh show dev eth1

192.0.2.1 lladdr 52:54:00:1b:8f:20 REACHABLE
192.0.2.9 lladdr 52:54:00:44:d1:02 STALE
192.0.2.20 lladdr 52:54:00:9d:1a:7c PERMANENT

Illustrative output

Read-only / Safesixty seconds of ARP while the pings fail, and not one request for 192.0.2.20

vyos@edge1:~$ sudo tcpdump -nei eth1 -c 20 arp

09:41:02.118447 52:54:00:1b:8f:20 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806),
Request who-has 192.0.2.2 tell 192.0.2.1, length 46
09:41:02.118501 52:54:00:0e:33:a1 > 52:54:00:1b:8f:20, ethertype ARP (0x0806),
Reply 192.0.2.2 is-at 52:54:00:0e:33:a1, length 28
09:41:07.664190 52:54:00:44:d1:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806),
Request who-has 192.0.2.1 tell 192.0.2.9, length 46

Illustrative output

Read-only / Safethe segment answers immediately — with a different MAC from the one in the table

vyos@edge1:~$ sudo arping -I eth1 -c 3 192.0.2.20

ARPING 192.0.2.20 from 192.0.2.2 eth1
Unicast reply from 192.0.2.20 [52:54:00:C3:0E:41]  0.712ms
Unicast reply from 192.0.2.20 [52:54:00:C3:0E:41]  0.688ms
Unicast reply from 192.0.2.20 [52:54:00:C3:0E:41]  0.701ms
Sent 3 probes (1 broadcast(s))
Received 3 response(s)

Illustrative output

Read-only / Safewhatever installed that entry, the configuration did not

vyos@edge1:~$ show configuration commands | match arp

Work the evidence before reading on

Every routing command on this router returns a healthy answer, and the traffic still does not arrive. That combination is the whole exercise.

The route is selected and installed, and ip route get confirms the kernel will send payments traffic out eth1 towards 192.0.2.20. What has to happen to a packet after the forwarding decision is made and before it reaches the wire, and which command shows you that step?
Read the ping output for what is not in it. A router that cannot resolve its next hop prints something specific and prints it fast. This one printed nothing but loss. What does that difference tell you about whether the packets were sent?
Three neighbour entries, two of which the kernel is maintaining and one of which it is not. What does the third state mean, what does the kernel do to the first two that it will never do to the third, and which of those behaviours would have fixed this by itself?
Sixty seconds of packet capture produced ARP for two other hosts and not one request for 192.0.2.20, during a period when pings to that address were failing every second. Silence is the strongest evidence in this ticket — what does it prove about what the kernel believes?
arping gets an answer in under a millisecond from a MAC that does not appear in the neighbour table. Why does that tool succeed where ping failed, and what is the difference between what the two commands consult?

Then the question that resolves it: traffic from the payments zone to the DC works perfectly. On a shared segment, what single kind of fault breaks one direction and leaves the other untouched?

Root cause

1. A hand-installed neighbour entry, and PERMANENT means what it says

In June this VLAN had a fault, and an engineer got traffic moving again by pinning the firewall’s MAC from the shell:

Configuration changethe command from the June change record — note what it does not say

vyos@edge1:~$ sudo ip neigh replace 192.0.2.20 lladdr 52:54:00:9d:1a:7c dev eth1

With a link-layer address and no explicit nud state, that installs the entry as PERMANENT. A permanent entry is not a cache line, it is an assertion. The kernel never ages it, never probes it, and never lets an ARP reply overwrite it. It stays exactly as typed until somebody deletes it or the box reboots.

At the weekend the chassis was replaced. Same address, new NIC, new MAC. Every other device on that VLAN learned the new MAC within seconds of the first frame, because their entries are dynamic and dynamic entries are revalidated. edge1 did not, because its entry is not permitted to change.

So the router is doing exactly what it was told: encapsulating every packet for 10.90.0.0/16 in a frame addressed to 52:54:00:9d:1a:7c, a MAC that is now sitting in a box on a shelf.

2. Every layer a routing engineer can see is healthy

This is why two people confirmed the routing and moved on. The static route is in the RIB, selected, installed in the FIB, and ip route get — the kernel’s own answer — names the right interface and the right next hop. Nothing in the routing subsystem is wrong, and no amount of staring at show ip route will find this, because address resolution happens after the routing decision and show ip route has no visibility into it.

The ping output carries the tell, in what it does not print. When a next hop cannot be resolved at all, the local kernel gives up and ping reports Destination Host Unreachable from the router’s own address, usually within a second. Here there was no such line: just three probes sent, nothing received. The packets went out. They were addressed to nobody.

3. Silence on the wire is the finding

Sixty seconds of tcpdump -nei eth1 arp during active failure produced ARP for two other hosts and not a single request for 192.0.2.20.

That is the most decisive piece of evidence in the incident, and it is a negative. A router that cannot resolve a next hop is noisy about it — the kernel retries, and the requests are all over the capture. Silence means the kernel is not asking, and the only reason it would not ask is that it believes it already knows.

arping cuts through it because it does not consult the neighbour table at all. It builds an ARP request itself, puts it on the interface, and reports what comes back, which is why it returned the correct MAC on the first probe while ping was still failing.

4. Why the other direction never broke, and why the far end saw nothing

The firewall resolves edge1 the ordinary way: it ARPs, edge1 answers, and it caches a correct entry. Nothing about this fault touches that path, so every session the payments zone initiates works perfectly — including the file drop and the monitoring poll that were both being read as evidence of health.

One-way reachability across a shared segment is the signature of a mapping problem rather than a path problem, and it is worth learning as a shape: cabling, VLANs and firewall policy tend to break both directions, whereas an address-to-MAC mapping is per-device and per-direction.

The far end’s evidence is a second consequence of the same mechanism. A frame addressed to a MAC the chassis does not own is discarded by the NIC’s own filter before any software on it is aware that a packet arrived. The switch may even flood it to their port, since no entry for the dead MAC exists in its forwarding database, and the chassis will still count nothing. “We see nothing from you, not even drops” was true, precise, and unhelpful, because it sounds like an accusation and is actually a symptom.

Under the hood

The Linux neighbour subsystem is a state machine, and this incident is a disagreement about which states it may enter.

A dynamic entry cycles. It is REACHABLE while recently confirmed, drops to STALE when the confirmation ages out, and on the next packet moves through DELAY into PROBE, where the kernel sends unicast ARP to re-confirm it. If nobody answers, it becomes FAILED and the kernel starts broadcasting again. Every one of those transitions is an opportunity to learn a new MAC, which is why a chassis swap is normally a non-event: the old mapping fails validation and the next broadcast gets the new answer.

PERMANENT sits outside that cycle. There is no timer, no probe, and no path back into the machine, because the entry is a statement by the operator rather than an observation of the network. That is the entire point of the state, and it is a legitimate one — a genuine static ARP entry for a device that cannot ARP for itself is a real requirement. What makes it dangerous is that the state is invisible unless you look for it, and the command that produces it does so as its default rather than on request.

Two details are worth carrying away. ip neigh replace with an lladdr and no nud clause installs PERMANENT; adding nud stale instead installs an entry the kernel is allowed to revalidate and correct, which would have made this incident a ninety-second blip at the weekend. And the state is filterable: ip neigh show nud permanent prints exactly the entries that cannot heal themselves, on any Linux box, which makes the fleet sweep a one-liner.

A workaround with no expiry is a fault with a delayed fuse

The June command was a reasonable thing to do at the time. It restored traffic during an incident, and the change record even says it should be removed. What made it dangerous is that it had no mechanism to remove itself and no mechanism to be noticed.

It is invisible to show configuration commands, to compare, to the nightly configuration backup and to the config-as-code diff, because it never entered the configuration tree — the same blind spot as an edit made by hand in vtysh. A reviewer comparing Friday’s configuration with Monday’s would find them identical and be right.

It also would not have survived a reboot, and that is the part worth sitting with. An override that vanishes on reboot is not a fix; it is a bet that nothing will reboot before somebody remembers. Here the bet was collected by a hardware change instead, eight weeks later, with the original engineer on a different team.

The rule that follows is narrow and cheap: anything you do outside the configuration tree gets an owner, an end time, and a line in the ticket that somebody will read. VyOS gives you commit-confirm for exactly this reason on the configuration side. On the shell side, the deadline has to be human.

Resolution

Prove the segment can still resolve the address before you touch anything. arping -I eth1 -c 3 192.0.2.20 bypasses the neighbour table, answers in three seconds, and hands you the current MAC as a side effect.
If arping returns nothing, stop and re-scope. The pin is then the only thing that was ever making this path work, the fault is on the segment rather than in routing, the owner is the switch team, and the override stays in place with an explicit end time on the ticket.
With an answer in hand, delete the override and only the override: ip neigh del 192.0.2.20 dev eth1. The next packet towards the payments zone triggers a normal ARP request and the reply installs a dynamic entry.
Refuse the shortcut of pinning the new MAC. ip neigh replace with the current address would restore traffic in one command and rearm the identical trap, with its fuse handed to whoever next replaces that chassis.
Refuse the broad flush. ip neigh flush dev eth1 on a transit interface discards every neighbour on the segment, including the BGP peer, and buys a burst of re-resolution and a brief queue on every next hop to fix one entry.
Confirm traffic from a real client rather than from the router, then ask the firewall team to confirm inbound packets from edge1 on their counters.
Sweep the fleet before closing: ip neigh show nud permanent on every router. Any entry there that no configuration node asks for is the same landmine with a different fuse.
Close NET-4471 properly. The June note said the pin should be removed; the reason it survived eight weeks is that saying so on a ticket is not a mechanism.

Verification

ip neigh show 192.0.2.20 dev eth1 reports REACHABLE or STALE with the MAC that arping returned. The state is the test, not the ping: a correct mapping today plus a state the kernel may maintain is what makes it stay correct.
ip neigh show nud permanent on edge1 returns nothing, because no configuration node on this router asks for a static ARP entry.
A real client in the DC opens a real session into 10.90.0.0/16. The router pinging its own next hop tests one frame on one segment; it is not the same claim.
The firewall reports inbound packets from edge1 on its counters. That is the only evidence in this incident not produced by the box that produced the fault.
tcpdump -nei eth1 arp now shows a request for 192.0.2.20 when the entry ages, and an answer to it. Silence there was the finding; silence again would mean the pin is back.
The check can fail. On a lab router, install a permanent entry with a deliberately wrong MAC and confirm the sweep reports it, so you know what the failing state looks like on the check you now depend on.
The other two devices on the transit VLAN still forward normally, confirming that nothing in the resolution touched neighbours that were never part of the fault.

Prevention

Give every out-of-tree change an owner and an end time. VyOS has commit-confirm because an unattended configuration change needs a deadline; a command typed into the shell has no equivalent, so the deadline has to be written on the ticket with a name against it.
Sweep for the class, not the instance. ip neigh show nud permanent should be empty on any router whose configuration contains no static ARP, and that check would have raised this in June rather than in August.
Treat a device replacement on a shared segment as a change on every device on that segment. The team doing the swap can validate only their own box; the neighbours’ view of it is checked on the neighbours, and the check is a MAC diff — show arp before, show arp after, compare.
Record MACs in the change record for a chassis swap. A diff is a fact; a recollection of what the old MAC was is an argument.
Test both directions on every connectivity fault. One-way reachability is the signature of a Layer 2 mapping problem, and a ping run from one end cannot see it. Here the working direction was read as evidence of health for an hour.
Keep the two questions apart in habit and in runbooks. Whether the router chose the right next hop is show ip route; whether it can address a frame to that next hop is ip neigh. Only the second one is about this fault, and only the first one was run.

What if the June engineer had typed `nud stale` instead?

The incident would not exist, and the difference is one clause on one command.

An entry installed as STALE is inside the state machine. The kernel revalidates it on the next packet, sends a probe, and when the probe goes unanswered it falls back to broadcasting for the address. The new chassis would have answered, the entry would have been rewritten with the new MAC, and the whole event would have been a few seconds of loss on Sunday afternoon that nobody logged.

That is a useful way to think about the choice generally. A dynamic entry is an observation, and observations get corrected when reality moves. A permanent entry is an assertion, and assertions do not. Overriding the network with an assertion is occasionally the right call — a device that genuinely cannot ARP, a lab harness, a deliberate pin recorded in configuration — but it converts a self-healing subsystem into one that depends on human memory.

The related question is what would have happened had the old chassis been left powered on and plugged in with its address intact. Then two devices would answer for 192.0.2.20, the dynamic entries around the VLAN would flip between two MACs as replies raced each other, and traffic would fail intermittently for everyone rather than completely for one router. That is a harder incident to diagnose than this one, and arping is again the tool that exposes it, because replies from two different MACs to one request is not a thing the neighbour table will ever show you.

Why this matters

Routing engineers debug downwards from the routing table, and this fault sits one layer below where that habit stops. The RIB was right, the FIB was right, the kernel’s own forwarding answer was right, and every packet still went into a void — because the last translation before the wire, from next-hop address to next-hop MAC, is not part of routing at all and is not visible in any routing command.

The transferable move is to know where your evidence stops. show ip route answers “which next hop”, ip route get answers “which interface and which next hop, according to the kernel”, and ip neigh answers “which MAC”. A fault between the second and third question will pass every check you run at the first two, and the only thing that finds it is knowing that the third question exists.

The second lesson is about what a workaround becomes. Nothing here was incompetent: the June fix was correct in the moment, the chassis swap was executed and validated properly, and the routing was verified twice by people who read the output correctly. The incident happened in the gap between those things — a change that lived outside the configuration, so no review could see it, and outside anyone’s memory, so no procedure could remove it. Estates accumulate those. The cheap defence is a periodic sweep for state that no configuration explains, because the value of that sweep is not the entry it finds today, it is the eight weeks of latent fault it removes.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. A hand-installed neighbour entry, and PERMANENT means what it says

2. Every layer a routing engineer can see is healthy

3. Silence on the wire is the finding

4. Why the other direction never broke, and why the far end saw nothing

Resolution

Verification

Prevention