Skip to main content
RunBook Academy

← All break/fix scenarios in VyOS

advancedvyos-wireguard~30 min

WireGuard Up but No Traffic

Reported symptoms

  • A newly built VLAN at SITE-A, 192.168.11.0/24, cannot reach anything at SITE-B across the WireGuard tunnel
  • The tunnel is healthy by every measure the team trusts: recent handshake, transfer counters moving in both directions, monitoring green
  • Hosts on the older VLAN 192.168.10.0/24 reach SITE-B normally and always have
  • The new VLAN reaches the internet normally, so the VLAN itself is plainly working
  • Name resolution for SITE-B hostnames works from the new VLAN, which was read as proof the tunnel was fine
  • Connections from the new VLAN do not fail — they hang. No refusal, no ICMP unreachable, no timeout message from the router
  • SITE-B was not changed. The only change was at SITE-A, and it was reviewed and committed cleanly
  • No firewall counter at either end moves when the traffic is sent, including the default-drop counters

Evidence

  • · `show interfaces wireguard wg0 summary` on SITE-A — `latest handshake` under a minute, transfer non-zero in both directions
  • · `show ip route 192.168.20.0/24` on SITE-A — present, via wg0
  • · `show interfaces wireguard wg0 allowed-ips` on SITE-A — `10.10.10.0/30`, `192.168.20.0/24`
  • · `show interfaces wireguard wg0 allowed-ips` on SITE-B — `10.10.10.0/30`, `192.168.10.0/24`
  • · `sudo tcpdump -ni wg0 -c 4` on SITE-A — packets from 192.168.11.50 to 192.168.20.10 entering the tunnel
  • · `sudo tcpdump -ni wg0 -c 4` on SITE-B — traffic from 192.168.10.0/24 only; nothing from 192.168.11.0/24
  • · `show firewall ipv4 name TUNNEL-IN` on SITE-B — every counter unchanged, including `default-action drop`
  • · From a host on 192.168.11.0/24: ping to 192.168.20.10 returns nothing at all — no reply and no ICMP unreachable
  • · `show ip route 192.168.11.0/24` on SITE-B — no such route
Diagnosis and resolutionclick to reveal

Root cause

SITE-B is discarding the packets after decrypting them, because their source address is not inside the allowed-ips it holds for the SITE-A peer. On WireGuard, allowed-ips does two jobs from one list: outbound it selects which peer's key encrypts a packet, and inbound it is the policy that decides whether a decrypted packet is allowed to claim the source address it carries. A packet that decrypts correctly and arrives from a source outside that list is dropped between decryption and the interface — before any firewall chain, which is why not one counter at SITE-B moves, and in complete silence, because answering would confirm to an unauthenticated sender that a key is valid. The change at SITE-A was correct and complete for SITE-A: a new VLAN, an address, a route, and a tunnel whose allowed-ips already covered the destination it needed to reach. Nothing at SITE-A had to change, and nothing did. What was missed is that allowed-ips is a per-pair agreement rather than a local setting: what one end sends must be inside what the other end will accept from it, in both directions, and adding a prefix behind one site is therefore a change at the other site. The absence of an ICMP unreachable is the tell that points across the tunnel. A gap in the local allowed-ips produces an immediate destination unreachable on the router itself, because the WireGuard device has no key to encrypt the packet with and cannot silently drop it; a gap at the far end produces nothing at all. Same misconfiguration, one loud symptom and one silent one, depending on which router carries it.

Remediation

Two nodes are missing at SITE-B and both are needed: `192.168.11.0/24` in the allowed-ips of the SITE-A peer, and a static route for `192.168.11.0/24` out of wg0 so the reply is routed back into the tunnel rather than out of the default route. Sequence them so the intermediate state is the harmless one. Add the route first: on its own it changes nothing observable, because no packet from the new prefix is being accepted yet. Add the allowed-ips second. The reverse order is the one that does damage — with allowed-ips widened and no return route, SITE-B accepts the request, delivers it, and the reply for 192.168.11.50 follows SITE-B's default route and leaves the site in clear toward the internet. That is worse than the fault being repaired, and it is not hypothetical: allowed-ips installs no routes, so the return path only exists if someone writes it. Putting both nodes in one commit narrows the window but does not remove the ordering question, because a commit that fails partway has already applied the scripts that ran before the failure. Add the specific prefix rather than a summary. Allowed-ips is a grant of authority as well as a route, and on an interface carrying several peers the prefix-to-peer mapping is one-to-one — a summary given to this peer takes those prefixes away from the others, with every line still present in the configuration and nothing looking wrong. If the far-end change cannot be made now, hold: leave the new VLAN off the tunnel with a named owner and a stated end time, rather than reaching for a source-translation workaround that makes the traffic acceptable by lying to SITE-B about where it came from.

Verification

The first check is the capture pair, because it is the one that failed and the one that will fail again. The same packet must be visible on wg0 at both ends: `sudo tcpdump -ni wg0` on SITE-A showing 192.168.11.50 to 192.168.20.10, and the same flow on SITE-B. Absence on the far end was the entire fault, and presence there is the only direct evidence that the crypto policy now admits it. Second, confirm the traffic now reaches the firewall, which is a different claim from "it works": the TUNNEL-IN rule counters at SITE-B must advance for the new source prefix, where previously nothing moved anywhere. Third, read the kernel rather than the configuration — `show interfaces wireguard wg0 allowed-ips` reports what the running interface holds, and on an interface with several peers that can differ from what the configuration reads, which is precisely the failure a widened prefix causes. Fourth, test from a host in the new prefix and from a host in the old one, in both directions, and confirm the reply arrives over the tunnel rather than merely arriving. Fifth, confirm the other peers on that interface still hold their own prefixes, so the fix has not quietly taken traffic from a third site. Finally, prove the check can fail: on a lab pair, remove the prefix from the far end's allowed-ips and confirm the capture pair shows the packet at one end and not the other. Every check the team already had returned green throughout this incident, so a check that has only ever passed is not yet evidence of anything.

Prevention

Write allowed-ips changes as paired changes and refuse the single-ended version. The change template for "a new prefix is now routed behind this site" has to name three things at the far end — the allowed-ips entry for this peer, the static route out of the tunnel interface, and who commits them — because the local half of the work is complete and correct without any of them, which is exactly what makes it easy to stop there. Add an automated validation that diffs each site's routed prefixes against the far end's allowed-ips for that peer and fails the pipeline on a difference; this fault is machine-detectable from two configurations and is invisible in either one alone. Monitor a host inside each prefix rather than one probe per tunnel: a monitor that pings the far end's tunnel address is green for every allowed-ips fault there is, and it was green here. Make the capture pair on wg0 the standard first move for "tunnel up, no traffic", ahead of reading firewall rules, because it distinguishes in one step between a packet dropped by a rule and a packet that never reached one. And treat the ICMP that did not arrive as evidence in its own right: silence on the LAN side points at the far end, a destination unreachable from your own router points at the near end, and knowing which saves an hour of reading the wrong configuration.

Reported symptoms

SITE-A brought up a new VLAN for a payments application, 192.168.11.0/24, behind the same router that already carries 192.168.10.0/24. The change was small, reviewed and committed cleanly:

set interfaces ethernet eth0 vif 11 address '192.168.11.1/24'
set interfaces ethernet eth0 vif 11 description 'payments'

The site-to-site tunnel to SITE-B has been up for two years:

SITE-A  192.168.10.0/24  +  192.168.11.0/24 (new)   wg0  10.10.10.1/30
SITE-B  192.168.20.0/24                             wg0  10.10.10.2/30

The application cannot reach its database at SITE-B. Everything around it can:

  • The tunnel is healthy on every measure the team trusts. Handshake under a minute old, transfer counters moving in both directions, the tunnel monitor green all week.
  • Hosts on 192.168.10.0/24 reach SITE-B exactly as they always have.
  • The new VLAN reaches the internet normally, so the VLAN is not broken.
  • Name resolution for SITE-B hostnames works from the new VLAN. The resolver lives on 192.168.10.0/24 and does the lookup on the application’s behalf, which made “DNS works, so the tunnel is fine” a very persuasive and completely irrelevant observation.
  • Connections do not fail. They hang. No refusal, no ICMP unreachable, no message from any router.
  • SITE-B was not touched. Its configuration is unchanged since the last audit.

Evidence provided

Read-only / Safethe tunnel is up and carrying traffic
vyos@SITE-A:~$ show interfaces wireguard wg0 summary
interface: wg0
public key: rSuuvkAuSoyqvogoA5OJjYBbJKKaYS13pWs4K69PTOE=
listening port: 51820

peer: cVn4T2sM8xQ6yB1hJ0dR7kL3pW9zA5eG2uY8iO4nX1c=
endpoint: 198.51.100.20:51820
allowed ips: 10.10.10.0/30, 192.168.20.0/24
latest handshake: 47 seconds ago
transfer: 84.21 MiB received, 61.09 MiB sent

Illustrative output

Read-only / Safethe new VLAN enters the tunnel; the old VLAN gets answers
vyos@SITE-A:~$ sudo tcpdump -ni wg0 -c 4
IP 192.168.11.50.51422 > 192.168.20.10.5432: Flags [S], length 0
IP 192.168.11.50.51422 > 192.168.20.10.5432: Flags [S], length 0
IP 192.168.10.31.42188 > 192.168.20.10.5432: Flags [P.], length 118
IP 192.168.20.10.5432 > 192.168.10.31.42188: Flags [P.], length 96

Illustrative output

Read-only / Safethe same capture at SITE-B — 192.168.11.50 never appears
vyos@SITE-B:~$ sudo tcpdump -ni wg0 -c 4
IP 192.168.10.31.42188 > 192.168.20.10.5432: Flags [P.], length 118
IP 192.168.20.10.5432 > 192.168.10.31.42188: Flags [P.], length 96
IP 192.168.10.31.42188 > 192.168.20.10.5432: Flags [.], length 0
IP 192.168.10.31.42188 > 192.168.20.10.5432: Flags [P.], length 96

Illustrative output

Read-only / Safenothing is being dropped here — including nothing from the new VLAN
vyos@SITE-B:~$ show firewall ipv4 name TUNNEL-IN
ipv4 name TUNNEL-IN
rule 5   accept  state established,related   3908114 packets
rule 10  accept  saddr 192.168.10.0/24        142771 packets
rule 100 drop    state invalid                     0 packets
default  drop                                      0 packets

Illustrative output

Read-only / Safewhat SITE-B will accept from the SITE-A peer
vyos@SITE-B:~$ show interfaces wireguard wg0 allowed-ips
10.10.10.0/30 192.168.10.0/24

Illustrative output

Work the evidence before reading on

The two captures are the whole exercise. Compare them line by line before reading anything else.

  1. The packet from 192.168.11.50 is on wg0 at SITE-A and is not on wg0 at SITE-B. Traffic from 192.168.10.31 is on both. What is different about the two packets, given that they are carried by the same tunnel, the same key and the same handshake?
  2. SITE-B’s TUNNEL-IN default-drop counter is at zero. If a firewall rule had dropped the packet, which counter would have moved?
  3. A packet that arrives, decrypts, and is discarded before any firewall chain — what is left that can discard it?
  4. The host on the new VLAN receives no ICMP destination unreachable. If SITE-A itself had nowhere to send the packet, what would the sender have received instead, and how quickly?
  5. SITE-B was not changed and SITE-A’s change was correct. What kind of setting can be correct at one end and wrong at the other at the same time?

Before continuing: name the one configuration node that has to change, say which router it is on, and say why nothing at SITE-A had to change at all.

Root cause

1. Allowed-ips is a policy in both directions

The node reads like a route, and outbound it is one: a packet whose destination falls inside a peer’s allowed-ips is encrypted to that peer. That direction was satisfied. SITE-A’s peer entry for SITE-B lists 192.168.20.0/24, the packet was destined there, so it was encrypted and sent — which is why it appears on SITE-A’s wg0 capture and why the transfer counters are healthy.

Inbound the same list is a crypto policy. A packet that decrypts correctly is still discarded if its source address is not inside the allowed-ips of the peer it arrived from. That is what stops an authenticated peer from injecting traffic claiming to come from somewhere it does not own, and it is not optional or configurable — it is the same data structure doing the other half of its job.

SITE-B’s list for the SITE-A peer is 10.10.10.0/30, 192.168.10.0/24. 192.168.11.50 is not in it. The packet is dropped.

2. It is dropped before anything that counts

This is the piece of evidence that pins the diagnosis, and it is the piece that was read as exculpatory. The discard happens between decryption and the wg0 interface, which is upstream of every firewall chain on the router. So:

Where the loss happensWhat moves at SITE-B
A TUNNEL-IN rule drops itthat rule’s counter
No rule matches itthe default-action drop counter
Allowed-ips rejects itnothing, anywhere

Zero across the whole named rule-set, over a period in which SITE-A sent the packet thousands of times, is not “the firewall is fine”. It is positive evidence that the packet never reached the firewall.

3. The missing ICMP tells you which end to read

The two directions of an allowed-ips gap produce completely different symptoms, and knowing which one you are looking at is the difference between reading one configuration and reading two.

Locally, if a packet is routed to wg0 and no peer claims its destination, the WireGuard device has no key to encrypt it with. It cannot silently drop it without breaking the kernel’s contract, so it emits an ICMP destination unreachable back to the sender. That shows up immediately, on the router, as a loud failure.

Remotely, a decrypted packet whose source is outside the sending peer’s allowed-ips is simply discarded. No ICMP, no log, no counter — answering would tell an unauthenticated sender that a key is valid.

The application saw silence. Silence means the far end, and the far end is the router nobody had opened.

Resolution

  1. Confirm the diagnosis from both ends before changing anything. The same flow present on wg0 at SITE-A and absent on wg0 at SITE-B, with every SITE-B firewall counter static, is the evidence. Capture it into the ticket; it is also the before half of the verification.
  2. Add the return route at SITE-B first: a static route for 192.168.11.0/24 out of wg0. On its own it is inert, because no packet from that prefix is being accepted yet, which is exactly what makes it the safe half to apply first.
  3. Add the allowed-ips entry at SITE-B second: 192.168.11.0/24 on the SITE-A peer. Add the specific prefix, not a summary that happens to contain it.
  4. Do not rely on a single commit to make the ordering irrelevant. A commit that fails partway has already applied the scripts that ran before the failure, so the order in which the two nodes become live still matters even when they are committed together.
  5. Understand what the wrong order does. Allowed-ips widened without the return route means SITE-B accepts the request, delivers it, and sends the reply toward its default route — a packet for an internal address leaving the site in clear. That is a worse state than the fault, and it presents as still broken.
  6. Commit under commit-confirm on the far end. The operator making this change may be reaching SITE-B through a path this tunnel carries, and the tunnel interface is being reconfigured.
  7. Check the other peers on that interface afterwards, if there are any. A prefix change on one peer can move prefixes off another, and the peer that lost one will not report anything.

Verification

  1. The capture pair agrees. sudo tcpdump -ni wg0 on SITE-A and on SITE-B both show the flow from 192.168.11.50. This is the check that failed, so it is the check that has to pass, and it is direct evidence rather than an inference.
  2. The traffic now reaches the firewall. The TUNNEL-IN counters at SITE-B advance for the new source prefix. This is a distinct claim from the traffic working, and it is what separates a repaired crypto policy from a policy that is now admitting packets a rule will drop later.
  3. The kernel holds what the configuration says. show interfaces wireguard wg0 allowed-ips on SITE-B lists 192.168.11.0/24. Read the running interface, not the configuration tree — on a multi-peer interface those can differ and the configuration is the one that lies.
  4. The route exists and points into the tunnel. show ip route 192.168.11.0/24 on SITE-B resolves via wg0, so replies are encrypted rather than following the default route.
  5. Both prefixes work, in both directions, from hosts rather than from routers. A router has source addresses and routes a host does not, so a test run from the router can pass while the application still fails.
  6. Nothing else regressed. The old prefix still works, and any other peer on the interface still holds its own allowed-ips.
  7. The check can fail. On a lab pair, remove the prefix from the far end allowed-ips and confirm the capture pair shows the packet at one end and not the other. Every check this team had was green throughout the incident, so a check that has only ever passed proves nothing yet.

Prevention

  • Change allowed-ips in pairs. A new prefix behind one site is a change at the other site, and the local half is complete and correct without it — which is precisely why people stop there.
  • Put the far end in the change template. “New prefix routed behind a site” needs three named items at the other end: the allowed-ips entry, the return route, and the operator who commits them.
  • Validate it automatically. The routed prefixes at each site and the far end’s allowed-ips for that peer are two lists that must agree; comparing them is machine work, and the fault is invisible in either configuration read alone.
  • Probe what the business has. Monitor a host inside each prefix rather than the far end’s tunnel address. Every signal in this incident was green because every signal was scoped to the tunnel.
  • Make the wg0 capture pair the first move for “tunnel up, no traffic”. It separates a packet dropped by a rule from a packet that never reached one, in a single step, before anyone opens a rule-set.
  • Read the missing ICMP. Silence points at the far end; a destination unreachable from your own router points at the near end. That one distinction decides which of two configurations you spend the next hour in.