Reported symptoms
SITE-A brought up a new VLAN for a payments application, 192.168.11.0/24, behind the same router that already carries 192.168.10.0/24. The change was small, reviewed and committed cleanly:
set interfaces ethernet eth0 vif 11 address '192.168.11.1/24'
set interfaces ethernet eth0 vif 11 description 'payments'
The site-to-site tunnel to SITE-B has been up for two years:
SITE-A 192.168.10.0/24 + 192.168.11.0/24 (new) wg0 10.10.10.1/30
SITE-B 192.168.20.0/24 wg0 10.10.10.2/30
The application cannot reach its database at SITE-B. Everything around it can:
- The tunnel is healthy on every measure the team trusts. Handshake under a minute old, transfer counters moving in both directions, the tunnel monitor green all week.
- Hosts on 192.168.10.0/24 reach SITE-B exactly as they always have.
- The new VLAN reaches the internet normally, so the VLAN is not broken.
- Name resolution for SITE-B hostnames works from the new VLAN. The resolver lives on 192.168.10.0/24 and does the lookup on the application’s behalf, which made “DNS works, so the tunnel is fine” a very persuasive and completely irrelevant observation.
- Connections do not fail. They hang. No refusal, no ICMP unreachable, no message from any router.
- SITE-B was not touched. Its configuration is unchanged since the last audit.
Evidence provided
vyos@SITE-A:~$ show interfaces wireguard wg0 summaryinterface: wg0
public key: rSuuvkAuSoyqvogoA5OJjYBbJKKaYS13pWs4K69PTOE=
listening port: 51820
peer: cVn4T2sM8xQ6yB1hJ0dR7kL3pW9zA5eG2uY8iO4nX1c=
endpoint: 198.51.100.20:51820
allowed ips: 10.10.10.0/30, 192.168.20.0/24
latest handshake: 47 seconds ago
transfer: 84.21 MiB received, 61.09 MiB sentIllustrative output
vyos@SITE-A:~$ sudo tcpdump -ni wg0 -c 4IP 192.168.11.50.51422 > 192.168.20.10.5432: Flags [S], length 0
IP 192.168.11.50.51422 > 192.168.20.10.5432: Flags [S], length 0
IP 192.168.10.31.42188 > 192.168.20.10.5432: Flags [P.], length 118
IP 192.168.20.10.5432 > 192.168.10.31.42188: Flags [P.], length 96Illustrative output
vyos@SITE-B:~$ sudo tcpdump -ni wg0 -c 4IP 192.168.10.31.42188 > 192.168.20.10.5432: Flags [P.], length 118
IP 192.168.20.10.5432 > 192.168.10.31.42188: Flags [P.], length 96
IP 192.168.10.31.42188 > 192.168.20.10.5432: Flags [.], length 0
IP 192.168.10.31.42188 > 192.168.20.10.5432: Flags [P.], length 96Illustrative output
vyos@SITE-B:~$ show firewall ipv4 name TUNNEL-INipv4 name TUNNEL-IN
rule 5 accept state established,related 3908114 packets
rule 10 accept saddr 192.168.10.0/24 142771 packets
rule 100 drop state invalid 0 packets
default drop 0 packetsIllustrative output
vyos@SITE-B:~$ show interfaces wireguard wg0 allowed-ips10.10.10.0/30 192.168.10.0/24Illustrative output
Work the evidence before reading on
The two captures are the whole exercise. Compare them line by line before reading anything else.
- The packet from 192.168.11.50 is on wg0 at SITE-A and is not on wg0 at SITE-B. Traffic from 192.168.10.31 is on both. What is different about the two packets, given that they are carried by the same tunnel, the same key and the same handshake?
- SITE-B’s TUNNEL-IN default-drop counter is at zero. If a firewall rule had dropped the packet, which counter would have moved?
- A packet that arrives, decrypts, and is discarded before any firewall chain — what is left that can discard it?
- The host on the new VLAN receives no ICMP destination unreachable. If SITE-A itself had nowhere to send the packet, what would the sender have received instead, and how quickly?
- SITE-B was not changed and SITE-A’s change was correct. What kind of setting can be correct at one end and wrong at the other at the same time?
Before continuing: name the one configuration node that has to change, say which router it is on, and say why nothing at SITE-A had to change at all.
Root cause
1. Allowed-ips is a policy in both directions
The node reads like a route, and outbound it is one: a packet whose destination falls inside a peer’s allowed-ips is encrypted to that peer. That direction was satisfied. SITE-A’s peer entry for SITE-B lists 192.168.20.0/24, the packet was destined there, so it was encrypted and sent — which is why it appears on SITE-A’s wg0 capture and why the transfer counters are healthy.
Inbound the same list is a crypto policy. A packet that decrypts correctly is still discarded if its source address is not inside the allowed-ips of the peer it arrived from. That is what stops an authenticated peer from injecting traffic claiming to come from somewhere it does not own, and it is not optional or configurable — it is the same data structure doing the other half of its job.
SITE-B’s list for the SITE-A peer is 10.10.10.0/30, 192.168.10.0/24.
192.168.11.50 is not in it. The packet is dropped.
2. It is dropped before anything that counts
This is the piece of evidence that pins the diagnosis, and it is the piece that was read as exculpatory. The discard happens between decryption and the wg0 interface, which is upstream of every firewall chain on the router. So:
| Where the loss happens | What moves at SITE-B |
|---|---|
A TUNNEL-IN rule drops it | that rule’s counter |
| No rule matches it | the default-action drop counter |
| Allowed-ips rejects it | nothing, anywhere |
Zero across the whole named rule-set, over a period in which SITE-A sent the packet thousands of times, is not “the firewall is fine”. It is positive evidence that the packet never reached the firewall.
3. The missing ICMP tells you which end to read
The two directions of an allowed-ips gap produce completely different symptoms, and knowing which one you are looking at is the difference between reading one configuration and reading two.
Locally, if a packet is routed to wg0 and no peer claims its destination, the WireGuard device has no key to encrypt it with. It cannot silently drop it without breaking the kernel’s contract, so it emits an ICMP destination unreachable back to the sender. That shows up immediately, on the router, as a loud failure.
Remotely, a decrypted packet whose source is outside the sending peer’s allowed-ips is simply discarded. No ICMP, no log, no counter — answering would tell an unauthenticated sender that a key is valid.
The application saw silence. Silence means the far end, and the far end is the router nobody had opened.
Resolution
- Confirm the diagnosis from both ends before changing anything. The same flow present on wg0 at SITE-A and absent on wg0 at SITE-B, with every SITE-B firewall counter static, is the evidence. Capture it into the ticket; it is also the before half of the verification.
- Add the return route at SITE-B first: a static route for 192.168.11.0/24 out of wg0. On its own it is inert, because no packet from that prefix is being accepted yet, which is exactly what makes it the safe half to apply first.
- Add the allowed-ips entry at SITE-B second: 192.168.11.0/24 on the SITE-A peer. Add the specific prefix, not a summary that happens to contain it.
- Do not rely on a single commit to make the ordering irrelevant. A commit that fails partway has already applied the scripts that ran before the failure, so the order in which the two nodes become live still matters even when they are committed together.
- Understand what the wrong order does. Allowed-ips widened without the return route means SITE-B accepts the request, delivers it, and sends the reply toward its default route — a packet for an internal address leaving the site in clear. That is a worse state than the fault, and it presents as still broken.
- Commit under
commit-confirmon the far end. The operator making this change may be reaching SITE-B through a path this tunnel carries, and the tunnel interface is being reconfigured. - Check the other peers on that interface afterwards, if there are any. A prefix change on one peer can move prefixes off another, and the peer that lost one will not report anything.
Verification
- The capture pair agrees.
sudo tcpdump -ni wg0on SITE-A and on SITE-B both show the flow from 192.168.11.50. This is the check that failed, so it is the check that has to pass, and it is direct evidence rather than an inference. - The traffic now reaches the firewall. The TUNNEL-IN counters at SITE-B advance for the new source prefix. This is a distinct claim from the traffic working, and it is what separates a repaired crypto policy from a policy that is now admitting packets a rule will drop later.
- The kernel holds what the configuration says.
show interfaces wireguard wg0 allowed-ipson SITE-B lists 192.168.11.0/24. Read the running interface, not the configuration tree — on a multi-peer interface those can differ and the configuration is the one that lies. - The route exists and points into the tunnel.
show ip route 192.168.11.0/24on SITE-B resolves via wg0, so replies are encrypted rather than following the default route. - Both prefixes work, in both directions, from hosts rather than from routers. A router has source addresses and routes a host does not, so a test run from the router can pass while the application still fails.
- Nothing else regressed. The old prefix still works, and any other peer on the interface still holds its own allowed-ips.
- The check can fail. On a lab pair, remove the prefix from the far end allowed-ips and confirm the capture pair shows the packet at one end and not the other. Every check this team had was green throughout the incident, so a check that has only ever passed proves nothing yet.
Prevention
- Change allowed-ips in pairs. A new prefix behind one site is a change at the other site, and the local half is complete and correct without it — which is precisely why people stop there.
- Put the far end in the change template. “New prefix routed behind a site” needs three named items at the other end: the allowed-ips entry, the return route, and the operator who commits them.
- Validate it automatically. The routed prefixes at each site and the far end’s allowed-ips for that peer are two lists that must agree; comparing them is machine work, and the fault is invisible in either configuration read alone.
- Probe what the business has. Monitor a host inside each prefix rather than the far end’s tunnel address. Every signal in this incident was green because every signal was scoped to the tunnel.
- Make the wg0 capture pair the first move for “tunnel up, no traffic”. It separates a packet dropped by a rule from a packet that never reached one, in a single step, before anyone opens a rule-set.
- Read the missing ICMP. Silence points at the far end; a destination unreachable from your own router points at the near end. That one distinction decides which of two configurations you spend the next hour in.