VPN diagnostic methodology — the layered evidence-first approach

VPN failures show up as user-visible symptoms — “the tunnel is down”, “users can’t reach the remote network”, “the connection is slow” — but the causes sit at one of four layers: negotiation (is IKE running), authentication (is the peer recognised), routing (does the traffic know where to go), and firewall/NAT/MTU (does the traffic actually get through). The four layers are independent; a tunnel can be perfectly negotiated and have no traffic flowing because a single firewall rule is missing. The discipline that finds the cause fast is to walk the layers in order, collect evidence at each layer, and identify the inconsistency — without rushing to change configuration. This lesson covers the layered model, the diagnostic commands at each layer, and the evidence-first discipline that prevents diagnosis from making the failure worse.

The four layers

A VPN tunnel has four layers, each with its own evidence:

Layer 1 — negotiation. The IKE messages between the two peers are flowing. Phase 1 establishes the IKE SA; Phase 2 establishes the ESP SAs. For WireGuard, the equivalent is the UDP 51820 handshake. Evidence: swanctl --list-sas, wg show, packet capture on UDP 500/4500 (IPsec) or UDP 51820 (WireGuard).

Layer 2 — authentication. The peers are mutually recognised. For IPsec: the PSK matches, certificates validate against trusted CAs, identifiers match (IP vs DN). For WireGuard: the public keys match, allowed IPs are configured on both sides. Evidence: charon negotiation logs, WireGuard hand-off logs.

Layer 3 — routing. The protected subnets know how to reach each other. For route-based IPsec: a route through the VTI exists. For policy-based IPsec: the Phase 2 traffic selector matches. For WireGuard: the AllowedIPs on both sides are correct. Evidence: netstat -rn, swanctl --list-sas --list-conns, wg show.

Layer 4 — firewall / NAT / MTU. Traffic that is destined for the remote subnet is allowed through, NAT is not interfering, MTU accommodates the encapsulated packet. Evidence: pfctl -s rules, pfctl -s state, packet captures on both sides of the tunnel, ping -M do MTU probes.

The four layers compose. A consistent picture at every layer is the working configuration. An inconsistency at any layer is the bug.

The diagnostic command sequence

A VPN investigation starts with read-only commands that produce evidence without modifying state. The sequence:

Step 1 — confirm the tunnel is configured. Look at the OPNsense web UI (or swanctl --list-conns for IPsec, wg show for WireGuard). The operator knows what they intended before they look at what is happening.

Step 2 — confirm the service is running. configctl ipsec status for IPsec, configctl wireguard status (if running WireGuard through the plugin) for WireGuard. The service must be running before the tunnel can do anything.

Step 3 — confirm the negotiation state. swanctl --list-sas shows every active IKE and ESP SA. An empty list means no negotiation has succeeded. A list with entries shows the negotiated algorithms and the byte counters.

Step 4 — confirm the routes. route -n get <remote_subnet> shows the egress interface. For IPsec route-based, the interface should be the VTI. For WireGuard, the interface is wg0 or similar. For IPsec policy-based, the route lookup will return the WAN interface (the routing table does not see the IPsec layer); the Phase 2 selectors are what encode the policy.

Step 5 — confirm the firewall allows the traffic. pfctl -s rules | grep -E 'opt_vpn|wg0|IPSEC' shows the rules on the tunnel interface. A default-deny policy on the tunnel is the most common silent failure for a brand-new tunnel.

Step 6 — capture packets and verify end-to-end. tcpdump -ni <tunnel_interface> shows encapsulated or decapsulated traffic. A ping -c 4 <remote_host> from a host on the LAN to a host on the remote subnet shows whether traffic flows.

The evidence-first discipline

The temptation in a non-working tunnel is to start changing things. The operator restarts the service, regenerates the keys, regenerates the firewall rules, hoping one change fixes the problem. Each change destroys evidence about what the actual failure was.

The discipline:

1. Collect before changing. Run the read-only command sequence. Save the output. Take a screenshot of the GUI. The collected evidence is what makes the eventual change targeted.
2. Identify the layer. The evidence at one of the four layers is inconsistent. The diagnostic question: “Which layer’s evidence disagrees with the rest?”
3. Form a hypothesis. The hypothesis names the layer, the configuration, and the symptom. “Layer 2: the PSK on the remote side does not match the local PSK; symptom: IKE_AUTH fails with AUTHENTICATION_FAILED.”
4. Test with a single targeted change. Change the specific configuration item the hypothesis names. Do not change multiple items at once; the operator who changes three items at once cannot identify which one fixed the problem.
5. Verify with fresh evidence. Re-run the diagnostic command sequence. The new evidence should show the configuration is consistent across all four layers and the end-to-end test succeeds.

The cycle repeats until every layer is consistent and the end-to-end test succeeds.

The negotiation layer in detail

For IPsec, swanctl --list-sas shows the negotiation state. An empty result means no SA has been established. A result with entries means at least one Phase 1 SA and at least one Phase 2 SA exist. The diagnostic questions:

• No Phase 1 SA after several seconds. The negotiation is not even starting, or the proposal exchange is failing. Capture on UDP 500 to confirm IKE packets are leaving; check charon.log for NO_PROPOSAL_CHOSEN.
• Phase 1 SA exists, no Phase 2 SA. The IKE negotiation succeeded but the Phase 2 traffic selectors or proposals do not match. Check charon.log for TS_UNACCEPTABLE or NO_PROPOSAL_CHOSEN on Phase 2.
• Phase 1 SA exists, Phase 2 SA exists, but byte counters are zero. The tunnel is up but traffic is not flowing. Look at Layer 3 and Layer 4.

For WireGuard, wg show shows the handshake state:

$ wg show

interface: wg0
public key: <local-server-public-key>
private key: (hidden)
listening port: 51820

peer: <remote-peer-public-key>
endpoint: 198.51.100.2:51820
allowed ips: 10.1.0.0/24
latest handshake: 1 minute, 12 seconds ago
transfer: 1.2 MiB received, 4.8 MiB sent
persistent keepalive: every 25 seconds

A wg show with no peers listed means the tunnel is configured on the local side but the peer configuration is missing; the operator must add the peer entry. A peer listed but no latest handshake means the peer is not reachable on UDP 51820. A recent handshake but zero transfer means Layer 3 or Layer 4 is broken.

The authentication layer in detail

For IPsec, authentication failures show up in charon.log as specific error codes:

charon:  16[IKE] authentication of <peer> with pre-shared key failed
charon:  16[IKE] IKE_AUTH response contained NO_PROPOSAL_CHOSEN
charon:  16[IKE] ID of <peer> does not match expected ID

The first error is a PSK mismatch — the operator who set the PSK has the wrong secret. The second is a proposal mismatch (Layer 1 disguised as Layer 2 because the proposal is part of the authentication exchange). The third is an identifier mismatch — the My Identifier and Peer Identifier settings do not line up.

For WireGuard, authentication is the public-key cryptography itself. The operator configures the peer’s public key on the local side and the local side’s public key on the peer side. A mismatch means the peer did not configure the right key. The diagnostic in wg show is a handshake that does not progress; the peer claims to be there but the cryptographic handshake fails.

The routing layer in detail

A tunnel can be perfectly negotiated but traffic is not flowing because no route sends it through. The diagnostic at Layer 3:

• Route-based IPsec: route -n get <remote_subnet> returns the VTI as the egress interface. If the route is missing, add it (System → Routes → Configuration).
• Policy-based IPsec: the Phase 2 traffic selector must match the traffic. A Phase 2 with local subnet 10.0.0.0/24 and remote subnet 10.1.0.0/24 does not carry traffic from 10.0.50.0/24 — the selector misses the new subnet.
• WireGuard: wg show wg0 lists the peer’s allowed ips. A peer with allowed ips: 10.1.0.0/24 only carries traffic destined to that subnet. A second peer needed for additional subnets; the allowed ips field on the correct peer covers the new subnet.

The firewall / NAT / MTU layer in detail

A tunnel that comes up but cannot carry traffic usually has a firewall, NAT, or MTU issue. The diagnostic:

• Firewall rule missing on the tunnel interface. pfctl -s rules | grep <tunnel> shows nothing for the interface; default-deny blocks the traffic. Fix: add a pass rule.
• Outbound NAT translating the source address. A traffic flow that left the LAN with source 10.0.0.50 arrives on the remote side as the firewall’s WAN IP because Outbound NAT translated it. The remote end’s firewall rules (which expect 10.0.0.0/24) drop the traffic. Fix: add a no-NAT rule for the protected subnets.
• MTU too low for the encapsulated packet. TCP segments larger than the tunnel MTU are dropped, with no ICMP feedback because ESP fragments are not spec-compliant for PMTUD. Fix: lower the VTI MTU and enable MSS clamping.

A tcpdump on both sides of the tunnel confirms which of these is happening:

$ tcpdump -ni wan0 'src host 198.51.100.1 and dst 198.51.100.2' -c 6

12:34:56.789012 198.51.100.1.4500 > 198.51.100.2.4500: UDP, length 96
12:34:56.789013 198.51.100.1.4500 > 198.51.100.2.4500: UDP, length 1380
12:35:01.789345 198.51.100.1.4500 > 198.51.100.2.4500: UDP, length 1380
12:35:01.801678 198.51.100.2.4500 > 198.51.100.1.4500: UDP, length 96
12:35:01.801678 198.51.100.2.4500 > 198.51.100.1.4500: UDP, length 1380
12:35:06.790012 198.51.100.1.4500 > 198.51.100.2.4500: UDP, length 96

A diagnostic that sees only outgoing ESP-in-UDP with no return ESP-in-UDP indicates the return path is broken. A diagnostic that sees ESP-in-UDP in both directions but no clear-text on the VTI indicates the VTI route or firewall is broken.

The diagnostic walkthrough

A fully worked example: a tunnel between two OPNsense firewalls stopped working after a configuration change. The operator’s investigation:

1. Collect the evidence. swanctl --list-sas shows no entries. swanctl --list-conns shows the connection definition is present. tcpdump -ni wan0 'udp port 500 or udp port 4500' -c 4 shows no IKE packets at all.
2. Identify the layer. Layer 1 (negotiation) is failing — no IKE packets. The negotiation has not even started.
3. Hypothesis. The IPsec service is not running on the firewall. A configctl ipsec status confirms the suspicion — charon is stopped.
4. Test. Restart the IPsec service. configctl ipsec restart.
5. Verify. swanctl --list-sas shows an entry. tcpdump -ni wan0 shows IKE packets. The tunnel comes up.

The walkthrough took 30 seconds because the operator stopped at each layer in order and identified the layer where the failure was, rather than trying to fix Phase 2 proposals, add firewall rules, or change the PSK.

Summary

• Four layers: negotiation (L1), authentication (L2), routing (L3), firewall / NAT / MTU (L4).
• Walk the layers top to bottom; identify the inconsistency before changing configuration.
• Diagnostic command sequence: tunnel config, service status, negotiation state, routes, firewall rules, end-to-end test.
• Read-only commands preserve evidence — restart, restart, regenerate only after collecting the evidence that names the cause.
• Form a hypothesis with a specific layer, configuration, and symptom; test with a single targeted change.