Skip to main content
RunBook Academy

← All break/fix scenarios in VyOS

advancedvyos-wireguard~30 min

WireGuard Handshake Fails

Reported symptoms

  • ●Both WireGuard tunnels out of SITE-A stopped passing traffic during a hardware replacement and have not come back
  • ●Neither tunnel has completed a handshake since; `show interfaces wireguard wg0 summary` prints the peer with no handshake line and no transfer line at all
  • ●Everything else about the rebuilt router is healthy: the eBGP session to ISP-A is Established, the LAN forwards, SSH from the management network works
  • ●The far ends report nothing wrong — their peer lists are unchanged and their firewall counters show the encrypted UDP arriving and being accepted
  • ●No log line anywhere names a peer, a key or a failure, on any of the three routers
  • ●The restore was reviewed: `compare` after loading the archived configuration showed no differences from the pre-failure config
  • ●The failure was found fourteen hours later by a batch job at SITE-B, not by monitoring

Evidence

  • · `show interfaces wireguard wg0 summary` on SITE-A — peer listed with endpoint and allowed ips; no `latest handshake` line, no `transfer` line
  • · `show interfaces wireguard wg1 summary` on SITE-A — the same absence for the SITE-C peer
  • · `sudo tcpdump -ni eth0 'udp port 51820' -c 10 -vv` on SITE-A — initiations leaving toward 198.51.100.20 at the retry cadence
  • · `sudo tcpdump -ni eth0 'udp port 51820' -c 10 -vv` on SITE-B — the same packets arriving; nothing leaving toward 198.51.100.10
  • · `show firewall ipv4 input filter` on SITE-B — the WireGuard accept rule counter is advancing
  • · `show interfaces wireguard wg0 public-key` on SITE-A — a 44-character key
  • · `show interfaces wireguard wg0 peers` on SITE-B — a different 44-character key
  • · `show configuration commands | match wireguard` on all three routers — no `preshared-key` node anywhere
  • · `show ip bgp summary` on SITE-A — the ISP-A session is Established with a recent uptime
  • · `show system commit` on SITE-A — one commit, timed with the hardware replacement
Diagnosis and resolutionclick to reveal

Root cause

The rebuilt router is not the router the far ends were configured to talk to. On VyOS 1.4 and later the WireGuard private key is a configuration value rather than a file referenced by name, so restoring the archived configuration onto replacement hardware also restores the router's WireGuard identity — and that is exactly what the operator verified with `compare`. The provisioning script then ran to completion. It is the standard script for bringing up a new VyOS router, and it generates a key pair per WireGuard interface, because a genuinely new router needs one. Running it on a restore path replaced both restored private keys with fresh ones and committed. From that moment SITE-A had a new public key on each interface and both far ends still held the old one. WireGuard never answers a packet it cannot authenticate, so SITE-B and SITE-C discard the initiations in complete silence: no ICMP, no log, no counter that names the peer. The two symptoms that steered the investigation away from this are both consequences of the same cause rather than independent facts. Both tunnels died together because both interfaces were re-keyed by the same script, which made the fault look router-wide and therefore like a WAN or platform problem. And the far ends looked healthy because, from their side, they are: they are receiving well-formed UDP on an open port from a stranger, and discarding it is correct behaviour, not a fault to report.

Remediation

There are two honest repairs and the choice between them is a security decision, not a technical one. Restoring the archived private key onto SITE-A is one commit on one router and brings both tunnels back together, because it restores the identity every far end already trusts; the cost is that it reuses a private key which has been sitting in a configuration backup and which also lived on the failed hardware now heading into an RMA pipeline. If that unit left custody with readable storage, the key is exposed and this option is off the table. The alternative is to finish the rotation that the script started by accident: publish SITE-A's new public key to every far end and commit there, one site at a time. That is N commits at N sites, each needing its own operator and its own window, the tunnels return one by one, and the outage lasts until the last one lands. Whichever is chosen, the far-end commits are the dangerous ones — an operator whose session reaches the router through the tunnel being edited is editing their own path, so `commit-confirm` is not optional there. Holding is a real third option where the traffic has an alternative path: leave the tunnels down, route the affected prefixes over the secondary transit, and make the change in a window with console access to both ends, provided the hold carries a named owner and a stated end time rather than drifting into the default.

Verification

Read both ends, always. The single most informative state WireGuard produces is bytes sent with zero bytes received, and it is only visible if you have both routers in front of you. A repaired tunnel shows a `latest handshake` inside the last couple of minutes and transfer counters moving in both directions on both routers; a handshake on one side with a flat received counter on the other is not a repaired tunnel. Then prove the identity directly rather than inferring it: `show interfaces wireguard wg0 public-key` on SITE-A must produce a value that appears in `show interfaces wireguard wg0 peers` on SITE-B, and the same comparison in the other direction. Do it for wg1 and SITE-C as well; the second tunnel is the one that gets forgotten because the first one coming back feels like the end of the incident. Confirm the encrypted UDP is being accepted at each end by watching the input-filter rule counter advance, and confirm the inner path by reaching the far tunnel address and then a host behind it. Finally, prove the check can fail: on a lab pair, configure a deliberately wrong peer key and confirm the key comparison reports a mismatch. A comparison that has only ever agreed is untested, and this incident ran for fourteen hours with every check anyone had returning its healthy answer.

Prevention

Separate provisioning from restoration in the automation, because the two paths differ on exactly one question — whether this router needs a new identity — and answering it wrongly is silent. A restore path must not generate key material; if the same script serves both, key generation belongs behind an explicit flag that the operator sets deliberately. Read `compare` immediately before the commit that matters, not at an earlier point in a script that keeps running afterwards; the review in this incident was real, competent and performed against a candidate configuration that was still going to change. Keep the key inventory that WireGuard does not keep for you: public key against router against site, checked as a step in any change that touches an interface, since the install command prints the new public key and a value that does not match the inventory means the identity moved. Monitor the handshake age per peer rather than pinging the tunnel address, and never leave a maintenance suppression without an expiry — the alert that would have fired at the swap was suppressed for the window and stayed suppressed for fourteen hours. And plan rotation as a paired change from the start: the two commits belong in one window, in a stated order, with the far end updated before the local key is cut over wherever the topology allows it.

Reported symptoms

SITE-A’s router lost a power supply and was replaced with an identical spare during a planned window. The archived configuration was loaded onto the new chassis, the operator read compare, saw no differences from the pre-failure configuration, and handed back.

The site runs two WireGuard tunnels:

SITE-A  198.51.100.10   wg0 port 51820 -> SITE-B  198.51.100.20   tunnel 10.10.10.0/30
SITE-A  198.51.100.10   wg1 port 51821 -> SITE-C  203.0.113.20    tunnel 10.10.20.0/30

Fourteen hours later a nightly batch job at SITE-B failed and the incident opened. Both tunnels had been down since the swap.

What made it hard is what was working:

  • The eBGP session to ISP-A came up on the replacement chassis and has been Established since. Whatever else is wrong, the WAN is not down and the router is reachable from the internet.
  • The LAN forwards. Management SSH works. Nothing about the router looks freshly broken.
  • SITE-B and SITE-C report nothing. Their peer configuration is untouched, their firewall counters show the encrypted UDP arriving and being accepted, and their logs are as empty as they always are.
  • Both tunnels failed at the same moment, which reads as a router-wide or circuit-wide event rather than anything per-peer.
  • The tunnel monitor — an ICMP probe of 10.10.10.2 — did go red at the swap. Its alert was suppressed for the maintenance window and the suppression was never lifted.

Evidence provided

Read-only / Safenote what is missing, not what is present
vyos@SITE-A:~$ show interfaces wireguard wg0 summary
interface: wg0
public key: ZF1DeQIGCVfpPnBLyVcmPh+KbxtaUShqXr9pwFHJ4j4=
private key: (hidden)
listening port: 51820

peer: cVn4T2sM8xQ6yB1hJ0dR7kL3pW9zA5eG2uY8iO4nX1c=
endpoint: 198.51.100.20:51820
allowed ips: 10.10.10.0/30, 192.168.20.0/24

Illustrative output

There is no latest handshake line and no transfer line. Those are printed only once a handshake has happened and once a byte has moved, so this is an absence to read rather than a “never” to report.

Read-only / SafeSITE-A is initiating, repeatedly, and nothing comes back
vyos@SITE-A:~$ sudo tcpdump -ni eth0 'udp port 51820' -c 4 -vv
IP 198.51.100.10.51820 > 198.51.100.20.51820: UDP, length 148
IP 198.51.100.10.51820 > 198.51.100.20.51820: UDP, length 148
IP 198.51.100.10.51820 > 198.51.100.20.51820: UDP, length 148
IP 198.51.100.10.51820 > 198.51.100.20.51820: UDP, length 148

Illustrative output

Read-only / Safethe packets arrive — and SITE-B sends nothing in reply
vyos@SITE-B:~$ sudo tcpdump -ni eth0 'udp port 51820' -c 4 -vv
IP 198.51.100.10.51820 > 198.51.100.20.51820: UDP, length 148
IP 198.51.100.10.51820 > 198.51.100.20.51820: UDP, length 148
IP 198.51.100.10.51820 > 198.51.100.20.51820: UDP, length 148
IP 198.51.100.10.51820 > 198.51.100.20.51820: UDP, length 148

Illustrative output

Read-only / Safethe underlay firewall is accepting them, with a counter to prove it
vyos@SITE-B:~$ show firewall ipv4 input filter
ipv4 filter input
rule 5   accept   state established,related      412984 packets
rule 10  accept   udp dport 51820 iif eth0         1174 packets
default  drop                                        26 packets

Illustrative output

Read-only / Safewhat SITE-A says it is
vyos@SITE-A:~$ show interfaces wireguard wg0 public-key
ZF1DeQIGCVfpPnBLyVcmPh+KbxtaUShqXr9pwFHJ4j4=

Illustrative output

Read-only / Safewhat SITE-B is waiting for
vyos@SITE-B:~$ show interfaces wireguard wg0 peers
rSuuvkAuSoyqvogoA5OJjYBbJKKaYS13pWs4K69PTOE=

Illustrative output

Work the evidence before reading on

The capture pair is doing most of the work here, so read it carefully before reaching for an explanation.

  1. The encrypted UDP leaves SITE-A and arrives at SITE-B. Which hypotheses does that eliminate outright?
  2. SITE-B’s input filter counter is advancing on the WireGuard rule. Which further hypothesis does that eliminate?
  3. SITE-B receives well-formed packets on an open port and sends nothing back. What are the reasons a WireGuard peer declines to answer, and what would each of them require to be true in the configuration?
  4. No preshared-key node exists on any of the three routers. Cross one of those reasons off.
  5. Both tunnels failed at the same instant. What do wg0 and wg1 have in common that a per-peer setting does not?

Before continuing: the far ends were not touched during the window. Name the thing that changed at SITE-A which would break every peer at once without breaking BGP, the LAN, or SSH.

Root cause

1. The identity moved, and the configuration diff was clean

On VyOS 1.4 and later the WireGuard private key is a configuration value: set interfaces wireguard wg0 private-key ..., carrying the base64 key itself rather than naming a file. That is why loading the archived configuration onto replacement hardware restores the tunnel identity along with everything else, and why the operator’s compare was clean and correct.

The provisioning script then ran. It is the standard script for standing up a VyOS router, and among the things it does for a new router is:

run generate pki wireguard key-pair install interface wg0
run generate pki wireguard key-pair install interface wg1

The install form stages set interfaces wireguard wgN private-key ... into the current configuration session and prints the corresponding public key for the operator to send to the far end. On a new router that is exactly right. On a restore it overwrites the identity that was just restored, and the commit that follows makes it real.

The review happened before that step, not after it. That is the whole defect: a real, competent compare read against a candidate configuration that had not finished changing.

2. Silence is the designed response

SITE-B receives an initiation it cannot authenticate — the key it holds for this peer no longer corresponds to the key SITE-A now signs with — and discards it without a reply. There is no ICMP rejection, no log line and no counter that names the peer, because answering would tell an unauthenticated sender that they found a live WireGuard endpoint.

That produces the exact signature in the evidence:

flowchart TD
  A["latest handshake absent"] --> B{"Encrypted UDP<br/>leaves SITE-A?"}
  B -- "yes" --> C{"Arrives at SITE-B?"}
  C -- "yes" --> D{"SITE-B replies?"}
  D -- "no" --> E["SITE-B will not authenticate it"]
  E --> F{"preshared-key<br/>configured anywhere?"}
  F -- "no" --> G["Public key mismatch"]

Every branch not taken is a hypothesis the capture pair eliminated for free, which is why the capture on both ends is the second command to run and not the tenth.

3. The blast radius pointed away from the cause

One identity per interface, two interfaces re-keyed by one script, two tunnels dead. The team read “everything VPN is down and everything else is up” as evidence of something router-wide or circuit-wide, and spent the first hour on the WAN handoff that had just been recabled.

The BGP session was the fact that should have ended that line of enquiry in a minute: an Established eBGP session over the same physical path is a working WAN. It was known and it was not used, because it was filed as “the parts that still work” rather than as evidence.

Resolution

  1. Confirm the mismatch in both directions before changing anything: the public key on SITE-A against the peer list on SITE-B, and the public key on SITE-B against the peer list on SITE-A. Do the same for wg1 and SITE-C. Two tunnels, four comparisons.
  2. Establish whether the failed chassis is still in custody. If its storage has left the building, the archived private key must be treated as exposed and the restore-the-old-key option is unavailable. This is a security question and it decides the rest of the procedure, so answer it before choosing.
  3. Where the old key is still trustworthy: put the archived private key back on SITE-A. One commit on one router, both tunnels return together, and no far end has to change. Read compare immediately before committing, and commit under commit-confirm.
  4. Where it is not: publish the new public key to SITE-B and SITE-C and update peer SITE-A public-key at each. This is one commit per far end, each in its own window with its own operator, and the site stays down until the last one lands. Sequence it and say so in the incident channel rather than letting sites come back in an unplanned order.
  5. On any far-end commit, use commit-confirm. The operator editing SITE-B may be reaching SITE-B through a path that this tunnel carries, and a wrong key committed unattended removes both the tunnel and the session that would fix it.
  6. Bring wg1 and SITE-C back explicitly rather than assuming the same action covered it. The second tunnel is the one that gets closed with the incident and discovered a week later.
  7. Lift the monitoring suppression as a named step, and confirm the probe is green rather than assuming it followed the tunnel.

Verification

  1. Both ends show a recent handshake. show interfaces wireguard wg0 summary on SITE-A and on SITE-B both report a latest handshake inside the last couple of minutes. One end alone is not a result.
  2. The transfer counters move in both directions on both routers. Bytes sent with a flat received counter is the signature of a tunnel that is still broken and is the state this incident started from.
  3. The identity comparison passes in both directions, for both tunnels. The public key printed here appears in the peer list there, and vice versa — four comparisons, recorded in the ticket.
  4. The underlay is confirmed by a counter rather than by a capture. The input-filter WireGuard rule counter advances on both routers; a capture proves the same thing but does not stay proved after you close it.
  5. The inner path works end to end. Reach the far tunnel address, then a host behind it, from a host on the LAN rather than from the router — the router has sources and routes a host does not.
  6. The check can fail. On a lab pair, set a deliberately wrong peer public key and confirm the comparison reports a mismatch and the handshake stops. A comparison that has only ever agreed has not been tested.
  7. Monitoring reflects reality. The per-peer handshake-age signal is present and green, the suppression is lifted, and forcing a tunnel down in the lab raises the alert.

Prevention

  • Split provisioning from restoration in the build automation. The two paths differ on one question — does this router need a new identity — and the wrong answer is silent. Where one script serves both, generating key material must sit behind an explicit flag.
  • Read compare immediately before the commit, not earlier in a sequence that keeps writing. A review performed against a candidate configuration that is still being modified is a review of something that will not be committed.
  • Keep the inventory WireGuard does not keep for you: public key against router against site. The install command prints the new public key; a value that does not match the inventory means the identity moved, and that is the moment to notice rather than fourteen hours later.
  • Alert on handshake age per peer, not on a ping of the tunnel address. A probe that pings the far tunnel address is a proxy that goes red for a dozen reasons and tells you which one for none of them.
  • Give every maintenance suppression an expiry. A suppression that must be removed by hand will eventually not be.
  • Treat key rotation as a paired change with a stated order, planned into one window. The accidental version of this rotation is exactly the deliberate one performed at one end only, which is why the fix and the fault have the same shape.