VPN fabric monitoring — knowing whether the tunnels are up, slow or silently broken

A VPN fabric that the operator does not monitor is a VPN fabric that fails silently. WireGuard does not have a “connection refused” indicator for a tunnel that is up but carrying no traffic; IPsec shows SAs as “ESTABLISHED” even when the underlying WAN path is degraded; BGP sessions show as “up” even when the routing has shifted to a less-preferred path. The user notices the problem first, often expressed as “the network is slow” or “I can’t reach the file share”, and the operator must work backwards from the symptom to the cause.

This lesson covers the monitoring discipline for a VPN fabric — the canary metrics, the thresholds, the alerts, the dashboards, and the failure modes that distinguish VPN problems from firewall, routing, or upstream problems.

The canary metrics

The metrics that indicate a healthy VPN fabric are a small set. The operator monitors them as the canary; any anomaly is investigated before the user reports it.

The discipline: monitor every one of these for every tunnel and every BGP peer. A single missing metric is a blind spot that will be discovered during an incident.

Monitoring WireGuard

The wg show command on OPNsense (or via the Prometheus exporter) exposes the canary metrics for every WireGuard peer. The operator exports them to Prometheus and alerts on staleness.

# WireGuard peer state — illustrative output.
$ wg show wg0
peer: <peer-public-key>
  endpoint: 203.0.113.1:51820
  allowed ips: 10.1.0.0/24
  latest handshake: 12 seconds ago
  transfer: 1.23 GiB received, 4.56 GiB sent
  persistent keepalive: every 25 seconds

The Prometheus exporter exposes the metrics as:

• wireguard_sent_bytes — bytes sent to the peer.
• wireguard_received_bytes — bytes received from the peer.
• wireguard_latest_handshake_seconds — seconds since the last handshake.

The alert for handshake staleness:

# Alert: WireGuard peer handshake stale.
- alert: WireGuardPeerHandshakeStale
  expr: time() - wireguard_latest_handshake_seconds > 180
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "WireGuard peer {{ $labels.peer }} handshake stale"

The discipline: alert on staleness, not on absence. A peer that has never had a handshake is misconfigured; a peer with a stale handshake has dropped.

Monitoring IPsec

IPsec monitoring is more complex than WireGuard because IPsec has SAs (security associations) that expire and re-key. The metrics:

• ipsec_sa_lifetime_remaining — seconds before the SA expires. Alert when remaining < 600 seconds without a successful re-key.
• ipsec_sa_state — state of the SA (INSTALLED, REKEYING, etc.). Alert when state is not INSTALLED for an extended period.
• ipsec_bytes — bytes processed by the SA. Throughput metric.

# IPsec SA state via swanctl.
$ swanctl --list-sas
site-a: #1, ESTABLISHED 2 minutes ago, 10.0.0.1[500]...203.0.113.1[500]
  child: #1, ESTABLISHED 2 minutes ago, ESP SPIs: 0xc0ffee00
    bytes_i (30s): 1.2 MiB, bytes_o (30s): 4.5 MiB
    rekey in 45 minutes

The discipline: monitor both the IKE SA (Phase 1) and the child SA (Phase 2). A re-key failure on either produces a brief outage; a re-key failure that is not detected and alerted produces a longer outage when the SA eventually expires without re-keying.

Monitoring BGP

For a fabric that uses dynamic routing, BGP session state is the canary metric. The OPNsense FRR plugin exports BGP state via the Prometheus exporter.

# BGP session state via vtysh.
$ vtysh
show ip bgp summary
Neighbor        AS    State    Up/Down  LastEvent
10.99.0.1       65000 Established 00:23:45 00:00:12

The Prometheus metrics:

• bgp_session_state — 1 for Established, 0 otherwise.
• bgp_routes_received — number of routes received from the peer.
• bgp_session_uptime_seconds — uptime of the session.

The alert for BGP session state:

# Alert: BGP session down.
- alert: BGPSessionDown
  expr: bgp_session_state == 0
  for: 2m
  labels:
    severity: critical

MTU black-hole detection

A tunnel that drops large packets is the worst kind of silent failure. Small packets work; large packets are silently dropped; TCP connections hang partway through because the larger segments are dropped.

The detection: a periodic ping with a payload close to the MTU.

# MTU black-hole detection.
# ping with a 1400-byte payload (close to the AWS MTU of 1437).
ping -c 5 -s 1400 10.0.1.5

# If the ping succeeds, MTU is OK.
# If the ping fails but ping with -s 500 succeeds, MTU is broken.

The discipline: run this test periodically (every 5 minutes, every site pair) and alert on failure. A simple Prometheus blackbox exporter configuration handles it:

# Prometheus blackbox exporter for MTU black-hole detection.
modules:
  icmp_mtu:
    prober: icmp
    timeout: 5s
    icmp:
      payload_size: 1400
      dont_fragment: true

Building the dashboard

A Grafana dashboard for the VPN fabric surfaces the canary metrics in one place. The panels:

1. Tunnel state. A grid showing every tunnel (WireGuard peer, IPsec SA, BGP session) with colour-coded status (green = up, red = down, yellow = degraded).
2. Handshake age. A line graph of wireguard_latest_handshake_seconds for every peer. A spike to high values indicates a tunnel drop.
3. Throughput. A line graph of bytes per second per tunnel. A sudden drop indicates a routing change or a tunnel problem.
4. BGP session state. A grid of every BGP session with uptime and last event.
5. MTU black-hole test results. A panel showing the success/failure of the periodic MTU test.

# Dashboard layout (illustrative).
+---------------------------------------------------+
| Tunnel state grid (green/red/yellow per tunnel)    |
+---------------------------+-----------------------+
| Handshake age (line)      | BGP session state grid|
+---------------------------+-----------------------+
| Throughput per tunnel (stacked area)              |
+---------------------------+-----------------------+
| MTU black-hole test results                       |
+---------------------------------------------------+

The discipline: the dashboard is the operator’s first stop when a VPN problem is reported. A glance tells them whether the tunnels are up, whether BGP is established, and whether throughput is normal. Without the dashboard, the operator is reading individual wg show outputs and correlating by hand.

Distinguishing VPN problems from other problems

The operator must distinguish a VPN fabric problem from:

• Firewall rule problem. The tunnel is up; BGP is up; the rules do not allow the traffic. Symptom: the firewall log shows blocks for the expected traffic.
• Routing problem. The tunnel is up; BGP is up; the routes do not point at the tunnel. Symptom: traceroute from the source shows the wrong next hop.
• Upstream WAN problem. The WAN is the bottleneck; the VPN has capacity but the underlying link is slow. Symptom: throughput is low; the WAN link counters are at capacity.

The cross-reference matrix:

Verification

After deploying monitoring, verify:

1. From the Prometheus exporter, every canary metric is exposed — handshake age, SA lifetime, BGP state, throughput.
2. From the alert rules, every failure mode has an alert — handshake stale, SA expiring, BGP down, throughput degraded, MTU black hole.
3. From the Grafana dashboard, all panels render data — no “no data” panels for healthy tunnels.
4. From a controlled failure (down a WireGuard interface, wait 5 minutes), the alert fires — and the operator is paged.
5. From a controlled MTU failure (set the MSS clamp to 200), the MTU black-hole test fails — and the alert fires.

A monitoring deployment that passes 1-3 but fails 4-5 has the metrics and the dashboard but no alerts wired to the on-call rotation. The discipline: alerts must page someone; a dashboard no one watches is decoration.