VPN protocol comparison — IPsec, WireGuard and OpenVPN at the firewall

OPNsense supports three VPN protocols out of the box: IPsec, WireGuard, and OpenVPN. They are not interchangeable. Each has a different architecture, a different performance profile, a different way of traversing NAT and firewalls, and a different operational footprint. Choosing the right one for a deployment is the difference between a VPN that works reliably for years and one that produces a steady stream of troubleshooting tickets.

This lesson compares the three protocols on the dimensions the operator has to weigh, identifies the scenarios each fits, and gives the operator the framework for making the choice deliberately rather than picking whatever the GUI shows first.

The three architectures

IPsec is a suite of protocols standardised by the IETF, with the firewall operator-facing pieces being IKE (Internet Key Exchange) for key negotiation and ESP (Encapsulating Security Payload) for the encrypted tunnel. IPsec runs at the network layer — it is part of the IP stack, not a userspace application. The current generation uses IKEv2 (RFC 7296) for the handshake.

IPsec has two modes: transport mode (encrypts only the payload, leaves the original IP header) and tunnel mode (encrypts the entire inner packet and wraps it in a new outer header). Production firewalls use tunnel mode almost exclusively.

WireGuard runs at the network layer too, but as a single, monolithic protocol designed from scratch with modern cryptography. It uses UDP for transport (typically port 51820), Curve25519 for the handshake, ChaCha20-Poly1305 for the bulk encryption, and a fixed set of cipher choices — there are no negotiation options to misconfigure because there is no negotiation.

OpenVPN runs at the application layer (userspace), uses TLS for the handshake (the same TLS that secures HTTPS), and tunnels traffic over UDP or TCP (typically port 1194 UDP or 443 TCP). OpenVPN is the most flexible of the three — almost any combination of cipher, transport, and authentication is possible — but that flexibility comes at the cost of complexity.

Performance

The performance difference is dramatic:

• IPsec with AES-GCM and AES-NI (hardware acceleration) approaches line rate on capable CPUs. A modern x86 server CPU with AES-NI handles 10 Gbit/s of IPsec traffic.
• WireGuard approaches line rate on every modern CPU because ChaCha20-Poly1305 is designed to be fast on CPUs without hardware acceleration. On a Raspberry Pi 4, WireGuard achieves ~1 Gbit/s; on a server CPU, it matches IPsec.
• OpenVPN is userspace and single-threaded per flow. Even with AES-NI, performance is significantly lower than IPsec or WireGuard. A typical OpenVPN deployment on OPNsense achieves 200-400 Mbit/s; a WireGuard or IPsec deployment on the same hardware achieves 1-5 Gbit/s.

The implication: for high-throughput deployments (1 Gbit/s+), WireGuard or IPsec is the right answer; OpenVPN is the wrong one. For low-throughput remote access with many users, the performance difference matters less and the protocol choice can be driven by other factors.

NAT and firewall traversal

A VPN endpoint behind NAT cannot be reached by IPsec in its native form — IPsec ESP (protocol 50) does not have port numbers and cannot traverse NAT cleanly. The fix is NAT-Traversal (NAT-T), which encapsulates ESP in UDP port 4500. NAT-T works through most NAT devices, but it adds overhead and complexity.

WireGuard runs over UDP (port 51820) and traverses NAT natively. There is no separate NAT-T mode; WireGuard is NAT-T by design. A WireGuard endpoint behind NAT sends keepalive packets every 25 seconds by default; the NAT device’s mapping stays alive; the connection survives.

OpenVPN over UDP also traverses NAT natively. OpenVPN over TCP (typically port 443, to look like HTTPS) traverses even the most restrictive firewalls — corporate networks, hotels, captive portals that block everything except port 80 and 443. The TCP-over-TCP overhead is real (TCP-over-TCP suffers from compounded retransmission behaviour) but for low-throughput remote access it is often acceptable.

$ tcpdump -ni igb1 'udp port 4500 or udp port 51820 or tcp port 443' -c 6

12:34:56.789012 198.51.100.1.4500 > 203.0.113.5.4500: UDP, length 168
12:34:57.012345 198.51.100.1.51820 > 203.0.113.5.51820: UDP, length 148
12:34:57.234567 198.51.100.1.51820 > 203.0.113.5.51820: UDP, length 92
12:34:57.456789 203.0.113.5.443 > 198.51.100.1.51234: TCP, length 64
12:34:57.678901 198.51.100.1.443 > 203.0.113.5.51345: TCP, length 1280
12:34:57.890123 198.51.100.1.51820 > 203.0.113.5.51820: UDP, length 92

Configuration complexity

IPsec has the most configuration surface: Phase 1 proposals, Phase 2 selectors, authentication method, lifetime, DPD (dead peer detection), NAT-T, IKEv1 vs IKEv2, route-based vs policy-based, hundreds of options. A complete IPsec configuration is a substantial piece of work, and misconfiguration is common.

WireGuard has the least: a private key, a public key, a peer (with its public key, AllowedIPs, and endpoint), and a listen port. The whole configuration fits on one screen. The minimalism is the security model — there is no negotiation to misconfigure, no proposals to weaken, no options to overlook.

OpenVPN is in between: a server or client config with cipher choices, compression, routing, authentication, and many other options. The configuration is plain text and auditable, but it is verbose.

When each fits

The discipline: pick the protocol that matches the deployment’s constraints. A site-to-site between two OPNsense firewalls with no legacy peers is a WireGuard deployment. A site-to-site to a Cisco ASA at a partner organisation is an IPsec deployment. A remote-access deployment for staff who travel to customer networks is an OpenVPN-over-TCP-443 deployment.

Summary

• IPsec is the most interoperable but the most complex; OpenVPN is the most flexible and the slowest; WireGuard is the simplest and the fastest.
• Hardware acceleration (AES-NI) closes the performance gap between IPsec and WireGuard on modern x86.
• NAT-T (IPsec), native UDP (WireGuard), and TCP 443 (OpenVPN) address firewall traversal differently. The most restrictive environments require TCP 443.
• Cipher negotiation is a feature and a footgun. Configure only modern ciphers; disable weak fallback.
• The right protocol depends on the deployment: peers, throughput, NAT, firewall traversal, and operator preference.