Crypto performance — AES-NI offload, IPsec throughput, kTLS, single-core bottleneck

Encryption is the most CPU-bound workload a router runs. A 10 Gbps link carrying AES-256-GCM-encrypted traffic can saturate a multi-core router’s CPU if the encryption pipeline is not offloaded. The bottleneck is usually not in the cryptographic primitive itself — modern AES-NI hardware handles AES at line rate — but in the way the packets are distributed to the encryption cores, the queueing discipline between encryption and transmission, and the per-flow locking that prevents a single flow from monopolising multiple cores.

This lesson is the production reference for crypto performance on VyOS 1.5 LTS: AES-NI verification, the IPsec single-core bottleneck, kTLS for HTTPS termination, WireGuard’s profile, and the canonical fixes for crypto saturation.

The crypto performance stack

flowchart LR
  subgraph APP["Application layer"]
    TLS["HTTPS / TLS termination<br/>(nginx, haproxy, ssh)"]
  end
  subgraph TUN["Tunnel layer"]
    WG["WireGuard<br/>(kernel wg, Noise protocol)"]
    IP["IPsec<br/>(kernel XFRM, ESP)"]
    VX["VXLAN<br/>(kernel vxlan)"]
  end
  subgraph HW["Hardware layer"]
    NIC["NIC<br/>(AES-NI, kTLS offload)"]
    CPU["CPU<br/>(AES-NI instructions)"]
  end
  APP --> HW
  TUN --> HW

Each layer has its own CPU profile and its own bottleneck. The operator must read mpstat, top, and protocol-specific counters to identify which layer is saturating.

AES-NI hardware offload

Modern x86 CPUs include AES-NI instructions that perform AES encryption 10-100x faster than software AES. Without AES-NI, IPsec throughput is capped at 1-3 Gbps per core; with AES-NI, the same core can do 10-40 Gbps.

The operator verifies AES-NI availability:

$ cat /proc/cpuinfo | grep -E "^flags" | head -1 | tr ' ' '\n' | grep -E "aes|avx|sha_ni"
aes
avx
avx2
sha_ni

If aes is present, the kernel can use AES-NI. The operator verifies the kernel modules are loaded:

$ lsmod | grep aes
aesni_intel          368640  0
cryptd               245760  3  aesni_intel,ghash_clmulni_intel,sha1_ssse3

If the aesni_intel module is not loaded, the operator loads it explicitly:

$ sudo modprobe aesni_intel
$ sudo systemctl restart strongswan

The operator benchmarks AES throughput:

$ openssl speed -evp aes-256-gcm
type             16 bytes     64 bytes    256 bytes    1024 bytes   8192 bytes
aes-256-gcm    1234567.89k   4567890.12k  12345678.90k  23456789.01k  34567890.12k

The throughput per byte size tells the operator the peak crypto rate. A 10 Gbps link carrying 256-byte packets needs ~3.7 Gbps of crypto throughput; the benchmark shows whether the CPU can sustain that.

IPsec single-core bottleneck

strongSwan on VyOS 1.5 LTS uses the Linux kernel’s XFRM subsystem for IPsec encryption. The XFRM subsystem encrypts each packet in softirq context on the CPU that processed the packet’s ingress. With single-NIC interrupt pinning (Part L-02), all encrypted traffic flows through a single core’s XFRM pass.

flowchart LR
  subgraph BEFORE["Before RSS"]
    ETH0["eth0 (10Gbps)"] --> CPU0["CPU 0<br/>softirq + XFRM<br/>(saturated)"]
    CPU0 --> X["1-3 Gbps"]
  end
  subgraph AFTER["After RSS"]
    ETH0A["eth0"] --> HASH["RSS"]
    HASH --> Q0["Q0 → CPU 0<br/>softirq + XFRM"]
    HASH --> Q1["Q1 → CPU 1<br/>softirq + XFRM"]
    Q0 --> Y["5-10 Gbps"]
    Q1 --> Y
  end

The fix is the same as for non-encrypted traffic: distribute the ingress interrupts across cores with RSS. Each core then encrypts its share of the traffic in softirq.

But IPsec has a secondary bottleneck: the encrypted packets must be sent to the egress interface, which may be on a different core. The cross-core traffic adds latency. The mitigation is XFRM device offload (a kernel feature) or IPsec hardware offload (a NIC feature that moves the encryption to the NIC’s processor).

The operator checks for NIC-level IPsec offload:

$ ethtool -k eth0 | grep -i "esp"
esp-hw-offload: on
esp-tx-csum-hw-offload: on

If esp-hw-offload: on, the NIC does the encryption in hardware. The operator verifies the offload is working:

$ ip -s xfrm state
src 203.0.113.1 dst 198.51.100.1
    proto esp spi 0xc8e0a8b1 reqid 1 mode tunnel
    ...
    hw-aead: cbc(aes)
    ...
    bytes 1234567890

The hw-aead line confirms the hardware is doing the encryption. If the line is absent, the encryption is CPU-bound.

kTLS for HTTPS / TLS termination

For HTTPS/TLS termination, the kernel’s kTLS (kernel TLS) subsystem offloads the TLS record processing to the kernel, freeing the userspace application (nginx, haproxy) from per-record crypto work. With NIC support, the TLS record processing can be further offloaded to the NIC.

The operator enables kTLS at boot:

configure
set system kernel module ktls enable
commit

And the operator verifies the NIC supports kTLS offload:

$ ethtool -k eth0 | grep tls
tls-hw-tx-offload: on
tls-hw-rx-offload: on

For nginx/haproxy on VyOS, kTLS requires the application to opt in via setsockopt(TCP_ULP, "tls"). The operator configures the application to use kTLS.

The kTLS throughput profile is much higher than userspace TLS: a single core can do 10-40 Gbps with kTLS, vs 1-3 Gbps with userspace TLS. For an HTTPS-heavy workload (a load balancer or a TLS-terminating proxy), kTLS is the canonical fix for CPU saturation.

WireGuard profile

WireGuard uses modern cryptography (ChaCha20-Poly1305 by default, Curve25519 for key exchange, BLAKE2s for hashing) that is fast on CPUs without AES-NI. WireGuard runs entirely in kernel space (the wg kernel module) and processes packets in softirq context.

$ wg show wg0
interface: wg0
  public key: <redacted>
  private key: (hidden)
  listening port: 51820
  peer: <redacted>
    endpoint: 203.0.113.5:51820
    allowed ips: 10.0.0.0/24
    latest handshake: 12 seconds ago
    transfer: 1.23 GiB received, 987.65 MiB sent
    persistent keepalive: every 25 seconds

WireGuard’s CPU profile is more balanced than IPsec because each packet’s encryption can run on the CPU that received it (no per-tunnel locking). On a multi-core router with RSS, WireGuard can scale across cores more naturally.

The operator benchmarks WireGuard throughput with iperf3:

# On the WireGuard peer
$ iperf3 -s -B 10.0.0.1

# On the VyOS router
$ iperf3 -c 10.0.0.1 -P 4

The result tells the operator the aggregate throughput. For a single WireGuard tunnel, expect 5-10 Gbps on modern hardware with AES-NI and RSS.

Production failure modes

The crypto performance failure modes the operator encounters:

• AES-NI not loaded. cat /proc/cpuinfo | grep aes returns the flag but lsmod | grep aes returns nothing. The kernel modules are not loaded; the kernel uses software AES; throughput is 10x slower. Fix: load the module or upgrade the kernel.
• IPsec pinned to one core. mpstat shows one core at 100% %soft and others idle. The encryption is on a single core. Fix: RSS for ingress, or NIC-level IPsec offload.
• IPsec without AES-NI. cat /proc/cpuinfo | grep aes returns nothing (the CPU does not support AES-NI, e.g., some AMD EPYC early-gen or some ARM CPUs). Throughput is capped at 1-2 Gbps. Fix: hardware upgrade (a CPU with AES-NI), or use ChaCha20-Poly1305 (WireGuard), which does not require AES-NI.
• TLS termination saturating userspace. nginx or haproxy consumes 100% of one or more cores because userspace TLS is slow. Fix: enable kTLS.
• StrongSwan rekey storm. A peer rekeys every 60 seconds; the rekey is CPU-intensive (Curve25519 or RSA key exchange). Fix: raise ike lifetime and lifetime to reduce rekey frequency; or move to IKEv2 with mobike for roaming.
• Tunnel MTU mismatch causing ICMP frag needed drops. Packets are dropped at the IPsec layer because they exceed the tunnel MTU. Fix: set the inner MTU (Part LI-02 and Part LI-05).

Rollback

Crypto performance fixes are typically small but the impact is large. The rollback discipline:

• modprobe aesni_intel — load only; unload with modprobe -r aesni_intel (rarely needed; the module is auto-loaded).
• ethtool -K <nic> esp-hw-offload off — disable NIC-level IPsec offload.
• system kernel module ktls — delete system kernel module ktls and commit.
• StrongSwan / WireGuard knobs — delete ... and commit.

For all changes, use commit-confirm 5:

configure
# ... make the change ...
commit-confirm 5
# If the change causes throughput regression or tunnel loss,
# the auto-rollback restores the previous configuration.

Production discipline

Cross-course references

• Part L-01 (L-VyOS-Performance / CPU saturation) covers the diagnostic method that identifies single-core softirq saturation, which is the root cause of many crypto bottlenecks.
• Part L-02 (L-VyOS-Performance / interrupt affinity) covers RSS, which is the canonical fix for single-core softirq on encrypted traffic.
• Part XLII (XLII-VyOS-IPsec) covers IPsec configuration; this lesson assumes the strongSwan configuration is in place.
• Part XLI (XLI-VyOS-WireGuard) covers WireGuard configuration; this lesson assumes the WireGuard configuration is in place.
• The Linux course’s V-Linux-NetConfig and XXII-Linux-NetTroubleshoot parts cover the same primitives from the host perspective (kTLS, XFRM, AES-NI).
• The Observability course covers the telemetry side (crypto throughput dashboards in Grafana).

Quiz