Container network performance — the datapath and how to measure it

This part has covered storage drivers, image size, startup and the Linux performance toolkit. The remaining dimension is the network, and it is the one where containers add genuinely new hops between the application and the wire.

The networking part of this course covers how those hops are configured. This lesson is about what they cost and how to find out — which is a different discipline, because a network performance problem is almost always misattributed on the first guess.

The datapath

flowchart LR
  A[App in container] --> B[eth0<br/>veth peer]
  B --> C[vethXXXX<br/>on host]
  C --> D[docker0 / br-*<br/>bridge]
  D --> E[netfilter<br/>NAT + conntrack]
  E --> F[Host NIC]

Each box is work the kernel does per packet. Compared with a process on the host, a container on the default bridge adds the veth pair, the bridge, and address translation with connection tracking.

The three network modes differ in how many of those they keep:

Measure each hop, do not guess

The discipline is to bisect. Run the same test at four points and the difference between two adjacent results names the hop responsible.

Start a server and client with a network toolkit image on the same host:

$ docker run -d --name iperf-server --network bench networkstatic/iperf3 -s

a41f7c93b2e08d5a6f1c4b7e0a29d3f5061b8c2d4e6f8a0b2c4d6e8f0a2b4c6d

$ docker run --rm --network bench networkstatic/iperf3 -c iperf-server -t 10 -f g

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  32.4 GBytes  27.8 Gbits/sec  0    sender
[  5]   0.00-10.00  sec  32.4 GBytes  27.8 Gbits/sec       receiver

Then the same test with --network host, which removes the veth pair and the bridge:

$ docker run --rm --network host networkstatic/iperf3 -c 127.0.0.1 -t 10 -f g

[  5]   0.00-10.00  sec  44.1 GBytes  37.9 Gbits/sec  0    sender

The gap between those two is the cost of the container datapath on this host, for this traffic pattern. Note that the figure is irrelevant in absolute terms — it depends entirely on your CPU — and the ratio is what you carry forward.

Extend the bisection outward when the problem is not on the host:

1. Container to container, same host, same network — isolates veth and bridge.
2. Container to the host IP — adds the published-port path and NAT.
3. Host to host, no containers involved — this is the physical network, and if it is slow here, nothing about Docker is the cause.
4. Container on host A to container on host B — the sum of everything.

Most investigations end at step three, with the discovery that the link itself, a switch, or an offload setting is responsible and containers were never involved.

The userland proxy

MTU, and the bug that looks like everything else

This is the highest-value diagnosis in the lesson because the symptom is so misleading.

Find the working MTU by bisection with the do-not-fragment bit set. The payload size is 28 bytes less than the MTU, for the IP and ICMP headers:

$ docker exec api ping -c 2 -M do -s 1472 192.0.2.10

PING 192.0.2.10 (192.0.2.10) 1472(1500) bytes of data.
ping: local error: message too long, mtu=1450

--- 192.0.2.10 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss

$ docker exec api ping -c 2 -M do -s 1422 192.0.2.10

PING 192.0.2.10 (192.0.2.10) 1422(1450) bytes of data.
1430 bytes from 192.0.2.10: icmp_seq=1 ttl=62 time=0.412 ms
1430 bytes from 192.0.2.10: icmp_seq=2 ttl=62 time=0.398 ms

1450 is the path MTU. Compare with what the interfaces claim:

$ ip -o link show docker0 | grep -o 'mtu [0-9]*'

mtu 1500

Set the network’s MTU to match the path. For a user-defined network:

docker network create --opt com.docker.network.driver.mtu=1450 bench

Or as a host-wide default in /etc/docker/daemon.json:

{
  "mtu": 1450
}

Existing networks keep the MTU they were created with, so a change here needs the networks recreated, which needs the containers recreated.

Connection tracking as a ceiling

Every connection through the bridge consumes a conntrack entry, and the table is finite:

$ cat /proc/sys/net/netfilter/nf_conntrack_max /proc/sys/net/netfilter/nf_conntrack_count

262144
1217

When count approaches max, new connections are dropped and the kernel says so:

nf_conntrack: table full, dropping packet

The symptom at the application is intermittent connection failures and timeouts under load, with no corresponding error anywhere in the application logs — because the connection never arrived. Every other signal looks healthy, which puts this alongside the MTU bug in the category of faults that are cheap to find and expensive to guess at.

Two things drive occupancy: a high connection rate, and entries that linger. TCP entries in TIME_WAIT are held for a timeout after the connection closes, so a service opening and closing thousands of short-lived connections per second can occupy far more of the table than its concurrency suggests. Monitor the ratio rather than waiting for the log line, and raise nf_conntrack_max if the workload is legitimate — it is a host-wide sysctl, so the decision affects everything on the host.

Knowledge check

Where next

That closes the performance part. The troubleshooting part applies these measurements to specific reported faults, and the capacity part turns the healthy baselines into sizing decisions.