Network capacity — bandwidth, connections, and the tables that fill up

Network capacity on a Docker host is three separate budgets that fail in three separate ways, and the one everybody plans for — bandwidth — is almost never the one that runs out first.

The two that actually run out are connection tracking entries and ephemeral ports, both of which are consequences of the NAT that Docker’s default bridge networking performs on every packet.

Budget one: bandwidth and packet rate

bandwidth  = rps x (request_bytes + response_bytes) x 8   bits/second
packets    = rps x packets_per_transaction

Bandwidth is easy and rarely binding: a modest 10 GbE link carries a lot of JSON. Packet rate is the term that binds on small payloads, because the per-packet cost is fixed regardless of size. A host doing 200 000 packets per second of 200-byte responses is using 320 Mb/s of a 10 Gb link and a great deal of CPU in softirq.

IFACE=$(ip -o route get 192.0.2.1 2>/dev/null | awk '{for(i=1;i<=NF;i++) if($i=="dev") print $(i+1)}')
IFACE=${IFACE:-eth0}
read -r RX1 TX1 PR1 PT1 < <(awk -v i="$IFACE" '$1 ~ i":" {gsub(/.*:/,"",$1); print $2, $10, $3, $11}' /proc/net/dev)
sleep 10
read -r RX2 TX2 PR2 PT2 < <(awk -v i="$IFACE" '$1 ~ i":" {gsub(/.*:/,"",$1); print $2, $10, $3, $11}' /proc/net/dev)
echo "rx: $(( (RX2-RX1)*8/10/1000000 )) Mb/s  tx: $(( (TX2-TX1)*8/10/1000000 )) Mb/s"
echo "pps in: $(( (PR2-PR1)/10 ))  pps out: $(( (PT2-PT1)/10 ))"

rx: 118 Mb/s  tx: 342 Mb/s
pps in: 41200  pps out: 39880

ip -s link show docker0
nstat -az | grep -E 'TcpExtListenOverflows|TcpExtListenDrops|TcpExtTCPBacklogDrop'

TcpExtListenOverflows           0                  0.0
TcpExtListenDrops               0                  0.0
TcpExtTCPBacklogDrop            0                  0.0

ListenOverflows above zero means a listening socket’s accept queue was full and connections were dropped. That is a capacity signal with no ambiguity in it: the application is not accepting fast enough, and raising net.core.somaxconn only moves the queue, it does not add capacity.

Budget two: the conntrack table

Every connection through a Docker bridge network is NATed, and every NATed connection occupies an entry in the kernel’s connection-tracking table. The table has a fixed size. When it is full, new connections are dropped, not queued, and the kernel logs nf_conntrack: table full, dropping packet.

sysctl net.netfilter.nf_conntrack_max net.netfilter.nf_conntrack_count

net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_count = 827

Sizing it

An entry exists for the life of the connection plus a timeout after it closes. That timeout is the term people forget:

entries = (new_connections_per_second x mean_connection_lifetime)
        + (closes_per_second x conntrack_timeout)

The default nf_conntrack_tcp_timeout_time_wait is 120 seconds. A service closing 2 000 connections a second therefore carries 240 000 entries of pure residue — which on the default table of 262 144 leaves room for about 20 000 live connections before the table fills.

Worked example for an edge proxy on a bridge network:

live connections    = 3000 concurrent
new/sec             = 2000
close timeout       = 120 s   (tcp_timeout_time_wait)
                      
entries = 3000 + (2000 x 120)   = 243000
sized at 2x for burst           = 486000  ->  nf_conntrack_max = 524288

# /etc/sysctl.d/60-conntrack.conf
net.netfilter.nf_conntrack_max = 524288
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30

Shortening time_wait from 120 to 30 seconds is usually the cheaper half of the fix, and for a host whose peer is a load balancer it is safe. Do not shorten it below 30 on a network with meaningful reordering.

Avoiding conntrack entirely

Containers on the same user-defined bridge talk to each other without NAT, so east-west traffic between containers costs conntrack entries only because of the bridge’s masquerade rule for egress. Two designs avoid the table:

• --network host for the one high-connection-rate component, which removes NAT and port publishing from the path. It also removes network isolation, so it is a deliberate trade rather than a default.
• macvlan or ipvlan, which give the container an address on the physical network with no NAT at all.

Both are covered elsewhere in the course; here the point is that they are capacity decisions as well as networking ones.

Budget three: ephemeral ports

A container making outbound connections allocates a source port from the ephemeral range for each one. The range is finite.

sysctl net.ipv4.ip_local_port_range

net.ipv4.ip_local_port_range = 32768	60999

The arithmetic that catches people:

available ports  = 60999 - 32768 + 1        = 28232
ports consumed   = new_conns/sec x time_wait

A worker opening 300 connections a second to a single database endpoint, with a 120-second TIME_WAIT, holds 36 000 ports against a 28 232-port range. It exhausts the range and outbound connections start failing with EADDRNOTAVAIL — “cannot assign requested address” — which is a message that sends people to look at DNS and firewalls for an hour.

ss -s | head -3
ss -tan state time-wait | wc -l

Total: 319
TCP:   79 (estab 20, closed 41, orphaned 0, timewait 0)

1

The fixes, in order of preference:

1. Connection pooling in the application. 300 connections a second to one database is almost always a missing pool. This removes the problem rather than raising its ceiling.
2. Keep-alive on HTTP clients, for the same reason.
3. Widen the range: net.ipv4.ip_local_port_range = 10240 65535 gives 55 296 ports. Do not go below 10240; you will collide with services that bind fixed ports.
4. Shorten TIME_WAIT via nf_conntrack_tcp_timeout_time_wait on the NAT path.

Which budget binds first

Work down that table in order. The bottom row is the one everyone checks first and the one that is almost never the answer.

Sanity check