The bandwidth-delay product

A 10 Gb/s link between two continents delivers 8 Mbit/s to a single scp. Both NICs report a clean link, a ping test shows no loss, and the switches are idle. This is the most common “the network is slow” report that is not a network fault, and it is entirely predictable from two bounds - one set by the window size, one set by the loss rate. Neither of them involves the link rate.

TCP can have at most one window of unacknowledged data in flight. To keep a pipe full, the window must be at least as large as the data that fits in the pipe:

BDP (bytes) = bandwidth (bits/s) / 8 x RTT (seconds)

Measure the RTT first:

ping -c 20 198.51.100.24 | tail -2

20 packets transmitted, 20 received, 0% packet loss, time 19031ms
rtt min/avg/max/mdev = 148.221/149.882/152.004/0.911 ms

Then compute, for a 10 Gb/s path at 150 ms:

awk 'BEGIN {
  bw_bps  = 10e9
  rtt_s   = 0.150
  bdp     = bw_bps / 8 * rtt_s
  printf "BDP = %.1f MB\n", bdp / 1048576
}'

BDP = 178.8 MB

To fill that path with one flow, 179 MB must be in flight. Now compare against what the kernel will allow:

sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem net.core.rmem_max net.core.wmem_max

net.ipv4.tcp_rmem = 4096	131072	6291456
net.ipv4.tcp_wmem = 4096	16384	4194304
net.core.rmem_max = 212992
net.core.wmem_max = 212992

The three values in tcp_rmem are minimum, default and maximum. Receive autotuning will grow the buffer up to 6 MB - which is 3% of the 179 MB needed. These defaults vary by distribution and by kernel version, so read them on the host rather than assuming the numbers above; some distributions ship a 32 MB ceiling, which changes the arithmetic but not the shape of the problem.

The ceiling on throughput follows directly from whatever the maximum turns out to be:

awk 'BEGIN {
  win_bytes = 6291456
  rtt_s     = 0.150
  printf "ceiling = %.1f Mbit/s\n", win_bytes * 8 / rtt_s / 1e6
}'

ceiling = 335.5 Mbit/s

335 Mbit/s on a 10 Gb/s link, with no loss and no congestion. The link is not slow; the window is small.

Raise the autotuning maximum, not the default

The fix is to raise the third value of tcp_rmem and tcp_wmem, which is the ceiling autotuning may grow to. Leave the default - the middle value - alone:

# Ceiling of 256 MB, for a 10 Gb path at 150 ms RTT with headroom.
sudo sysctl -w net.ipv4.tcp_rmem='4096 131072 268435456'
sudo sysctl -w net.ipv4.tcp_wmem='4096 16384 268435456'

Raising the default instead allocates that much per socket immediately, so a host with 50,000 connections commits an enormous amount of memory to buffers it does not need. Autotuning exists precisely so that the ten flows that need a large window get one while the other 49,990 do not.

net.core.rmem_max and net.core.wmem_max cap what an application can request explicitly with SO_RCVBUF/SO_SNDBUF. They do not cap TCP autotuning, which is governed by tcp_rmem/tcp_wmem. An application that sets SO_RCVBUF itself also disables autotuning for that socket and is then stuck at whatever it asked for - a surprisingly common cause of a tuned host that still runs slow.

Check the whole memory picture before raising anything, because these buffers come out of the same pool:

sysctl net.ipv4.tcp_mem
cat /proc/net/sockstat

Window scaling must also be on, or the window cannot exceed 64 KB regardless of the buffers:

sysctl net.ipv4.tcp_window_scaling

net.ipv4.tcp_window_scaling = 1

It has been the default for decades. It is worth checking anyway, because a middlebox that strips the window-scale TCP option produces exactly this symptom on a path where both endpoints are configured correctly.

Per-flow evidence with ss -ti

Arithmetic predicts a ceiling. ss -ti shows what a live flow actually achieved:

ss -ti state established '( dport = :443 )' | head -20

ESTAB  0  1179648  192.0.2.15:51234  198.51.100.24:443
	 cubic wscale:7,7 rto:352 rtt:149.882/0.911 mss:1448 pmtu:1500
	 rcvmss:536 advmss:1448 cwnd:1443 ssthresh:721 bytes_sent:184320000
	 bytes_retrans:1024 bytes_acked:184318976 segs_out:127293
	 data_segs_out:127280 send 111.5Mbps lastsnd:4 lastrcv:1204
	 pacing_rate 133.8Mbps delivery_rate 108.2Mbps busy:1651ms
	 retrans:0/12 rcv_space:14480 rcv_ssthresh:64088 minrtt:148.221

The fields that matter:

Field	Meaning
`cubic`	The congestion control algorithm in use for this flow
`rtt:149.882/0.911`	Smoothed RTT and its variance, in ms
`minrtt`	The lowest RTT ever seen - the path’s true propagation delay
`cwnd:1443`	Congestion window, in segments. Multiply by `mss` for bytes
`ssthresh`	Slow-start threshold; a value below `cwnd` means loss has been seen
`bytes_retrans`	Bytes retransmitted on this flow
`delivery_rate`	The rate actually achieved, measured by the kernel
`pacing_rate`	The rate the kernel is willing to send at
`wscale:7,7`	Window scale factors, send and receive. Missing means no scaling

Turn cwnd into bytes: 1443 segments times 1448 bytes is about 2.1 MB in flight. Against a 179 MB BDP, this flow is using 1% of the pipe, which matches the 108 Mbit/s delivery_rate.

Two readings distinguish the two causes:

cwnd large, bytes_retrans near zero, but throughput low. The receive window is the limit. Look at the peer’s rcv_space and its tcp_rmem ceiling.
cwnd small and ssthresh well below it, with bytes_retrans climbing. Loss is the limit, and no amount of buffer will fix it.

Loss, and why it dominates on long paths

This course has already established the Mathis bound:

throughput  ~  (MSS / RTT) x (1 / sqrt(p))

The consequence on a long path is severe, and it is worth computing rather than asserting:

awk 'BEGIN {
  mss = 1448; rtt = 0.150
  split("0.0001 0.001 0.01", losses, " ")
  for (i = 1; i <= 3; i++) {
    p = losses[i] + 0
    printf "loss %6.4f%%  ceiling %8.1f Mbit/s\n", \
           p * 100, mss * 8 / rtt / sqrt(p) / 1e6
  }
}'

loss 0.0100%  ceiling      7.7 Mbit/s
loss 0.1000%  ceiling      2.4 Mbit/s
loss 1.0000%  ceiling      0.8 Mbit/s

A hundredth of a percent of loss caps a single flow at 7.7 Mbit/s on this path. One percent caps it at 0.8. The link rate never enters the calculation at all - which is why “we upgraded to 10 Gb and nothing got faster” is such a familiar report.

Compare the two bounds we now have for this path:

Bound	Ceiling
Window: 6 MB over 150 ms RTT	335 Mbit/s
Loss: Mathis at 0.01%	7.7 Mbit/s

Throughput is limited by whichever is lower, so on this path a loss rate too small for a ping test to detect dominates completely. Establish the loss rate before you tune buffers. Raising tcp_rmem to 256 MB on a path with 0.01% loss moves the window bound from 335 Mbit/s to 14 Gbit/s and changes the achieved throughput by nothing at all, because the flow was never against that bound.

A ping test cannot clear a path for bulk transfer

ping sends a few dozen small packets and reports “0% packet loss”. At a real loss rate of 0.05% you would need roughly 2,000 packets to see one drop, so a 20-packet test reports clean on a path that is capping a single TCP flow at a few hundred Mbit/s.

Two better tests. Read the actual per-flow retransmission count from a transfer that ran:

ss -ti state established | grep -E 'bytes_(sent|retrans)'

Or send enough packets for the rate to be measurable:

ping -c 2000 -i 0.01 198.51.100.24 | tail -2

2000 packets transmitted, 1999 received, 0.05% packet loss, time 21882ms
rtt min/avg/max/mdev = 148.102/149.771/168.442/1.204 ms

-i 0.01 requires root below 0.2 s on most systems. And note the asymmetry: ping loss is bidirectional, whereas TCP cares mostly about loss in the data direction, so the two numbers do not have to agree.

Congestion control

sysctl net.ipv4.tcp_congestion_control
cat /proc/sys/net/ipv4/tcp_available_congestion_control

net.ipv4.tcp_congestion_control = cubic
cubic reno

cubic is the default nearly everywhere. It is loss-based: it grows the window until a packet is dropped, then backs off. That works well when loss means congestion, and works badly when loss is caused by anything else - a lossy radio link, a policer, or a cable with a marginal connector. On such a path, cubic interprets every random drop as congestion and keeps its window small.

bbr models the path instead, estimating bottleneck bandwidth and minimum RTT and pacing to that estimate. It largely ignores loss that is not accompanied by evidence of queueing, which is why it can be dramatically faster on a long path with a low background loss rate.

# bbr may need its module loaded before it appears as available.
sudo modprobe tcp_bbr
cat /proc/sys/net/ipv4/tcp_available_congestion_control

reno cubic bbr

sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

This is a sender-side setting and takes effect on new connections. It requires no change at the far end, which makes it unusually easy to A/B test: set it, re-run the transfer, compare delivery_rate.

bbr is not a general speed-up, and it changes how you share a link

Three things worth knowing before rolling it out.

It does not fix a small window. If tcp_rmem caps the window below the BDP, bbr is capped by exactly the same arithmetic. Fix the buffers first, then evaluate the algorithm.

It does not fix real congestion. On a genuinely oversubscribed link, bbr competing against cubic flows can take a larger share - which looks like an improvement on the host you changed and a regression on every other host using that link. On a shared corporate link, that is somebody else’s incident caused by your change.

It benefits from fq. bbr paces its sends, and pacing works best with the fq queueing discipline:

tc qdisc show dev eth0
sudo tc qdisc replace dev eth0 root fq

Modern kernels can pace internally without fq, so this is a recommendation rather than a requirement - but check what the qdisc actually is before concluding that bbr underperformed.

Evaluate it with a measurement on both sides: delivery_rate and bytes_retrans from ss -ti on the sender, and the application’s own throughput. Roll it out per-service if you can, rather than fleet-wide from one good result.

Parallel streams: the workaround that works

When the path has loss you cannot fix and the transfer must go faster today, run several flows. Each is capped by the Mathis bound independently, so N flows get roughly N times the throughput:

iperf3 -c 198.51.100.24 -P 8 -t 30

This is what rsync cannot do and what modern object-storage clients do by default. It is a workaround, not a fix - it consumes N times the share of a congested link, so it is antisocial on a shared path and entirely reasonable on a dedicated one. Say which you are on before reaching for it.