Packet loss and retransmits - when the network drops · Linux

Packet loss and TCP retransmits are the network’s “something is wrong” signal. A few percent loss can reduce throughput by 10x. This lesson covers detection and causes.

Detect packet loss

# ICMP loss
ping -c 100 server
# Check the "packet loss" line

# TCP loss
ss -ti
# retrans shows retransmits per connection

ss -ti shows TCP info for each connection, including retransmits.

Read TCP stats

cat /proc/net/netstat | grep -E 'Tcp:|TcpExt:'

Key fields:

TcpActiveOpens: connections initiated.
TcpPassiveOpens: connections accepted.
TcpEstabResets: connections reset.
TcpEstabResets: established connections reset.
TcpInSegs: segments received.
TcpOutSegs: segments sent.
TcpRetransSegs: segments retransmitted.
TcpInErrs: incoming errors.
TcpOutRsts: resets sent.

Retransmits vs total segments gives the retransmit rate.

Read /proc/net/snmp

cat /proc/net/snmp | grep Tcp

Same fields as netstat, in a different format.

TCP retransmits

A TCP retransmit is a packet that the sender did not get an ACK for. Common causes:

Packet loss: physical link issue, congestion, NIC driver bug.
Path change: routes changed mid-connection.
Buffer full: receive queue full; packets dropped.
Duplex mismatch: half-duplex on one side, full on the other.

A few retransmits are normal. A high rate (1% or more) is a problem.

Throughput impact

TCP throughput is bounded by the bandwidth-delay product and loss rate:

throughput = (MSS / RTT) * (1 / sqrt(p))

where p is the loss probability. A 1% loss rate can halve throughput on a high-latency link.

Common causes

Cause	Detection
Network congestion	High loss, high latency, jitter
Bad cable	Loss increases over time, often correlated with traffic
Duplex mismatch	Errors on one side, retransmits on the other
Switch/router buffer full	Drops under load, not at idle
Driver bug	`dmesg` for driver errors, ethtool for counters
Firewall drops	`nft list ruleset`, `conntrack -L`

Diagnose

# Path-level: traceroute
traceroute server

# Interface errors
ip -s link show eth0
ip -s -s -s link show eth0   # more detail

# NIC errors
ethtool -S eth0 | grep -E 'err|drop|missed'

# TCP-level
ss -tinpe

# Connection-specific
ss -ti dst :443

For UDP packet loss:

# iperf3 UDP test
iperf3 -c server -u -b 1G
# Loss shown in output

Fix

Bad cable: replace.
Duplex mismatch: force both sides to the same setting (ethtool -s eth0 autoneg off speed 1000 duplex full).
Switch buffer full: enable ECN, increase buffer.
Network congestion: reduce traffic, increase capacity.
Driver bug: update kernel or NIC driver.