Network saturation and retries — the network bottleneck

Network saturation is the performance failure that leaves every application metric looking healthy: CPU is low, memory is low, and requests still take seconds because packets are queueing or being dropped below the application. The evidence lives in retransmit counters, interface drop counts, and MTU mismatches, and none of it appears on a Pod’s own dashboard. The failure also amplifies itself — a client retrying a timed-out request adds load to the link that caused the timeout — so a retry storm can hold a network saturated long after the original cause has gone.

The network saturation

The network saturation is the degree of “fullness” of the network link. The saturation is the network’s bottleneck.

flowchart LR
    A[Pod] --> B[Network namespace]
    B --> C[Node network]
    C --> D[Cluster network]
    D --> E[External network]

The saturation is the network’s bottleneck.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
POD=checkout-api-7d9f6c8b45-r4nq2   # the Pod whose network you are examining
NS=production                       # its namespace
TARGET=payments.production.svc.cluster.local   # the peer it talks to
NODE=worker-03.example.com          # the node the Pod is scheduled on

# 1. Check the network stats
kubectl exec -it "$POD" -n "$NS" -- netstat -s

# 2. Check the packet drops
kubectl exec -it "$POD" -n "$NS" -- cat /sys/class/net/eth0/statistics/rx_dropped
kubectl exec -it "$POD" -n "$NS" -- cat /sys/class/net/eth0/statistics/tx_dropped

# 3. Check the TCP retries
kubectl exec -it "$POD" -n "$NS" -- ss -ti

# 4. Check the network latency
kubectl exec -it "$POD" -n "$NS" -- ping "$TARGET"

# 5. Check the node's network
ssh "$NODE" "ifconfig eth0"

The diagnostic is the network stats, the packet drops, the TCP retries, the network latency, and the node’s network.

Common failures

• Network link saturated. The network link is saturated. The remediation is to upgrade the link.
• Packet drops. The network is dropping packets. The remediation is to fix the network.
• TCP retries. The TCP is retrying. The remediation is to fix the network.
• MTU mismatch. The MTU is mismatched. The remediation is to align the MTU.

flowchart TD
    A[Network slow] --> B{Link saturated?}
    B -->|Yes| C[Upgrade the link]
    B -->|No| D{Packet drops?}
    D -->|Yes| E[Fix the network]
    D---|No| F{TCP retries?}
    F -->|Yes| G[Fix the network]
    F -->|No| H{MTU mismatch?}
    H -->|Yes| I[Align the MTU]
    H -->|No| J[Unknown]

The retry storm

A retry storm is the network’s amplification failure. The client retries the failed request; the server is overwhelmed; the client’s retries cause more failures; the server is more overwhelmed.

flowchart LR
    A[Client] -->|Request| B[Server]
    B -->|Timeout| A
    A -->|Retry| B
    B -->|Timeout| A
    A -->|Retry| B

The retry storm is the network’s amplification.

The remediation

The remediation depends on the cause:

# Option 1: Upgrade the link
# (network-specific)

# Option 2: Reduce the network usage
# (application-specific)

# Option 3: Add retries with exponential backoff
# (client-specific)

# Option 4: Use a service mesh
# (service mesh-specific)

The remediation is the network recovery.

Production discipline

A network saturation issue is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the network layer, identify the cause, apply the remediation. The network is the cluster’s connectivity; the remediation is the network recovery.

• Check the network stats. The network stats are the network’s view.
• Check the packet drops. The drops are the network’s failure.
• Check the TCP retries. The retries are the network’s amplification.

Quiz