Switch saturation and buffering

Why this matters in production

Ceph produces a traffic pattern — many senders to one receiver — that exhausts switch buffers more readily than most workloads, and the resulting loss degrades Ceph disproportionately.

Incast

a client reads an erasure-coded object
  → k OSDs respond simultaneously
  → all responses converge on one switch port
  → the port's buffer must absorb k × chunk size instantly
  → if it cannot, frames are dropped

flowchart LR
  A[OSD 1] --> S[Switch port buffer]
  B[OSD 2] --> S
  C[OSD 3] --> S
  D[OSD 4] --> S
  S --> E[Client port<br/>buffer overflows]

This pattern is called incast, and it is worse with wider erasure coding, larger objects, and shallower switch buffers.

Reading the evidence

# host side
IFACE=bond0
ethtool -S ${IFACE} | grep -iE 'drop|discard|missed|no_buffer|pause'
netstat -s | grep -iE 'retransmit|timeout|listen drops'
ip -s link show ${IFACE}

# TCP-level evidence of loss
ss -ti | grep -oE 'retrans:[0-9]+/[0-9]+' | head

On the switch, the counters that matter are output discards and buffer occupancy:

show interface ethernet 1/12 counters errors
show queuing interface ethernet 1/12
show hardware internal buffer info pkt-stats

Output discards on a port facing a Ceph client or OSD are direct evidence of incast.

Symptoms in Ceph

The last is diagnostic: if an EC pool is disproportionately affected relative to a replicated one on the same hardware, incast is a strong candidate.

Mitigations

# reduce the burst Ceph generates
ceph config set osd osd_max_backfills 2
ceph config set osd osd_recovery_max_active_hdd 3

Reducing what Ceph sends is the mitigation available without touching the network, and it is often enough during an incident while a fabric change is planned.

Quiz

Production discipline

Check switch output discards and host retransmit counters before attributing latency to devices; incast loss produces spikes far larger than the loss rate suggests. Reducing recovery concurrency lowers the burst Ceph generates and is the only mitigation available on an incident timescale.

Cross-course references

• Kubernetes: fan-out request patterns produce the same incast on node interfaces
• Linux: TCP retransmission timeouts dominate tail latency in any lossy fan-in pattern