Asymmetric and unequal paths

Why this matters in production

A leaf-spine fabric with ECMP hashes each flow onto a path, and different flows take different paths. When one path is degraded, some flows are affected and others are not — which produces symptoms that appear random.

How the asymmetry arises

flowchart TD
  A[Host A] --> L1[Leaf 1]
  L1 --> S1[Spine 1]
  L1 --> S2[Spine 2]
  S1 --> L2[Leaf 2]
  S2 --> L2
  L2 --> B[Host B]

A flow from A to B hashes onto spine 1 or spine 2; the return flow hashes independently and may take the other. If spine 2 has a degraded link, the outbound direction is fine and the return is not — or vice versa, per flow.

Testing both directions

# Substitute your own value before running:
HOSTB=ceph-osd-02.example.com

# forward
iperf3 -c "$HOSTB" -t 30 -P 8

# reverse, on the same connection
iperf3 -c "$HOSTB" -t 30 -P 8 -R

A large discrepancy between the two is direct evidence of an asymmetric problem.

# path visibility
HOSTB=hostB
mtr -rwzbc 100 ${HOSTB}
traceroute -n ${HOSTB}

HOST                    Loss%  Snt   Last   Avg  Best  Wrst StDev
1. 10.0.0.1              0.0%  100    0.2   0.2   0.1   0.4   0.0
2. 10.0.255.2            4.0%  100    0.4   1.9   0.3  38.2   5.1   ← this hop
3. 10.0.1.12             0.0%  100    0.3   0.3   0.2   0.6   0.1

Loss at an intermediate hop that does not appear at the destination is usually ICMP rate limiting rather than real loss; loss that persists to the final hop is real.

Identifying an unequal path

# Substitute your own value before running:
HOSTB=ceph-osd-02.example.com

# run many flows and look at the distribution
for i in $(seq 1 8); do
  iperf3 -c "$HOSTB" -t 10 -p $((5201+i)) 2>/dev/null | grep -E 'sender' &
done
wait

If some flows achieve full rate and others do not, the flows are on different paths and at least one path is degraded. Uniform results across many flows indicate the paths are equal.

# per-flow path, where the fabric supports it
HOSTB=hostB
ss -ti dst ${HOSTB} | grep -E 'cwnd|retrans'

Designing to avoid it

# from the Ceph side, the symptom is per-OSD-pair variation
SLOW_OSD=12
ceph osd perf | sort -k2 -rn | head -10
ceph osd find ${SLOW_OSD}

OSDs on one host being slow only when talking to OSDs on another specific host is the Ceph-visible signature of a path problem.

Quiz

Production discipline

Test throughput in both directions with iperf3 -R; ECMP hashes the return path independently and a forward-only test misses half the fabric. Read scattered OSD latency with no host or device pattern as a fabric signature rather than continuing to investigate devices.

Cross-course references

• Kubernetes: pod-to-pod latency varying by pair points at the underlay, not the pods
• Linux: asymmetric routing has always complicated network troubleshooting this way