Asymmetric paths and one-way problems

Why this matters in production

Most network diagnostics assume that if A can reach B, B can reach A. Modern fabrics with ECMP, policy routing, and multiple uplinks routinely violate that, and the resulting failures are among the hardest to diagnose because every symmetric test comes back clean.

How asymmetry arises

• ECMP hashing sends the two directions of a flow over different physical paths
• Policy routing or source-based routing on multi-homed hosts
• One failed member of a bond in a design where directions hash differently
• Asymmetric firewall rules permitting one direction only
• A single degraded optic in one direction of a bidirectional pair

What Ceph shows you

The symptoms are confusing precisely because they are one-sided:

• osd.A reports osd.B down, while osd.B reports osd.A perfectly reachable
• Heartbeats fail in one direction only
• Writes hang while reads work — or the reverse, depending on direction
• ping succeeds because it exercises both directions and only needs one packet each way to work sometimes

journalctl -u ceph-osd@12 | grep heartbeat_check
# osd.12 heartbeat_check: no reply from osd.47 back
# ... while osd.47's log shows no complaint about osd.12

Asymmetric complaint is the diagnostic. When A complains about B and B does not complain about A, the path from B to A is the broken one.

Directional testing

# per-hop, per-direction — run from BOTH ends
mtr --report --report-cycles 500 10.30.0.47      # from A
mtr --report --report-cycles 500 10.30.0.12      # from B

# directional throughput
iperf3 -c 10.30.0.47 -t 30            # A → B
iperf3 -c 10.30.0.47 -t 30 -R         # B → A (reverse)

The -R flag is the essential one. A test that only measures one direction will pass on an asymmetric fault half the time.

# confirm packets actually arrive
tcpdump -i any -n host 10.30.0.47 and port 6800 -c 100

Run tcpdump on both ends simultaneously: packets leaving A and not arriving at B localises the fault to the path, definitively.

Asymmetric routing is not necessarily a fault

Traffic taking different paths in each direction is normal and works fine, provided both paths are healthy. The problem arises when one of them is degraded — then only that direction suffers, and only tests exercising it will find it.

Quiz

Production discipline

Make bidirectional testing the default in your network verification procedure — iperf3 -R and mtr from both ends — because a one-directional pass on an asymmetric fault is actively misleading. When OSD complaints are one-sided, start from the direction the logs name rather than re-running symmetric tests that have already passed.

Cross-course references

• Kubernetes: one-way NetworkPolicy or security-group rules produce the same asymmetric behaviour
• Linux: asymmetric routing with reverse path filtering is a related and equally confusing class of fault