Asymmetric failures — when reachability is not mutual

Advanced⏱ ~16 mincephpingss

What you'll learn

Explain how asymmetric connectivity arises
Predict monitor behaviour under one-way reachability
Diagnose asymmetric faults with directional testing
Distinguish asymmetric partition from monitor resource problems

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A clean partition is easy to reason about: two groups, one majority, one minority. An asymmetric fault produces monitors that disagree about who is reachable, elections that never settle, and health output that contradicts itself depending on which host you run it from.

How asymmetry arises

MTU mismatch. Small packets pass, large ones are dropped. Since message sizes differ by direction and by message type, the effect looks one-way.
Asymmetric firewall rules. A rule allowing established connections in one direction but not new ones in the other.
One-way routing faults. A missing return route, or asymmetric ECMP hashing across a partially failed path.
A failing NIC or optic that transmits correctly and receives with errors, or the reverse.
Rate limiting or policing applied in one direction only.

What it looks like

from ceph-01:  ceph quorum_status  →  quorum [0,1], leader ceph-01
from ceph-03:  ceph quorum_status  →  hangs
mon.ceph-03 log: "calling new monitor election" every few seconds
mon.ceph-01 log: "ceph-03 out of quorum"
mon.ceph-03 log: "I am in quorum"

Monitors disagreeing about quorum membership is the signature. In a symmetric partition both sides agree about who is on which side; in an asymmetric fault they do not.

Test both directions explicitly — the reflexive test hides the fault

The instinct is to ping from the host you are logged into. That tests one direction of a round trip and passes for most asymmetric faults, because ICMP echo is small and the reply path may be the healthy one.

Test properly:

# from every monitor to every other, both directions, at real MTU
ping -M do -s 8972 -c 5 10.0.1.11        # for MTU 9000
ping -M do -s 1472 -c 5 10.0.1.11        # for MTU 1500

# and the actual ports
nc -zv 10.0.1.11 3300
ss -tn state established '( dport = :3300 or sport = :3300 )'

The do-not-fragment flag at full payload size is the test that finds MTU faults. A default-size ping is exactly the probe that passes when MTU is the problem.

Distinguishing it from resource problems

Observation	Asymmetric network	Slow monitor
Monitor CPU	idle	high
RocksDB commit latency	normal	high
Quorum membership	monitors disagree	monitors agree
Elections	frequent, unresolved	frequent, resolve slowly
Directional ping at MTU	fails one way	passes both ways

Run the CPU and commit-latency check first; it is quick and it eliminates half the possibilities.

Why asymmetric faults break consensus so thoroughly

Paxos assumes that if A can talk to B, B can talk to A. Leases, proposals, and acknowledgements all depend on bidirectional exchange.

Under asymmetry, a monitor can receive a lease grant and fail to have its acknowledgement delivered. The leader concludes the peon is unreachable and excludes it; the peon believes it holds a valid lease and continues serving reads from a map that may be superseded.

Worse, the excluded monitor may call an election, and its election proposal may reach the others even though their responses do not reach it. The result is repeated disruption caused by a monitor that cannot participate but can interfere.

This is why the fastest mitigation, once an asymmetric fault is confirmed and while the network is being fixed, is to stop the affected monitor deliberately — provided the remaining monitors hold quorum. A stopped monitor is a known quantity; a half-connected one is not.

The procedure

Check monitor CPU and RocksDB commit latency on each. If high, this is a resource problem, not a network one.
Run ceph quorum_status from each monitor host and compare. If they disagree, the fault is asymmetric.
Test connectivity between every monitor pair, both directions, with do-not-fragment at full MTU.
Test the actual ports, not only ICMP.
If a monitor is half-connected and the others hold quorum, stop it while the network is repaired.
Fix the network. Restart the stopped monitor. Confirm quorum.

Quiz

Knowledge check · 4 questions

Q1. Monitors report different quorum membership depending on which host you query. What does this indicate?

A monitor store has been corrupted and is reporting stale membership state An asymmetric connectivity fault, since a symmetric partition produces agreement about sides Clock skew large enough that the lease arithmetic differs between the monitors A normal transient state during an election, which will resolve on its own shortly
Q2. A default-size ping succeeding between two monitors is exactly what an MTU fault on that path produces.

True False
Q3. One monitor is half-connected: it receives messages but its replies do not arrive. The other two hold quorum. Decide what to do while the network team investigates.

Three monitors. mon.ceph-03 shows repeated elections in its log and believes it is in quorum; ceph-01 and ceph-03 report it out of quorum. ceph-01 and ceph-02 hold a stable two-of-three quorum. Client I/O is working but shows periodic latency spikes correlating with election events. The network team estimates two hours to trace the fault.
Q4. Explain why Paxos handles asymmetric connectivity poorly, and what that implies operationally.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Check monitor CPU and RocksDB commit latency first — it takes seconds and separates a resource problem from a network one. Compare quorum_status from every monitor host, since disagreement about membership is the signature that distinguishes asymmetric faults from clean partitions. Test both directions at full MTU with fragmentation disabled, because the reflexive default-size ping passes exactly when MTU is the cause. And stop a confirmed half-connected monitor rather than leaving it to propose elections, without removing it from the monmap.

Cross-course references

Ceph: Part V (Distributed Systems Foundations) for partitions generally.
Ceph: Part XXX (Network Failure Behaviour) for the OSD-side equivalent.
Ceph: Part CXVI (Network Partition) for the incident runbook.