Network partition — when both halves are alive and neither is sure

Why this matters in production

Node failure is asymmetric: one side stops, the other keeps going. A partition is symmetric. Both sides are running, both sides are serving requests, and each observes the other as unreachable — which is indistinguishable, from where each stands, from the other having died.

If both sides act on that belief, they diverge, and there is no correct way to merge divergent writes to the same object afterwards.

The shape of the problem

flowchart TD
  subgraph "Side A: 3 monitors"
    A1[mon.a] --- A2[mon.b] --- A3[mon.c]
  end
  subgraph "Side B: 2 monitors"
    B1[mon.d] --- B2[mon.e]
  end
  A2 -.->|"link down"| B1
  A1 -->|"quorum: 3 of 5"| OK[Serves I/O, marks side B out]
  B1 -->|"no quorum: 2 of 5"| ST[Stops serving]

Ceph resolves this with a strict majority. The side with more than half the monitors keeps quorum, updates maps, marks the other side’s OSDs down and then out, and continues. The minority side cannot commit any map change, so its OSDs cannot obtain updates and stop serving — even though their disks are intact and their clients can reach them.

Why the minority stops rather than continuing read-only

It would be tempting to let the minority serve reads. Ceph does not, and the reason is that the minority cannot know whether it is the minority.

From inside the two-monitor side, the observation is “three monitors are unreachable”. That is consistent with a partition and equally consistent with three monitors having failed — in which case serving reads from stale data would be actively wrong, since the majority side may have accepted writes it knows nothing about.

Asymmetric partitions

The harder case is when connectivity is not symmetric: mon.a can reach mon.b, but mon.b cannot reach mon.a. This happens with one-way routing faults, asymmetric firewall rules, and MTU mismatches that let small packets through and drop large ones.

The symptoms are confusing: monitors repeatedly calling elections, quorum forming and dissolving, and ceph status timing out from some hosts but not others. ceph health detail may report monitors as out of quorum that themselves believe they are in it.

The diagnostic is direct connectivity testing from each monitor to every other, in both directions, at the MTU actually in use — not ping with default packet sizes, which is precisely the test that passes when an MTU mismatch is the cause.

What to do when you meet one

1. Do not restart monitors reflexively; that produces more elections.
2. Establish which monitors are actually in quorum from a host that can reach all of them.
3. Test connectivity between every monitor pair in both directions at the real MTU.
4. Fix the network. Restoring connectivity restores quorum automatically, and the minority side rejoins and catches up without intervention.

Quiz

Production discipline

Keep monitor counts odd and place them so that the majority side of any plausible partition is the side you want to survive. When elections start, resist restarting monitors — that generates more elections and destroys the evidence. Check monitor CPU and RocksDB latency first to separate a monitor problem from a network problem, then test every monitor pair in both directions at the real MTU with fragmentation disabled, because the default-size ping is exactly the test an MTU fault passes.

Cross-course references

• Ceph: Part IX (Monitor Quorum) for the quorum model in detail.
• Ceph: Part CXVI (Network Partition) for the incident runbook.
• Ceph: Part CXVII (Lost Monitor Quorum) for recovery when the majority is gone.