Cluster IR: network partition - deciding which side is real

A node-unreachable incident has one broken thing and one point of view. A partition has neither. Every node is running, every node can see some of its peers, and every node has an internally consistent story about what happened — and at most one of those stories is true.

linux-split-brain-explained covers what a partition is and why it corrupts data. This lesson is about the twenty minutes after the pager goes off, when you have to decide which side is real without being able to trust the host you are logged into.

The defining symptom: you cannot see it from one node

Ask a node in the minority partition what happened and it will tell you that two of its peers died. Ask a node in the majority partition and it will tell you that one peer died. Both answers are correct descriptions of what each node observed. Neither answer contains the word “partition”, because no node can observe a partition — it can only observe the peers it lost.

The comparison that names it is the ring ID:

# corosync-quorumtool -s

Quorum information
------------------
Date:             Tue Aug 11 03:12:44 2026
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          1
Ring ID:          1.2f4
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2
Flags:            Quorate

Membership information
----------------------
  Nodeid      Votes Name
       1          1 node1 (local)
       2          1 node2

# corosync-quorumtool -s

Quorum information
------------------
Date:             Tue Aug 11 03:12:47 2026
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          3
Ring ID:          3.2f4
Quorate:          No

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      1
Quorum:           2
Flags:

Membership information
----------------------
  Nodeid      Votes Name
       3          1 node3 (local)

Read the two outputs together:

• Different ring IDs (1.2f4 and 3.2f4) mean the two nodes have formed different memberships. The number before the dot is the node ID that formed the ring, so the two sides have each elected their own. This is the partition, in one field.
• Expected votes: 3 on both sides confirms neither side thinks the cluster shrank. They both still expect three votes; they simply cannot find them.
• Quorate: Yes on one side only is the good case. It means the design is working: exactly one side is allowed to act.

Then find out which link failed:

# corosync-cfgtool -s

Local node ID 3, transport knet
LINK ID 0 udp
addr	= 192.0.2.13
status:
	nodeid:          1:	connected:0
	nodeid:          2:	connected:0
	nodeid:          3:	localhost
LINK ID 1 udp
addr	= 198.51.100.13
status:
	nodeid:          1:	connected:0
	nodeid:          2:	connected:0
	nodeid:          3:	localhost

Two independent links failing to the same two peers at the same second is not two link faults. It is one upstream fault — a switch, a trunk, a VLAN change, a firewall rule pushed to the fleet — that both links happen to traverse. If only LINK ID 1 had dropped, the cluster would have stayed up on link 0 and you would have a degraded-redundancy ticket rather than an incident.

Deciding which side is real

Three questions, in this order. Do not skip to the third.

1. Which side is quorate? If exactly one side reports Quorate: Yes, that side is the cluster. The other side is a set of hosts that used to be in it. This question is usually the whole answer.
2. Is the non-quorate side still running resources? A node that lost quorum stops making cluster decisions, but a resource it already started does not necessarily stop — that depends on no-quorum-policy. Check pcs status on the minority side, and check the service directly, not only the cluster view.
3. Has fencing completed? If the quorate side has fenced the minority nodes, those nodes are already off or about to be. If fencing has failed, the quorate side will not start anything, and both sides are now stuck.

The answer to question 1 comes from the cluster. The answer to question 2 has to come from outside it. The cluster’s own view of “is the service running” travels over the network you are investigating, so ask the thing the service actually serves:

# Substitute your own values before running:
VIP=192.0.2.100

# Who currently answers on the service address? Run this from a
# client network, not from a cluster node.
curl -sS -o /dev/null -w '%{http_code}\n' "http://$VIP/health"

# Which node owns the MAC behind the VIP right now? Two answers
# over successive runs means both sides are claiming it.
ip neigh show "$VIP"

Why restarting Corosync is the wrong first action

It is the first instinct because it looks like the smallest possible action: one service, one node, and the membership will re-form. Every part of that reasoning is wrong during a partition.

• It is not one node. Membership is shared state. Dropping and re-adding this node’s membership forces a ring re-formation that every peer participates in. On the quorate side, a membership change can trigger resource recovery on hosts that were not part of the incident until you typed the command.
• It destroys the evidence. The ring ID sequence and the membership transitions in the journal are the record of what happened and when. A restart resets the ring, and the next investigator sees a clean cluster with no explanation.
• It fixes nothing. The partition is a network fault. Corosync is reporting it accurately. Restarting the process that is telling you the truth does not repair the switch.
• It can get the node fenced. Peers that see a member vanish and reappear may treat the disappearance as a node failure, and a node that is fenced mid-incident takes its resources — and its logs — with it.

The case the design is supposed to prevent

Everything above assumes exactly one side is quorate. Two configurations break that assumption, and both are found during the incident rather than before it:

Check which one you have before you need to know:

# corosync-quorumtool -s | grep -E 'Flags|Expected'

Expected votes:   2
Highest expected: 2
Flags:            2Node Quorate WaitForAll

2Node without WaitForAll in that flag list is the first configuration in the table. Fix it with a quorum device or wait_for_all: 1 in a maintenance window, not during the incident.

Preserving the evidence

Take this before the network is repaired. Once the ring re-forms, the state that explains the partition is gone.

# Substitute your own values before running:
OUT=/var/tmp/partition-$(hostname -s)-$(date -u +%FT%H%M%SZ)
mkdir -p "$OUT"

# Cluster view from THIS node, whichever side it is on
corosync-quorumtool -s      > "$OUT/quorumtool.txt" 2>&1
corosync-cfgtool -s         > "$OUT/cfgtool.txt"    2>&1
pcs status --full           > "$OUT/pcs-status.txt" 2>&1
pcs stonith history show    > "$OUT/fence-history.txt" 2>&1

# Membership transitions, in UTC so the nodes can be compared
journalctl -u corosync -u pacemaker --since '-2 hours' \
  --utc -o short-iso-precise > "$OUT/cluster-journal.txt" 2>&1

# The network side of the story
ip -br addr                 > "$OUT/addr.txt"
ip -s link                  > "$OUT/link-stats.txt"
ip neigh show               > "$OUT/neigh.txt"

Run it on every node, including the ones that look fine, and keep the outputs separate per node. A partition report that contains one node’s view is the same report you could have written without leaving your desk.

Recovery

The partition is repaired by repairing the network. The cluster re-forms a single ring on its own, usually within the token timeout, and Pacemaker recovers from there.

1. Fix the network fault. Confirm from a node on each former side that it can reach the other on every corosync link address.
2. Watch the ring re-form rather than forcing it: journalctl -fu corosync on one node from each side. A single new ring ID appearing on both is the recovery.
3. Confirm one membership: corosync-quorumtool -s on every node must now show the same ring ID and the same member list.
4. Clear any failed actions Pacemaker recorded during the split, with pcs resource cleanup, and only after you have read them.
5. Verify the service, not the cluster. A green pcs status with a VIP nobody can reach is not a recovery.

Knowledge check