Membership and partitions - the failure mode that breaks quorum

Membership changes when nodes join or leave the cluster. Partitions split the cluster into two groups. This lesson covers what happens and how to recover.

Membership changes

A node joins the cluster:

• It starts the cluster software (Corosync, etc.).
• It contacts the existing members.
• The members vote to include the new node.
• If accepted, the new node is in the membership.

A node leaves the cluster:

• Graceful: it sends a leave message.
• Ungraceful: it stops responding (crash, network loss).

The cluster adjusts membership accordingly.

What is a partition

A network partition is when the cluster is split into two groups that cannot communicate:

Before partition:
  +-----+
  |  A  +----+
  |     |    |
  +-----+    +------+
            |  C  |
            |     |
  +-----+    +------+
  |     |    |
  |  B  +----+
  |     |
  +-----+

After partition (A-B can talk, C cannot reach either):
  +-----+
  |  A  +----+
  |     |    |
  +-----+    +      +------+
            |  ?   |  C   |  (orphaned)
  +-----+    +      +------+
  |     |    |
  |  B  +----+
  |     |
  +-----+

A and B form one group, C is isolated. Without quorum, A-B might promote themselves and C might also promote itself. With quorum, only the majority group (A-B) makes decisions; C is fenced.

Symptoms of a partition

• Cluster status shows nodes as offline.
• Resources do not migrate to the surviving nodes.
• Service is interrupted.

Corosync logs show membership changes and quorum loss.

Isolated node vs split brain

• Isolated node: one node cannot reach the cluster. The cluster continues; the isolated node is fenced.
• Split brain: two groups each think they are the cluster. Without quorum, both may promote themselves. With quorum, only the majority continues.

Quorum prevents split brain but does not prevent the isolated node from being a single point of failure (it is, but the workload is on the majority group).

Recovery from a partition

When the network heals:

• The minority group rejoins.
• Membership is updated.
• Resources may be re-balanced (the new node takes workload).
• Quorum is restored.

If the minority group has been fenced (powered off), the node must be manually restarted.

For automatic recovery, configure the cluster to auto-rejoin. For manual recovery, an operator intervenes.

Avoiding partitions

• Redundant network: dual switches, dual NICs, multi-path.
• Geographic distribution: sites with redundant WAN.
• Latency monitoring: alert if inter-node latency exceeds threshold.
• Test partitions: simulate a partition and verify the cluster survives.

Knowledge check