Pacemaker 2-node cluster - the quorum challenge · Lab · Linux

A 2-node cluster has a quorum challenge: quorum is 2, which means a single failure breaks the cluster. This lab adds a witness to make a 2-node cluster viable.

Tasks

Task 1: The 2-node problem

For a 2-node cluster, quorum is 2. If one node fails, the remaining 1 node does not have quorum. The cluster stops serving traffic.

Solutions:

Add a third node (or VM).
Add a witness (corosync-qnetd).
Use two_node: 1 in corosync.conf. This is the standard, supported votequorum option, and pcs cluster setup writes it automatically for a two-node cluster. It is safe only with working fencing, for the reason in Task 1b.

The witness is the solution that gives you a real third vote. two_node: 1 does not; it changes the arithmetic instead. You need to be able to read both.

Task 1b: Read the votequorum block

Quorum arithmetic is implemented by the votequorum service inside corosync, and it is configured in the quorum {} block of /etc/corosync/corosync.conf. You will inherit clusters with this block already set, so you must be able to read it.

quorum {
    provider: corosync_votequorum
    two_node: 1              # implies wait_for_all: 1
    # wait_for_all: 1        # do not form quorum until all
                             # nodes have been seen at least once
    # last_man_standing: 1   # shrink expected_votes as nodes
                             # leave cleanly
    # last_man_standing_window: 10000
    # auto_tie_breaker: 1    # on an even split, the partition
                             # holding the lowest node ID wins
}

Read the options like this:

expected_votes: the total the cluster expects. Quorum is floor(expected_votes / 2) + 1. Corosync derives it from the nodelist; you rarely set it by hand.
two_node: 1: forces quorum to 1 for a two-node cluster, so a surviving single node keeps running. It is what makes a two-node cluster useful at all.
wait_for_all: 1: the cluster refuses to become quorate until it has seen every node at least once since starting. two_node: 1 turns this on for you.
last_man_standing: 1: recalculates expected_votes downwards as nodes leave cleanly, so a cluster can shrink from 5 to 1 over time. Only meaningful with working fencing.
auto_tie_breaker: 1: on an exact even split, the partition containing the lowest node ID stays quorate. It is an alternative to a witness for even-numbered clusters, and it is weaker than one.

You also need no-quorum-policy, which is a Pacemaker property rather than a corosync one. It decides what happens when quorum is lost:

sudo pcs property set no-quorum-policy=stop      # default: stop resources
sudo pcs property set no-quorum-policy=freeze    # keep running, do not recover
sudo pcs property set no-quorum-policy=demote    # demote promotable resources

You will meet no-quorum-policy=ignore in inherited clusters. It tells Pacemaker to carry on regardless of quorum. On a two-node cluster with proven fencing it is an accepted configuration. On any larger cluster, or on a two-node cluster with stonith-enabled=false, it is a split-brain generator: every partition keeps running every resource. Treat finding it as a finding, and check the fencing configuration before you change anything.

Task 2: Install the witness

On a third host (cloud VM, separate network):

sudo apt install corosync-qnetd
sudo pcs qdevice setup model net --enable --start
sudo pcs qdevice status net

pcs qdevice setup builds the qnetd service and its certificate database on this host. It takes model net and --enable --start, and nothing else. It has no --cluster_name and no --qdevice_host option, because one qnetd host can serve many clusters and at this point it knows about none of them.

The witness is now running and listening for corosync clients.

Task 3: Configure the cluster with the witness

On one of the cluster nodes:

sudo pcs host auth node1 node2
sudo pcs cluster setup mycluster node1 node2 \
    --start --enable

# Attach this cluster to the device that already exists on the
# witness host. Authenticate to that host first.
sudo pcs host auth witness.example.com
sudo pcs quorum device add model net \
    host=witness.example.com algorithm=ffsplit

The two commands are easy to confuse because both contain the word qdevice. pcs qdevice setup ran on the witness host in Task 2 and created the device. pcs quorum device add runs on a cluster node and is what registers this cluster against it. Neither one does the other’s job.

algorithm=ffsplit gives the device exactly one vote, cast for the partition with more than half the nodes, or on an exact tie for the partition holding the lowest node ID. That is the right choice for a two-node cluster.

The cluster now has 2 nodes and 1 qdevice vote: 3 total votes, quorum 2. A single node failure leaves 1 node plus the witness = 2 votes, which is still quorate.

The witness is not a third node. It holds no CIB, hosts no resources, is never fenced, and never appears in the node list. It shows up as a separate Qdevice line and adds to the vote total only. Count votes, not nodes.

Check what pcs decided about votequorum for you:

sudo pcs quorum status
# Expect a Qdevice line, Quorate: Yes,
# Total votes 3, Quorum 2

sudo corosync-quorumtool -s
sudo grep -A8 '^quorum' /etc/corosync/corosync.conf

Note that two_node: 1 is gone from that block. pcs removes it when a quorum device is added, because the device now supplies the third vote that two_node was faking.

Task 3b: Configure fencing (MANDATORY - do not skip)

A two-node cluster without fencing is not a cluster. It is two hosts that will both start the same resource the first time the interconnect drops. Configure STONITH now, before any failover test, because the failover test is meaningless without it.

sudo pcs stonith create node1_fence fence_ipmilan \
    pcmk_host_list=node1 \
    ip=10.0.99.11 username=stonith \
    password_script=/etc/pacemaker/bmc-pw.sh \
    lanplus=1 pcmk_delay_max=15s

sudo pcs stonith create node2_fence fence_ipmilan \
    pcmk_host_list=node2 \
    ip=10.0.99.12 username=stonith \
    password_script=/etc/pacemaker/bmc-pw.sh \
    lanplus=1 pcmk_delay_base=5s

sudo pcs property set stonith-enabled=true

Two details in those commands carry the whole design.

password_script rather than password. A password passed as an argument is visible in the CIB and in the process table of every node. A script that prints the password to stdout keeps it in one root-owned file.

The different delays. pcmk_delay_max=15s on one device and pcmk_delay_base=5s on the other break the fence race. During a partition both nodes are quorate and both issue a fence at the same instant. With identical timing they can shoot each other simultaneously and the cluster ends with zero nodes running - a worse outcome than the split brain you were preventing. Asymmetric delays make one node reliably win.

Now test it. Fencing that has never been fired is a configuration, not a capability:

sudo pcs stonith status
sudo stonith_admin --list-registered

# Fire it for real and watch node2 power off
sudo pcs stonith fence node2
sudo pcs status                       # node2 OFFLINE, resources moved
sudo stonith_admin --history node2    # the action must show "OK"

Then bring node2 back and confirm it rejoins:

sudo pcs cluster start node2
sudo pcs status

Task 4: Verify

sudo pcs status
sudo pcs quorum status

Verify:

2 nodes online in pcs status. The witness is not one of them, and waiting for it to appear there is an acceptance criterion that can never pass.
A Qdevice line in pcs quorum status, with total votes 3 and quorum 2.
Resources running.
stonith-enabled=true in pcs property list.
Both fence devices Started, on the node that is not their target where possible.
A successful entry in stonith_admin --history for each node, proving the path works rather than merely existing.

Task 5: Test failover

# Stop one node
sudo pcs cluster stop node1

# Verify
sudo pcs status

Resources should be on node2 (with witness support).

Task 6: Test witness failure

# Stop the witness, on the witness host
sudo systemctl stop corosync-qnetd

# Verify, from a cluster node
sudo pcs quorum status

Without the witness the vote total drops from 3 to 2 and quorum stays at 2. Both nodes are up, so the cluster holds 2 of 2 votes and continues.

If you also stop node1, 1 vote remains against a quorum of 2. The cluster loses quorum and stops. Losing the witness does not break the cluster; it removes the margin that let it survive a node failure.

Task 7: Document

2-NODE CLUSTER WITH WITNESS
=========================
Cluster: mycluster
Nodes: node1, node2
Witness (qdevice, not a node): witness.example.com
  model net, algorithm ffsplit, 1 vote

Votes: 3 total (2 nodes + 1 qdevice), quorum 2

Fencing: fence_ipmilan, one device per node
  node1_fence -> BMC 10.0.99.11, pcmk_delay_max=15s
  node2_fence -> BMC 10.0.99.12, pcmk_delay_base=5s
  stonith-enabled=true
  BMC network: separate switch from the interconnect

Failure modes:
- One node failure: tolerated (witness breaks tie)
- Witness failure: tolerated (2 votes remain, quorum 2)
- Interconnect partition: survivor fences the peer, delays
  decide the winner
- Interconnect AND BMC path lost together: NOT tolerated,
  resources may run twice - this is the design risk
- Both nodes failure: NOT tolerated
- Both nodes and witness failure: NOT tolerated

Tests:
- Single node failure: PASS
- Witness failure: PASS
- Fence node1 from node2: PASS (powered off, verified at BMC)
- Fence node2 from node1: PASS (powered off, verified at BMC)
- Fence with interconnect down: PASS
- Both nodes failure: cluster stops (expected)

Pacemaker 2-node cluster - the quorum challenge

Objectives

Prerequisites