Two-node clusters and tie-breaking - the honest version

Two nodes is the cluster size people actually buy, and the one the quorum arithmetic handles worst. linux-quorum-concepts gives it one line in a table - two nodes, quorum 2, survives zero failures. This lesson is what that line means in practice and what the available options really do about it.

Why two nodes is the hard case

Quorum for two nodes is (2/2) + 1 = 2. Both nodes must be present for the cluster to act. So:

• If a node dies, the survivor has one vote out of two. It is inquorate and will not run anything. The redundancy you paid for does nothing.
• If the link between them breaks, each side has one vote out of two. Both are inquorate. The service is down on both.

Neither outcome is acceptable, which is why every two-node cluster runs with an override. The important question is which override, and what it gives away.

two_node: quorum turned off, honestly

two_node: 1 in the quorum section of corosync.conf tells votequorum to treat a single vote as quorum:

quorum {
    provider: corosync_votequorum
    two_node: 1
}

Read that carefully. It does not make one node a majority in any meaningful sense. It says: each node may act alone.

During a partition, both nodes satisfy quorum simultaneously. Both conclude they may run the resources. The quorum layer has been removed from the safety argument entirely, and one thing remains standing between the partition and two writers on the same storage: the fence device.

wait_for_all: the cold-boot protection

two_node creates a second problem. If the whole cluster is powered off and only one node comes back - because the other has a failed PSU, or because the datacentre came up in stages - that single node satisfies quorum immediately and starts everything, with no idea what the other node was doing when the power went out.

wait_for_all: 1 fixes that: the cluster does not become quorate at startup until every node has been seen at least once. votequorum enables it automatically when two_node is set, unless you explicitly turn it off.

Do not turn it off. The cost is that a cold start with one node dead requires a deliberate operator action; the benefit is that the cluster never starts resources without knowing the state of its peer.

quorum {
    provider: corosync_votequorum
    two_node: 1
    wait_for_all: 1
}

That deliberate action, when a peer genuinely will not return, is where pcs quorum unblock belongs - after you have confirmed from the BMC or the console that the peer is powered off. As a first response to an inquorate cluster it is the command that produces the corruption, because “I cannot reach it” is not “it is down”.

The fence race

Give both nodes permission to act alone and they will both act at once. The moment the link breaks:

t+0.0   link fails
t+3.7   node1 declares node2 lost; node2 declares node1 lost
t+3.8   node1 sends a power-off to node2's BMC
t+3.8   node2 sends a power-off to node1's BMC
t+?     whichever BMC completes first wins

Three outcomes, all of them real:

• One winner. The usual case. Whichever fence completes first kills the other node, and the survivor runs the service. Which node survives is decided by BMC response time, which is to say arbitrarily.
• Both die. The two power-offs land close enough together that neither node survives to run anything. The service is down until a human intervenes.
• A fence loop. Both nodes come back, the link is still broken, and they fence each other again. This can continue until someone unplugs something.

The problem is not that fencing failed. It is that the design never named which node should win.

Breaking the race with a delay

The standard remedy is to give one node a head start, so the race has a predetermined winner.

# Static delay on the device that fences node2, so node1 wins
pcs stonith update node2_ipmi pcmk_delay_base=10s

# Or a random delay up to a bound, when you have no preference
pcs stonith update node2_ipmi pcmk_delay_max=15s

pcmk_delay_base makes the delay deterministic: put it on the device that fences the node you would rather lose, and the other node always shoots first. pcmk_delay_max picks a random delay within a bound, which breaks the tie without expressing a preference - useful when the two nodes are genuinely interchangeable.

Pacemaker also offers a cluster property that decides the question by resource weight rather than by hostname, so the node currently running the important resources is the one that wins:

sudo pcs property set priority-fencing-delay=15s

This applies a delay to fencing the node with the higher total resource priority, which means the node with the workload survives. It requires resource priorities to be set to be useful, and it exists only in more recent Pacemaker releases - check what your version supports before relying on it.

Tie-breaking mechanisms compared

two_node plus a delay is the minimum viable design. There are better ones, and votequorum offers two more options that are frequently confused with them.

Two cautions on that table.

auto_tie_breaker and last_man_standing are not two-node features. votequorum treats them as mutually exclusive with two_node, and both solve different problems: ATB breaks even splits in clusters of four or six, and LMS keeps a cluster quorate as it shrinks through clean departures. Neither protects a two-node partition. Check votequorum(5) for the exact constraints in your version rather than assuming a combination works because it was accepted at parse time.

last_man_standing deserves particular care: it lowers the bar for quorum over time. Combined with a partition rather than a clean shutdown, that is the opposite of what you want, which is why votequorum requires wait_for_all alongside it.

The design that actually works

If a third location exists - another rack, another site, a small cloud instance, a management server - use it:

node1                -> 1 vote
node2                -> 1 vote
qdevice (ffsplit)    -> 1 vote
                        3 votes total, quorum 2

Now a partition has a real answer. One node plus the quorum device is a majority; the isolated node is not, and knows it. Quorum is doing work again, fencing enforces it as it was meant to, and the fence race disappears because only one side is ever entitled to act. linux-witness-and-quorum-devices covers the configuration, including why ffsplit rather than lms is the right algorithm when the risk is a link failure.

Verify which regime you are in before trusting either:

# corosync-quorumtool -s

Quorum information
------------------
Nodes:             2
Node ID:           1
Quorate:           Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice

Membership information
----------------------
  Nodeid      Votes    Qdevice Name
       1          1    A,V,NMW node1 (local)
       2          1    A,V,NMW node2
       0          1            Qdevice

Expected votes: 3 with a Qdevice flag is a working three-vote cluster. Expected votes: 2 with Quorum: 1 is a two_node cluster where the fence device is the entire safety argument. The two look similar in pcs status and behave completely differently under a partition.

Knowledge check