Lab: Quorum math - calculate, simulate, design

This lab calculates quorum for various cluster sizes, simulates partitions, and designs a cluster with witness.

Tasks

Task 1: Calculate quorum

For each cluster size, calculate quorum and surviving failures:

Answer:

Task 2: Simulate a partition

For a 5-node cluster (A, B, C, D, E):

• A, B, C can talk to each other.
• D, E can talk to each other.
• A-B-C cannot reach D-E.
• D-E cannot reach A-B-C.

Quorum is 3. A-B-C has 3 nodes (quorum). D-E has 2 nodes (no quorum).

Answer:

• A-B-C continues to serve.
• D-E is fenced.
• When the partition heals, D-E rejoins.

Task 3: Design with witness

For a 2-site cluster (Site A 3 nodes, Site B 3 nodes):

• A-B-C are at Site A.
• D-E-F are at Site B.
• A partition between A and B: A has 3 (quorum for 7 nodes is 4 - so 3 is minority).
• D-E-F: same, 3 is minority.

Solution: add a witness.

Site A: A, B, C
Site B: D, E, F
Witness: W (in third location)
Total: 7 nodes, quorum 4

Partition: A-B-C + W (4 nodes) has quorum.
D-E-F: 3 nodes, no quorum. They must stop — but see below
for HOW they stop, because it is not by being fenced from Site A.

The witness fixes the quorum arithmetic. It does not fix fencing, and quorum without a working fence is not a safe cluster.

Task 3b: how does the minority site actually stop?

Write the fence path out before you accept the design:

Inter-site link DOWN
  Site A (has quorum) wants to fence D, E, F
  Fence agent: IPMI to the BMCs of D, E, F
  Those BMCs are at Site B
  The route to Site B is the link that is down
  => the fence request times out

The fence can never succeed, because the thing being fenced is unreachable by construction — the failure that triggered the fence is the same failure that removes the fence path. With stonith-enabled=true, Pacemaker at Site A will not promote or start resources it cannot prove are stopped elsewhere, so the service stays down at both sites even though one has quorum.

The predictable next move under pressure is to disable STONITH to “get the service up”. That works, and it is how a split-brain becomes data corruption: the nodes at Site B are still running and still writing.

Two designs actually solve this. Choose one deliberately.

Option A — SBD watchdog self-fencing (one stretch cluster). The minority site fences itself. Each node runs a hardware watchdog; when a node loses quorum (or loses its SBD device) it stops petting the watchdog and the watchdog resets it after a fixed timeout. No reachability required.

pcs stonith sbd enable --watchdog=/dev/watchdog
pcs property set stonith-watchdog-timeout=10s

The survivor does not need to contact anything: it waits stonith-watchdog-timeout, after which the minority nodes are guaranteed dead. That guarantee is only as good as the watchdog, so use a real hardware or hypervisor watchdog, never softdog, in production.

Option B — booth tickets (two independent clusters). Each site is its own Pacemaker cluster. A ticket, granted by a booth arbitrator in a third location, decides which site may run the service. Resources are constrained on the ticket, so a site that does not hold it cannot start them.

# /etc/booth/booth.conf — identical on both sites and the arbitrator
transport = UDP
port = 9929
arbitrator = "203.0.113.10"
site = "10.1.0.10"
site = "10.2.0.10"
ticket = "svc-ticket"
  expire = 600
  timeout = 10

pcs constraint ticket add svc-ticket svc-group loss-policy=fence

Losing the ticket triggers the local loss policy, which the local cluster can always enforce — local fencing works because the fence target is on the same side of the failed link.

The rule that generalises: a fencing method is only valid if it works in the failure it is meant to handle. IPMI is excellent for a single dead node in a single site and useless across a partitioned link.

Task 4: Document the design

CLUSTER DESIGN
=============
Sites: 2 (A and B)
Nodes: 3 per site (6 total)
Witness: 1 in third location
Total: 7 nodes
Quorum: 4

Failure modes:
- Single node failure: tolerated (quorum preserved)
- Two node failures: tolerated
- Site A failure: tolerated (Site B + Witness = 4)
- Site B failure: tolerated (Site A + Witness = 4)
- Witness failure: tolerated (Site A + B = 6, quorum 4)
- A-B partition + Witness failure: NOT tolerated

Monitoring:
- Each node\'s health
- Quorum state
- Witness reachability
- Network between sites

Fencing:
- Intra-site, single node failure: IPMI to that node's BMC
- Cross-site partition: IPMI CANNOT be used - the BMCs are
  behind the failed link. SBD watchdog self-fencing, with
  stonith-watchdog-timeout 10s, on a hardware watchdog.
- Fence-path statement: for EACH failure mode, which method is
  used and why it is reachable during that failure
- Tested quarterly, including one test with the inter-site
  link deliberately down

Validation

This is a design lab, so the deliverable is a document rather than a running system — but it is still checkable. Each item below either passes or fails; “it looks reasonable” is not a result.

1. The quorum table is arithmetically right. Check it against the formula rather than against your memory:

   for n in 2 3 4 5 6 7 9; do
     printf 'N=%d quorum=%d survives=%d\n' \
       "$n" "$(( n/2 + 1 ))" "$(( n - (n/2 + 1) ))"
   done

   Your table must match line for line. If N=4 does not show the same survivable-failure count as N=3, the table is wrong.

2. Every partition scenario in Task 2 names exactly one surviving side, or states explicitly that neither survives. A scenario with two surviving sides is a split brain you have designed in. Re-check the vote count on both sides.

3. The witness changes at least one outcome. Remove it from the design on paper and re-run the scenarios. If every result is identical with and without it, the witness is not placed in a third failure domain and is not doing its job.

4. The fence-path statement has one row per failure mode, and no row’s fence path crosses the component that failed. Walk the table and, for each row, answer out loud: “during this failure, is the fence device reachable?” A row where the answer is no — IPMI over the link that just dropped — is a finding, not a design.

5. Task 3b answers the question with a mechanism, not an intention. “The minority site stops” is not an answer. “The minority site self-fences via hardware watchdog after stonith-watchdog-timeout=10s” is.

The design is validated when all five hold. If item 4 fails, go back to Task 3b before writing the document — everything downstream of an unreachable fence path is fiction.