Lab: Rolling maintenance on a 3-node cluster

This lab performs a rolling maintenance change on a 3-node cluster. The change is a simulated package update; the discipline (gate, drain, update, validate, return) applies to any change.

The word that carries the weight is return. One node is out of service at any moment, and it goes back in before the next one comes out. Placeholders below — <vip>, <package>, <new-version>, <old-version> — are yours to substitute.

Draining a node before returning the previous one is not rolling maintenance

A Pacemaker node in standby runs no resources. Drain node1, then node2 without returning node1, and two-thirds of the cluster is carrying nothing. Drain node3 as well and every resource in the cluster is stopped — the VIP is down, the service is down, and any health check against <vip> fails.

That is a staged full outage, not a rolling change. It is an easy shape to fall into because each individual pcs node standby looks harmless, and the cluster does not stop you.

Even the intermediate state is wrong. With 2 of 3 nodes drained, all load sits on one node with no failover target: a hardware fault on the survivor during your change window is an unrecoverable outage. Rolling maintenance holds N+1 at every step, which on a 3-node cluster means one node out, never two.

Tasks

Task 1: Go/no-go gate

A baseline you never assert against is decoration. Capture the baseline and make it a gate: if any check below fails, the change does not start.

# --- GO/NO-GO GATE - abort the change if any of these fail ---

# 1. No failed actions anywhere in the cluster. Must print nothing.
sudo pcs status --full | grep -A5 -i 'Failed'

# 2. Quorum is held. Must report "Quorate: Yes".
sudo pcs quorum status

# 3. Every node Online, none already in standby or maintenance.
sudo pcs status nodes

# 4. No resource is stopped or unmanaged.
sudo pcs status resources

Capture the baseline the later validations compare against:

sudo pcs status --full > /tmp/baseline-status.txt
curl -sf http://<vip>/health > /tmp/baseline-health.txt
time curl -sf http://<vip>/ > /tmp/baseline-time.txt

The capacity check nobody runs

Draining one node moves its load onto the remaining N-1. If the survivors cannot carry 100% of the load, the change causes the outage it was meant to avoid.

For an N-node cluster, per-node steady-state utilisation must sit below (N-1)/N before you start. On 3 nodes that is 67%; on 2 nodes it is 50%. Measure the real number — CPU, memory, connection count, whichever is the binding constraint — and record it in the change record:

# Example: peak CPU per node over the last week, from your metrics store
# node1 41%  node2 44%  node3 39%   -> max 44% < 67%  -> GO

At 70% per node on a 3-node cluster, draining one node puts the other two at 105%. That is a NO-GO. Add capacity first, or shrink the change to a window with lower load.

Record the quorum arithmetic too. Three nodes means expected_votes = 3 and a quorum threshold of 2. Standby does not remove a node’s vote — a node in standby still votes, so draining does not cost you quorum. Stopping the cluster stack on a node does. That distinction is why pcs node standby is the drain primitive and pcs cluster stop is not.

Task 2: Prepare the change (simulated)

Pick a package to update and confirm the target version exists before you touch any node:

# Substitute your own values before running:
PACKAGE=nginx
NEW_VERSION=1.26.0-1

apt-cache policy "$PACKAGE"
sudo apt install -y --simulate "$PACKAGE"="$NEW_VERSION"

Note the current version. You need it for rollback:

# Record this - it is the version you roll back to
OLD_VERSION=$(dpkg-query -W -f='${Version}' "$PACKAGE")
echo "current: $OLD_VERSION"

Task 3: The per-node loop

This is the whole procedure. Run it for node1 first as the canary, then — only after node1 is back and healthy — for node2, then node3.

Drain one node. sudo pcs node standby node1
Confirm it is empty. sudo pcs status must show zero resources on node1, and every resource that moved must be Started elsewhere - not Stopped, not FAILED
Confirm the cluster is still quorate and the service is still up. sudo pcs quorum status and curl -sf http://<vip>/health. The VIP is being served by the other two nodes; if it is down now, stop and investigate before changing anything
Apply the change on the drained node. sudo apt install -y <package>=<new-version>
Validate the node locally, while it is still out of service. Smoke-test the service on the node itself, and check its logs
Return the node. sudo pcs node unstandby node1 - this is the step that must never be deferred to a later task
Confirm it came back. sudo pcs status --full | grep -A5 -i Failed must still print nothing, and the node must show Online with resources able to run on it
Assert end-to-end health now that the node is back. curl -sf http://<vip>/health and a latency comparison against the baseline
Wait for any replicated storage to catch up before moving on - for example DRBD must read UpToDate/UpToDate, not UpToDate/Inconsistent
Only now move to the next node. If any step above failed, roll back this node and stop the change

Written out for the canary:

# Substitute your own values before running:
VIP=192.0.2.10
PACKAGE=nginx
NEW_VERSION=1.26.0-1

# --- node1 ---
sudo pcs node standby node1
sudo pcs status                              # 0 resources on node1
sudo pcs quorum status                       # Quorate: Yes
curl -sf "http://$VIP/health"                 # service still up on node2/node3

sudo apt install -y "$PACKAGE"="$NEW_VERSION"

# Local validation while node1 is still drained
curl -sf http://node1/"$PACKAGE"-health
sudo journalctl -u "$PACKAGE" --since "5 minutes ago" | grep -i error

sudo pcs node unstandby node1                # RETURN IT BEFORE TOUCHING node2

sudo pcs status --full | grep -A5 -i 'Failed'   # must be empty
curl -sf "http://$VIP/health"
time curl -sf "http://$VIP/"                     # compare with /tmp/baseline-time.txt

Read-only / Safeone node out, two in— This is the only shape the cluster should ever take during the loop. If a second name ever appears on the Standby line, the change has become an outage in progress - unstandby one of them immediately.

$ sudo pcs status nodes

Pacemaker Nodes:
Online: node2 node3
Standby: node1
Maintenance:
Offline:

Illustrative output

Task 4: Decide, per node

After each node returns, you have a decision, and it is a decision about the whole change, not just that node:

All checks pass → proceed to the next node.
Any check fails → roll this node back and stop. Do not drain the next node to “see if it also fails”.

# Substitute your own values before running:
PACKAGE=nginx
OLD_VERSION=1.24.0-1
VIP=192.0.2.10

# Roll back the node you are working on
sudo apt install -y --allow-downgrades "$PACKAGE"="$OLD_VERSION"
sudo pcs node unstandby node1
sudo pcs status --full | grep -A5 -i 'Failed'
curl -sf "http://$VIP/health"

Nodes you already updated are still on the new version. Decide explicitly whether the cluster is safe to leave in a mixed state overnight, or whether you roll those back too. Record the decision.

Task 5: Repeat for node2, then node3

Run Task 3 again with node2, and again with node3. Before each iteration, re-run the gate from Task 1 — a node that failed since you started changes the answer.

Task 6: Final assertion

After the last node returns, assert on the cluster as a whole rather than declaring success:

# Substitute your own values before running:
VIP=192.0.2.10
PACKAGE=nginx

sudo pcs status --full                       # all nodes Online, no Failed actions
sudo pcs quorum status                       # Quorate: Yes, expected votes back to 3
sudo pcs status resources                    # resources distributed, none Stopped

# The Standby line must be empty. Fail loudly if it is not.
sudo pcs status nodes | grep -q '^ Standby: *$' \
  || echo 'FAIL: a node is still in standby'

curl -sf "http://$VIP/health"
dpkg-query -W -f='${Version}\n' "$PACKAGE"    # run on each node; all three match

Do not record a PASS you did not observe

The synthetic check is only meaningful while the VIP is actually being served. Recording “Synthetic: PASS throughout” because the procedure said it should pass — rather than because you watched it pass after each node returned — is how a change record certifies an outage as a success.

Run the synthetic check continuously if you can:

while true; do
  printf '%s %s\n' "$(date -Is)" "$(curl -so /dev/null -w '%{http_code}' http://<vip>/health)"
  sleep 1
done | tee /tmp/synthetic.log

Then the change record quotes the log, not the intention. A handful of failures during a resource move is normal and worth recording; a continuous block of failures is the outage you must not paper over.

Task 7: Document

CHANGE RECORD
Date: 2026-08-09 14:00
Change: <package> <old-version> -> <new-version>
Reason: Security update / simulated
Nodes: 3-node cluster, ONE node drained at a time, returned before the next

Go/no-go gate (14:02):
- Failed actions: none
- Quorum: Quorate Yes, expected votes 3, threshold 2
- Nodes: node1 node2 node3 all Online, none in standby
- Capacity: peak per-node CPU 44% < 67% ((N-1)/N for N=3)  -> GO

Per-node results (drained -> updated -> RETURNED -> asserted):
- node1  standby 14:05, unstandby 14:11  smoke PASS, latency 45ms, errors 0
- node2  standby 14:14, unstandby 14:20  smoke PASS, latency 47ms, errors 0
- node3  standby 14:23, unstandby 14:29  smoke PASS, latency 46ms, errors 0
- Maximum nodes in standby at any moment: 1

Synthetic check (observed, from /tmp/synthetic.log):
- 1441 samples, 1437x 200, 4x 000 during resource moves at 14:05 and 14:14
- Longest unavailable window: 3s

Final assertion (14:31):
- pcs status --full: all Online, no Failed actions
- Quorum: Quorate Yes, expected votes 3
- Package version identical on all three nodes

Total time: 29 minutes
Rollback tested: yes, in staging
Rollback decision point: any failed check on a node stops the change

Notes:
- All nodes healthy
- Service available throughout the change (4s of errors, all during moves)
- No issues observed

Objectives

Prerequisites