Rolling maintenance workflow - the loop that keeps a cluster serving · Linux

Rolling maintenance is how a cluster is patched, upgraded or reconfigured without an outage: one node at a time, with the service carried by the rest. The technique is straightforward. The discipline is not, and almost every rolling-maintenance outage comes from breaking one of two rules - starting without headroom, or moving to the next node before the last one came back.

The go/no-go gate

Run this before the first node is touched. Every item is a stop condition, not a preference.

1. The cluster is healthy right now. Not “healthy this morning”. A rolling maintenance that starts on a cluster with a failed resource removes the redundancy that was masking it.

pcs status --full
pcs resource failcount show
systemctl --failed

2. Quorum survives losing the node you are about to drain, and one more. Draining is planned; a failure during the window is not.

corosync-quorumtool -s
# Expected votes, Total votes, Quorum: the drain must leave
# Total votes strictly above Quorum with a vote to spare.

3. Capacity passes the N-1 test at current load. The surviving nodes must carry the whole load with the drained node out, not just on average but at today’s peak.

L <= (N-1) x C     where C is MEASURED per-node capacity

If that fails, the maintenance is an outage with extra steps. Reschedule to a lower-load window or add a node first.

4. Replication and storage are in sync. A DRBD device still resyncing, a database replica lagging, a Ceph cluster backfilling - each means the redundancy you are about to spend is already partly spent.

cat /proc/drbd 2>/dev/null
pcs status | grep -i sync

5. No other change is in flight. One change at a time per cluster. Two concurrent changes turn “which change broke it” into a research project during an incident.

6. Rollback is available for the specific change. Not a general belief in rollback - the actual command, tested, and the previous artefact still present on the node.

The loop

For each node, in order, one at a time:

Drain

Move work off the node before changing anything on it.

# Pacemaker: stop hosting resources, stay in the cluster
sudo pcs node standby node1

# Confirm the node hosts nothing
pcs status | grep -A2 node1

# Load balancer: stop new connections and let existing ones finish
# (weight 0 / drain, then wait for the connection count to fall)

Draining is not stopping. The node stays a cluster member and keeps voting, so quorum is unaffected; it simply stops carrying work. Stopping the cluster stack removes the vote too, which is a much larger change to the cluster’s tolerance for a second failure.

Change

Apply exactly one change. Patch, upgrade, reconfigure - whatever the plan says, and nothing that was noticed on the way past.

Validate

Validate on the node before it carries traffic again.

systemctl --failed
journalctl -p err -b --since '10 min ago'
uname -r                    # for a kernel change, is the intended kernel running?
ss -tlnp                    # are the expected sockets bound?
curl -sf http://127.0.0.1:8080/healthz

A health check that only tests the port is not validation. Test something that exercises the change.

Return

Put the node back and prove it is carrying work.

sudo pcs node unstandby node1
pcs status | grep -A3 node1     # resources are running here again

Then wait. Watch error rates and latency for a defined observation period - long enough for the node to serve real traffic across a full request mix. Ten minutes is a common floor; a node that only fails under a nightly batch needs longer.

Then the next node

Only after the previous node is fully back. Repeat until every node is done.

When a node does not come back

Stop the maintenance. Do not proceed to the next node, and do not leave the cluster in the half-changed state while you investigate at leisure.

The decision is between two paths and it should take minutes, not an hour:

Roll back that node to the previous state and return it to service. The cluster ends the window mixed-version but at full capacity, which is a stable place to stand.
Leave it drained and abort, if rollback is not possible. The cluster is at N-1 and every subsequent event is more dangerous, so this is a page, not a to-do item.

A mixed-version cluster is usually acceptable for hours or days - most cluster stacks support it explicitly during upgrades. A cluster stuck at N-1 with nobody watching is not.

Worked example: patching a 4-node web tier

Gate
  pcs status                clean
  corosync-quorumtool -s    4 votes, quorum 3   -> drain leaves 3, one to spare  OK
  peak L = 1800 RPS, C = 700 RPS
  N-1 test: 1800 <= (4-1) x 700 = 2100          OK
  DRBD in sync, no other change in flight       OK
  rollback: previous kernel still installed     OK
  -> GO

Loop
  node1  drain -> patch -> reboot -> validate -> unstandby -> observe 10 min  DONE
  node2  drain -> patch -> reboot -> validate -> FAILED health check
         roll back to previous kernel, unstandby, observe                    RECOVERED
         ABORT the maintenance; node3 and node4 untouched
  Result: cluster at full capacity, 1 node patched, 3 not.
          Investigate the failure outside the window.

The maintenance stopped after two nodes and that is a successful outcome. The cluster is at N, the failure is reproducible, and nobody is debugging a kernel panic on three nodes at once at 02:00.