Break/Fix: the canary node patched perfectly and then the cluster lost quorum · Break/Fix · Ansible

Reported symptoms

The patch window opens at 02:00. The seven-node consensus cluster that holds service discovery and configuration for the whole estate is scheduled first.

At 02:04 the first node has been patched, rebooted and rejoined. The cluster reports all seven members healthy. The engineer watching the run notes that the canary went perfectly.

At 02:06 every dependent service in the estate starts failing. The cluster is not accepting writes. It stays that way for eleven minutes.

The play was not modified. It is the same play that patches the web tier, reused because it already had batching, health checks and a reboot loop - all the things a patch play needs.

A pre-flight check confirmed the correct seven hosts were targeted. It was correct.

Evidence provided

Read-only / Safeseven hosts, and not one word about how they will be grouped

$ ansible-playbook -i inventory patch.yml --limit cluster --list-hosts

playbook: patch.yml

play #1 (cluster): Patch the cluster	TAGS: []
  pattern: ['cluster']
  hosts (7):
    etcd05
    etcd02
    etcd06
    etcd07
    etcd04
    etcd01
    etcd03

Read-only / Safeone, then five, then one

$ grep -E '^PLAY \[|^ok: |^changed: ' logs/patch-0200.log | head -14

PLAY [Patch the cluster] *******************************************************
changed: [etcd01]
PLAY [Patch the cluster] *******************************************************
changed: [etcd02]
changed: [etcd03]
changed: [etcd04]
changed: [etcd05]
changed: [etcd06]
PLAY [Patch the cluster] *******************************************************
changed: [etcd07]

Read-only / Safeserial as a list is a ramp, not a batch size

$ grep -n -A6 'hosts: cluster' patch.yml

3:  hosts: cluster
4:  serial:
5:    - 1
6:    - 5
7:    - 100%
8:  gather_facts: true

Read-only / Safethe commit message says exactly what everybody believed

$ git log --oneline -1 --format='%h %s' -- patch.yml; git log -1 --format='%b' -- patch.yml | head -2

7d31a90 patch: reuse the web tier patch play for the cluster group
Same play, just a different --limit. Nothing to change.

Read-only / Safethe cluster describes the fault precisely

$ ssh etcd01 'journalctl -u etcd --since 02:05 --until 02:07 | grep -i quorum'

Aug 11 02:06:03 etcd01 etcd[912]: lost quorum: 2 of 7 members reachable, 4 required
Aug 11 02:06:03 etcd01 etcd[912]: refusing writes

Read-only / Safethe play has no idea it is patching a cluster

$ grep -c 'cluster health\|member list\|quorum' patch.yml

Work the evidence before reading on

The play targeted the right seven hosts. The canary succeeded. Nothing was modified.

Read the serial value as data rather than as a setting. It is a list of three numbers. What does each number correspond to in the run output?
The cluster needs four of seven members. How many were unavailable during the second stage?
--list-hosts was run as the pre-flight and reported seven hosts. Was it wrong?

Before continuing: what would you have had to run, before the play, to see that the second stage takes five nodes at once?

Root cause

1. `serial` as a list means a ramp

serial takes either a single value, which is the size of every batch, or a list, which is the size of each batch in turn. A list is a ramp: start small, confirm nothing broke, then accelerate.

serial:
  - 1
  - 5
  - 100%

Seven hosts under that policy split into batches of one, five and one - which is exactly the shape the run output shows, once you know to read the repeated PLAY header as a batch boundary.

On the web tier this is a good policy. One host is patched, the load balancer absorbs its absence, the operator confirms nothing is broken, and the remainder proceed quickly. Losing five of seven web servers is a capacity problem that a load balancer manages.

2. A consensus cluster fails at a threshold, not gradually

Seven members require four to be present for quorum. Removing one member is unremarkable; removing five is total failure. There is no degraded middle state - the cluster accepts writes or it does not.

That is the entire difference between the two tiers, and it is not expressed anywhere in the play. serial: [1, 5, 100%] is not wrong in general. It is catastrophic for this specific group and there is nothing in the file that says so.

3. The successful canary supplied confidence about the wrong thing

The first node patched, rebooted and rejoined perfectly. That is genuine evidence: the package works, the reboot works, the rejoin works, the play is correct.

It is no evidence whatever about what happens when five nodes do the same thing simultaneously. And it arrived thirty seconds before the stage that caused the outage, at the moment when everybody watching was most reassured.

A ramp is designed to convert confidence into speed. When the risk being managed is per-host, that is sound. When the risk is about how many hosts are affected at once, the canary proves nothing about the stage that follows it, and the ramp accelerates into the danger rather than away from it.

Under the hood

Batching is easy to observe and worth observing once, because nothing in the pre-flight tooling will show it to you.

A play with serial: [1, 5, 100%] over seven hosts produces the PLAY header three times, with one, five and one host respectively. That repeated header is the only visible marker of a batch boundary, and it appears in the run output, not before it.

--list-hosts reports the play pattern and the resolved hosts. It does not evaluate serial, so it prints hosts (7) under both a safe policy and a catastrophic one. The pre-flight check everybody runs is structurally incapable of catching this.

Two further consequences of batching are worth knowing while reading a play that has any:

run_once executes once per batch, not once per play. Under this ramp, a task marked run_once runs three times.
max_fail_percentage is evaluated against the current batch. Under this ramp the same percentage means something different in each stage: in a batch of one, any failure is 100 percent; in a batch of five, one failure is 20 percent.

Both mean that a value which is correct for one batch size is silently wrong for another, and a ramp changes the batch size mid-play.

Batching policy is a property of the tier, not of the playbook

The commit message said “same play, just a different --limit. Nothing to change.” That is true of the tasks and false of the play.

Tasks describe what to do to a host. serial, max_fail_percentage and any_errors_fatal describe how many hosts may be affected at once and what to do when one fails - and those answers come from the failure semantics of the tier, not from the work being performed.

Two questions separate them:

Does this tier degrade gradually or fail at a threshold? A load-balanced web tier degrades; a consensus cluster, a replicated database with a write quorum, and a pair of firewalls in active-passive all have thresholds.
What is the largest number of members that may be unavailable simultaneously? For a threshold system the answer is arithmetic, it is usually one or two, and it belongs in the play as a comment and as an assertion.

Reuse the tasks freely. Never reuse the batching policy without answering those two questions again.

Resolution

Restore quorum first. Bring enough members back to reach the threshold; which members and in what order depends on the cluster technology and belongs in its own runbook.
Let the cluster resynchronise and confirm it from the cluster tooling. A member that is running is not necessarily a member that has caught up, and a cluster at quorum with lagging replicas is still fragile.
Do not resume the play. It would continue from the next batch, and the cluster state no longer matches what the play assumes.
Confirm which nodes actually received the patch and which did not, from the nodes rather than from the run log, since the run was interrupted by the outage rather than by a failure.
Change serial to 1 for this group and add a comment stating the quorum arithmetic, so the constraint is visible to the next reader.
Add a health gate between members that requires all members present and quorum held before the next node is touched, and make it able to fail the play.
Add an assertion at the start of the play that the batch size cannot exceed the tolerable number of unavailable members, so a future edit to serial is caught rather than discovered.
Restart the patch cycle deliberately once the cluster is fully healthy, one node at a time, with the gate in place.

Verification

The batch plan is one host at a time. Run the corrected play on a staging cluster and confirm the PLAY header appears once per host in the run output.
The health gate can fail. Stop one member by hand during a staging run and confirm the play refuses to proceed to the next node. This is the check that can fail and the one the original play did not have.
The batch-size assertion can fail. Set serial to 2 on a scratch branch and confirm the play refuses to start.
Quorum is confirmed from the cluster between every node, using the cluster tooling. The Ansible run reporting ok is not evidence about cluster state.
Every member is present, synchronised and on the intended version after the patch cycle, checked from the cluster rather than from the package manager.
Dependent services recovered. Confirm from the services that depend on the cluster, not only from the cluster itself.
The runbook records the arithmetic. The number of members, the quorum threshold, and the maximum number that may be unavailable at once are written down where the next operator will read them.

Prevention

Set serial: 1 for any tier with a quorum, a write threshold or an active-passive pair, and write the arithmetic beside it:

# 7 members, quorum 4, so at most 3 may be unavailable.
# One at a time is the only batch size that also survives an
# unplanned failure during the window.
serial: 1

Never reuse a batching policy across tiers. Reuse the tasks; re-derive serial, max_fail_percentage and any_errors_fatal from the failure semantics of the tier being patched.
Assert the batch size in the play, so an edit to serial is caught by the play rather than by the cluster.
Put a health gate between members and require it to be able to fail. A gate that only ever passes is decoration, and on a cluster it is the only thing standing between a slow patch and an outage.
Print the batch plan before running anything that writes. --list-hosts shows the hosts and hides the grouping, and the grouping is what decides whether a threshold system survives.
Treat a successful canary as evidence about the change, never about the batching that follows it. On a ramp, the canary is immediately followed by the riskiest stage.
Pause deliberately between stages of any ramp, with a check that a human or a gate has to pass. Automatic acceleration is the feature that turns a good first result into a bad second one.

What if the play had used any_errors_fatal instead of a ramp?

That would have addressed a different problem, and it is worth knowing which one.

any_errors_fatal stops the play for every host as soon as any host fails. On a cluster that is usually the right failure policy, because the members are not independent and a partial change is often worse than no change - if node two fails to rejoin, you want the play to stop rather than proceed to node three.

But it says nothing about how many hosts are touched at once. With serial: [1, 5, 100%] and any_errors_fatal: true, the second stage still takes five nodes simultaneously, and if none of them fails a task the play continues cheerfully while the cluster is down. Losing quorum is not a task failure; every task succeeded.

The two settings answer different questions:

serial decides how many hosts may be affected at once. On a threshold system this is the setting that prevents the outage.
any_errors_fatal and max_fail_percentage decide what to do after something fails. They limit the damage of a bad patch, not the damage of a correct patch applied too widely.

A cluster patch play needs both, and it needs a health gate as well, because the state that matters - quorum - is not observable from task results at all.

Why this matters

The play was correct, the hosts were correct, the canary succeeded, and the pre-flight check passed. The cluster went down because five machines that have to agree with each other were restarted at the same time, and nothing anybody ran before the play could have shown that was going to happen.

Two things generalise. The first is specific and learnable: serial as a list is a ramp, batching is invisible to --list-hosts, and the repeated PLAY header in the output is the only place the batch plan appears. If a play writes to a threshold system, print the batch plan before running it.

The second is the more valuable one. Tasks are portable between tiers and execution policy is not. How many hosts at once, and what to do when one fails, are answers about the failure semantics of the thing being changed. Copying a play from a tier that degrades to a tier that has a threshold carries the tasks across correctly and the safety across completely wrongly - and the commit that does it looks like it changes nothing at all.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. serial as a list means a ramp

2. A consensus cluster fails at a threshold, not gradually

3. The successful canary supplied confidence about the wrong thing

Resolution

Verification

Prevention

1. `serial` as a list means a ramp