Staged rollout - canaries, cohorts and the health gate

Testing in staging tells you the change works on a system that resembles production. Staged rollout tells you it works on production, on a small enough slice that being wrong is affordable.

The two are not substitutes. Staging catches the failures that depend on the software; staged rollout catches the ones that depend on real traffic, real data, real hardware and the four years of accumulated configuration that no staging environment has ever reproduced.

The cluster lessons in this course bound the batch by quorum and capacity. This lesson is about the sequence of batches: what goes first, how fast it widens, and what stops it.

Why the first stage is different

Stage one exists to answer a question no later stage can: does this change work at all, here?

That is why a canary is one host rather than a percentage. The first host is where you discover that the package needs a dependency the fleet does not have, that the config template renders wrong against real inventory data, that the service takes four minutes to start rather than four seconds, and that the rollback you wrote does not run.

Everything after stage one is asking a different question - whether the change holds up across variety and scale - and it is worth being explicit about which question each stage is answering, because it determines what you watch and for how long.

Choosing a canary that would fail

A canary chosen for convenience answers nothing. The usual bad choice is the newest, quietest, most standard host in the fleet: it is the least likely to exhibit any problem, so it passes, and the failure appears at stage two along with everything else.

Choose a canary that is representative of the risk:

• Serves real production traffic - not drained, not a spare, not out of the load balancer
• Sits in the same configuration cohort as the majority it will vouch for
• Has the hardware, kernel and distribution version the failure would depend on
• Carries enough load that a performance regression would show above the noise
• Is one you can lose: the load balancer can shed it, and taking it out does not breach N+1

The last two are in tension, and that tension is the real work. A host quiet enough to lose safely may be too quiet to reveal a regression. Where the fleet is heterogeneous, the answer is usually several canaries - one per meaningful cohort - rather than one carefully-argued compromise.

Sizing the cohorts

After the canary, each stage should multiply exposure by a bounded factor, so that no single step is the one where you find out. A common and defensible shape:

Stage 0  canary        1 host          soak 1-24 h
Stage 1  early adopter 5% of fleet     soak 2-4 h
Stage 2  quarter       25% of fleet    soak 1-2 h
Stage 3  remainder     100%            soak, then close

Two constraints shape it beyond the percentages.

Failure-domain spread. Each stage should cross domains - racks, availability zones, power feeds, switch pairs - so that a stage passing does not simply mean “everything in rack 4 is fine”. A rollout that follows the inventory order usually follows the rack order, which is exactly the wrong shape.

Never leave the fleet split across a version boundary you have not tested. Mixed-version operation is a state your rollout creates deliberately, and it needs to be a state you have verified: protocol compatibility between old and new, schema compatibility in both directions, and cache or serialisation formats that both versions can read. If old and new cannot coexist, staged rollout is not available to you and the change needs a different strategy - which is a design finding, best made before the window.

Soak time comes from detection latency

The most common way to run a “staged” rollout badly is to stage it and then move through the stages in ten minutes. The stages exist to give the failure time to appear, so the soak time has to come from how long the failure takes to become visible.

The bottom two rows are the ones that make people uncomfortable, and correctly so: some failure modes cannot be caught by waiting. Reboot the canary as part of the canary stage rather than discovering at the next unplanned reboot that the initramfs was never rebuilt.

The gate

Between stages, gate on health, and make the gate a command that returns an exit status rather than a person forming an opinion.

$ systemctl is-active myapp.service; systemctl show myapp.service -p NRestarts -p ActiveEnterTimestamp

active
ActiveEnterTimestamp=Tue 2026-08-11 15:38:02 UTC
NRestarts=4

$ curl -sS -o /dev/null -w 'code=%{http_code} total=%{time_total}s\n' --max-time 5 http://127.0.0.1:8080/health; journalctl -u myapp --since '15 min ago' -p err --no-pager | wc -l

code=200 total=0.041s
0

A usable gate has four clauses, and the fourth is the one people leave out:

1. The service is running and has not restarted since the change was applied
2. A functional check passes from outside the host, not only from localhost
3. Error rate and latency are within the band recorded in the pre-change capture
4. The change is actually present - verify the version, the config value, the running kernel, rather than assuming the apply succeeded

The fourth clause catches the rollout that reports success on every stage while having applied nothing: a configuration management run that failed silently, a package that was held, a drop-in that landed in the wrong directory. A gate that only measures health will happily pass a fleet where the change never took effect, and then the same change is “already applied” for the next six months.

Stop, do not pause

A failed gate stops the rollout. It does not become a conversation about whether to continue, because that conversation is always held under time pressure by the person who most wants the change to succeed.

Write the stop condition into the tooling so it is the default:

set -euo pipefail
for host in $(cat cohort-2.txt); do
  ssh "$host" 'sudo /usr/local/sbin/apply-change.sh'
  if ! ssh "$host" '/usr/local/sbin/health-gate.sh'; then
    echo "GATE FAILED on $host - stopping rollout" >&2
    exit 1
  fi
  sleep "${SOAK_SECONDS:-120}"
done

A half-applied fleet is an uncomfortable state and a recoverable one. A fully-applied broken fleet is neither.

Knowledge check