Runbook: Perform a canary deployment

A canary deployment is a bet that one host tells you what forty will do. The bet only pays if two conditions hold: the canary carries real traffic, and you have decided in advance what observation would make you stop.

Without the first, the canary proves the software starts. Without the second, the canary becomes a formality - it went green, everyone proceeds, and the thing that would have been visible in the error rate was never anybody’s job to look at.

When to use this runbook

• Deploying a new application version to a group of servers.
• Rolling out a configuration change with a plausible failure mode.
• Any change where “it worked in staging” is true and insufficient.

Blast radius

Stated as a sequence, because that is the point:

canary        1 host      real traffic, reversible in minutes
batch 1       25%         reversible, but ten hosts to reverse
batch 2       25%         half the fleet is now on the new version
batch 3       25%
batch 4       25%         complete

The number to write in the change record is not “the web group”. It is hosts (40), taken from --list-hosts on the exact command you will run.

Inputs

• The version identifier being deployed, and the one being replaced.
• The canary hostname.
• The health check, and evidence it can fail.
• The stop rule, agreed with the service owner.
• Baseline metrics: error rate, p95 latency, request rate.

Step 1: Baseline

# Version currently running, read from the service
ansible web -m uri -o \
-a 'url=http://{{ ansible_host }}:8080/version return_content=true' \
| tee baseline-versions.txt

# Metrics, from wherever they live
curl -sS 'http://metrics.example.com/api/v1/query?query=error_rate_5m' \
> baseline-errors.json

Record the numbers, not the impression. “Latency looked normal” is not something you can compare against forty minutes later, and forty minutes later is when you need to.

Step 2: Prove the health check can fail

This is the step that distinguishes a canary from a ceremony.

- name: Application answers correctly
ansible.builtin.uri:
  url: "http://{{ ansible_host }}:8080/healthz"
  status_code: 200
  return_content: true
  timeout: 10
register: health
retries: 6
delay: 10
until: health.status == 200 and 'ok' in health.content

Now break something in staging on purpose - stop the upstream, corrupt the config, point it at a dead database - and run the check. It must fail.

Step 3: Deploy to the canary

ansible-playbook -i inventories/production deploy.yml \
--limit web01.example.com \
-e app_version=2.4.0 \
--list-hosts

Run --list-hosts on the exact command, including every flag. It is two seconds and it is the only thing between a mistyped limit and a fleet-wide deployment.

ansible-playbook -i inventories/production deploy.yml \
--limit web01.example.com \
-e app_version=2.4.0 \
--diff | tee "canary-deploy-$(date -u +%Y%m%dT%H%M%SZ).log"

If the host is behind a load balancer, drain it first and return it afterwards. Deploying to an in-rotation host means users see the restart, and it also means your canary observation is contaminated by the deployment itself.

- name: Canary deployment
hosts: canary
become: true
tasks:
  - name: Remove from the load balancer pool
    ansible.builtin.uri:
      url: "https://lb.example.com/api/pool/web/{{ inventory_hostname }}/drain"
      method: POST
      status_code: [200, 204]
    delegate_to: localhost

  - name: Wait for connections to finish
    ansible.builtin.wait_for:
      timeout: 30
    delegate_to: localhost

  - name: Deploy the application
    ansible.builtin.include_role:
      name: app_deploy

  - name: Application answers correctly before it takes traffic
    ansible.builtin.uri:
      url: "http://{{ ansible_host }}:8080/healthz"
      status_code: 200
      return_content: true
    register: health
    retries: 6
    delay: 10
    until: health.status == 200 and app_version in health.content

  - name: Return to the load balancer pool
    ansible.builtin.uri:
      url: "https://lb.example.com/api/pool/web/{{ inventory_hostname }}/enable"
      method: POST
      status_code: [200, 204]
    delegate_to: localhost

The order matters: the health check sits between the deployment and the return to the pool. A host that fails the check never takes traffic, and the play fails there rather than putting a broken instance back into rotation.

Step 4: Verify the canary

# Version, read from the running service
curl -sS http://192.0.2.11:8080/version

# Real request, with timing
curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' \
http://192.0.2.11:8080/api/orders

# Errors since the deployment
ansible web01.example.com -b -m command \
-a 'journalctl -u app --since "10 min ago" --no-pager -p err' -o

Read the version from the service, not from the package manager. A package can be installed while the old process keeps running, which is the exact failure a canary is supposed to catch and the exact failure rpm -q reports as success.

Step 5: Hold, and watch against the baseline

The hold is where the canary earns its name. Watch:

• Error rate compared with the baseline, on this host specifically. A one-in-forty host contributes 2.5% of fleet traffic; a fleet-level error rate graph will not show it.
• Latency, same reason.
• Log volume. A large increase in log lines is often the first sign of a retry loop that has not failed yet.
• Memory, if the change could plausibly leak. This is the class of problem that only appears after the hold, which is why the hold has a duration rather than being “until it looks fine”.

Duration comes from the traffic pattern: long enough that the service has done its normal range of work at least once. For a request-serving tier that is often thirty to sixty minutes. For something with a nightly batch, the honest answer is a day.

Step 6: The gate

Three outcomes, decided against the stop rule agreed before you started:

Record the decision and the numbers behind it. The value of writing it down is not the audit trail; it is that a decision you have to write is a decision you have to actually make.

Step 7: Widen in batches, with the play gating itself

- name: Rolling deployment
hosts: web
become: true
serial:
  - 1
  - "25%"
max_fail_percentage: 0
tasks:
  - name: Deploy the application
    ansible.builtin.include_role:
      name: app_deploy

  - name: Host answers correctly before the batch is considered done
    ansible.builtin.uri:
      url: "http://{{ ansible_host }}:8080/healthz"
      status_code: 200
      return_content: true
    register: health
    retries: 6
    delay: 10
    until: health.status == 200 and app_version in health.content

max_fail_percentage: 0 means any failure in a batch ends the play, and subsequent batches never start. That is the automatic version of the gate, and it works when nobody is watching.

Verified batch arithmetic on 2.21.3: a percentage that does not divide evenly rounds down to whole hosts with a minimum of one. serial: [1, 2, "30%"] on six hosts produced batches of 1, 2, 1, 1, 1 - five batches, not three. Read the batch boundaries in the output; do not calculate them in your head and assume.

Also verified: run_once: true under serial runs once per batch, not once per play. A “notify the channel that the rollout started” task marked run_once will fire on every batch.

Step 8: Verify the whole group

ansible web -m uri -o \
-a 'url=http://{{ ansible_host }}:8080/version return_content=true' \
| tee final-versions.txt

grep -c '2.4.0' final-versions.txt
ansible-playbook -i inventories/production deploy.yml --limit web --list-hosts \
| grep -c 'example.com'

Those two counts must match. A host on the old version at the end of a rollout is version skew, and version skew is the state where two hosts behind the same load balancer answer the same request differently - which produces intermittent, unreproducible failures that nobody attributes to the deployment.

Rollback

Rollback is per batch, which is the entire reason batches exist.

ansible-playbook -i inventories/production deploy.yml \
--limit web01.example.com \
-e app_version=2.3.7 \
--diff

curl -sS http://192.0.2.11:8080/version   # must report 2.3.7

# The hosts that received the new version, from the run log
grep -E 'ok:|changed:' "canary-deploy-*.log" \
| awk -F'[][]' '{print $2}' | sort -u > deployed-hosts.txt

LIMIT=$(paste -sd, deployed-hosts.txt)
ansible-playbook -i inventories/production deploy.yml \
--limit "$LIMIT" -e app_version=2.3.7 --list-hosts
ansible-playbook -i inventories/production deploy.yml \
--limit "$LIMIT" -e app_version=2.3.7 --diff

Roll back by deploying the recorded previous version, not by checking out an old branch and re-running. Those are different things: the branch may contain other changes, and “the state before” is a version identifier, not a point in Git history.

After any rollback, verify version consistency across the whole group. A fleet half on 2.4.0 and half on 2.3.7 is worse than either, and it is the state a panicked partial rollback produces.

Common patterns

Escalation

Escalate when:

• The canary observation is ambiguous. Ambiguity is a stop, and overriding a stop is the service owner’s call.
• Something went wrong after an irreversible step.
• Hosts were skipped and their state is unknown.
• Hosts cannot be drained before deployment.