RunBook AcademyAnsible

← All checklists in Ansible

As neededDeployment

Checklist: Canary validation

19 items ·15 critical ·3 warn ·1 info

Run this after the canary has taken the change and before the rolling deployment starts. It is the decision point: everything above the rollout depends on somebody being able to say that one real host took this change and is demonstrably fine.

The commands assume a canary called web01.example.com in the web group. Substitute your own. The uri checks run from the controller with -c local so that the request comes from off the host, which is the point.

What a failure means

A canary finding is not a delay, it is the canary doing its job. The correct response to any critical failure here is to stop the rollout, fix the change, and run the canary again from the top - not to fix it on the canary by hand and proceed, which produces a fleet configured by the playbook and one host configured by you.

The warn items are about whether the rollout will be tidy. They do not invalidate the canary result.

Access this needs

Every command reads. setup, service_facts and uri gather or fetch; --check --diff reports without changing. Two of them are worth naming:

The playbook runs at applied-cleanly and idempotent are real runs, not check-mode runs - they are what puts the change on the canary and then proves it converged. They change the canary and nothing else, because --limit restricts them to it, which is exactly why the limit item comes first.

The journalctl invocation goes through the command module. It reads, but the command module will run whatever you give it, so adapt that item with care.

Where the evidence goes

The recap lines, the uri results and the soak period go in the change record. So does the person who decided the canary passed - because “the canary was green” is a judgement about criteria that were written down beforehand, and the record should show both the criteria and who applied them.

Sign-off

Canary run by: ____________ Time: ___________
Soak start: _______________ Soak end: _______
Promotion approved by: ____ Time: ___________

Critical15 items

What counts as a successful canary was written down before the canary ran: which checks, which thresholds, how long. A criterion invented after the result is a criterion the result will always meet.
The canary runs the same OS release, the same package versions and the same role as the hosts it is standing in for. Prints the facts to compare against a peer; a canary on a newer kernel or an older package set proves the change works somewhere the fleet is not.
```
ansible -i inventories/production web01.example.com,web02.example.com -m ansible.builtin.setup -a 'filter=ansible_distribution*,ansible_kernel' --one-line
```
The canary is a real production host serving real traffic, or the limitation is written down. A canary drained of traffic proves the change installs; it does not prove the service works under load, which is where most of these changes fail.
The run was limited to the canary and the limit was printed before it ran. Prints the host list the run will actually use; one line of output is the pass condition.
```
ansible-playbook -i inventories/production playbooks/deploy.yml --limit web01.example.com --list-hosts
```
The recap shows changes applied and nothing failed or unreachable. Prints the recap line if anything failed; no output is the pass condition. A canary that reports `changed=0` on a change that should have changed something is a canary that did nothing.
```
ansible-playbook -i inventories/production playbooks/deploy.yml --limit web01.example.com | awk '/failed=[1-9]|unreachable=[1-9]/ {print "FINDING: " $0}'
```
A second run against the canary reports `changed=0`. Prints the recap line if it does not. A change that reports itself as changed forever cannot be rolled out with confidence, because every subsequent run looks like it is doing something.
```
ansible-playbook -i inventories/production playbooks/deploy.yml --limit web01.example.com | awk '/changed=/ && !/changed=0/ {print "NOT IDEMPOTENT: " $0}'
```
The service is running and enabled after the change. Prints the host line if the module could not gather service state; read the state of the unit you changed in the output.
```
ansible -i inventories/production web01.example.com -m ansible.builtin.service_facts --one-line
```
The service answers a request from off the host, at the address real clients use. The module fails the task on any status outside the list, so a red result is the finding. Checking from localhost proves the daemon started; it does not prove the listener, the firewall or the load balancer agree.
```
ansible -i inventories/production localhost -c local -m ansible.builtin.uri -a 'url=https://web01.example.com/health status_code=200 timeout=10'
```
The response body is right, not merely 200. A service that returns a cheerful health page while serving the previous version of the application passes every status check and fails the deployment.
```
ansible -i inventories/production localhost -c local -m ansible.builtin.uri -a 'url=https://web01.example.com/version return_content=yes status_code=200' | grep -o '"content": "[^"]*"'
```
The things that talk to this service still work. A configuration change that is correct for the service and wrong for its clients passes every check run against the service itself.
The service log has no new errors since the change. Prints the last error-level entries; anything that appeared after the change timestamp is a finding, and "it was already there" must be verified against a peer rather than assumed.
```
ansible -i inventories/production web01.example.com -m ansible.builtin.command -a 'journalctl -u app --since -15min -p err --no-pager' --one-line
```
Where the change touches anything loaded at boot - a unit file, a sysctl, a mount, a kernel module, a network configuration - the canary has been rebooted and re-validated. This is the only check that proves the change is in the persistent state and not only in the running one.
```
ansible -i inventories/production web01.example.com -m ansible.builtin.setup -a 'filter=ansible_uptime_seconds' --one-line
```
The canary has carried the change for the period agreed in advance, under representative load, including at least one busy period. Fifteen minutes on a quiet afternoon is not a soak.
The canary looks normal in the monitoring system, on the dashboards the team actually uses - latency, error rate, saturation - and not only in the checks this checklist ran.
The rollback has been executed on the canary and the host returned to a working previous state. The canary is the cheapest place in the estate to find out that the rollback does not work; the alternative is finding out on forty hosts.

Warning3 items

The file on disk is what the template was supposed to produce. Re-running in check mode with `--diff` after the change should show nothing; if it shows a diff, the converged state and the desired state disagree, which means something else on the host is also writing that file.
```
ansible-playbook -i inventories/production playbooks/deploy.yml --limit web01.example.com --check --diff
```
Memory, disk and file-descriptor usage on the canary are comparable to a peer that has not taken the change. Prints both hosts so the numbers can be compared; a change that leaks slowly looks perfect for the first twenty minutes.
```
ansible -i inventories/production web01.example.com,web02.example.com -m ansible.builtin.setup -a 'filter=ansible_memfree_mb,ansible_mounts' --one-line
```
The rolling run that follows either skips the canary or converges it harmlessly. A canary re-run by the rollout is fine if the play is idempotent - which the idempotence item above has just proved - and is a second unnecessary restart if it is not.
```
ansible-playbook -i inventories/production playbooks/deploy.yml --limit 'web:!web01.example.com' --list-hosts
```

Info1 item

Anything the canary revealed is written into the change record before the rollout proceeds, including things that were fixed. A canary whose findings are fixed silently teaches the team nothing and leaves the review with no evidence the canary was worth running.