Runbook: Test a playbook

Testing a playbook is a ladder, not a switch. Each rung costs more than the one below it and proves something the one below it could not, and the discipline is to climb in order - because a failure on rung 2 costs nothing and the same failure discovered on rung 7 costs a service.

The other half of the discipline is honesty about what each rung does not prove. Every rung here says so explicitly.

When to use this runbook

• A new playbook is going to production for the first time.
• An existing playbook has been changed materially.
• A playbook is being pointed at a new inventory or a new group.
• Before any change window, as the pre-flight for what you are about to run.

Blast radius

Rungs 1 to 3 do not open a network connection at all. Rungs 4 and 6 connect but do not change. That is four rungs of useful signal before anything is at risk.

Rung 1: Does it parse, and does it follow the rules

ansible-lint site.yml
ansible-playbook site.yml --syntax-check
echo "exit=$?"

--syntax-check parses the playbook and everything it statically imports. It does not evaluate Jinja, does not resolve variables and does not connect anywhere.

What it cannot prove: anything about values. A playbook full of undefined variables passes --syntax-check cleanly, because the templates are not rendered until a task runs.

Rung 2: Does it contain what you think

ansible-playbook site.yml --list-tasks
ansible-playbook site.yml --list-tags

Read the task list against the change you intended. The usual finding is a role that pulls in a dependency you forgot about, or a task list twice as long as expected because a role is applied by two plays.

--list-tags matters when the change window will use --tags. A tag that does not appear here does not exist, and a mistyped --tags runs nothing at all while reporting success.

Verified on 2.21.3, a playbook whose only task is tagged deploy, run with --tags deploi:

PLAY [Tagged play] *************************************************************

PLAY RECAP *********************************************************************

An empty recap, no warning, exit code 0. There is no host line to read, which is the tell - but only if you are looking at the recap rather than at the exit code. --list-tags before a tagged change window costs one second and removes the entire failure mode.

Rung 3: What would it target

ansible-playbook -i inventories/production site.yml --list-hosts
ansible-playbook -i inventories/production site.yml --limit web --list-hosts

Verified output shape on 2.21.3:

  play #1 (web): Front-end web tier	TAGS: []
    pattern: ['web']
    hosts (2):
      web02.example.com
      web01.example.com

Read the count. Not the pattern - the count. hosts (2) when you expected two is a pass; hosts (240) when you expected two is the finding this rung exists for, and it is the cheapest place in the whole ladder to find it.

Do this against the production inventory even though you are about to test against staging. The question “what would this do in production” has an answer now, for free, and it is the answer that matters.

Rung 4: What would change, against staging

ansible-playbook -i inventories/staging site.yml \
--check --diff | tee "check-staging.txt"
echo "exit=$?"

--check asks each module to report what it would do. --diff shows the content change for file-shaped modules. Together they are the most information you can get without changing anything.

What check mode cannot prove, and this list is the whole reason rung 5 exists:

• Modules without check-mode support are skipped. Their changes are not predicted at all, and anything downstream of them is evaluated against a world where they did not run.
• command and shell do not run, so a task with when: result.rc == 0 is evaluated against a result that does not exist.
• validate: does not run, because there is no rendered file to validate. A broken template passes check mode and fails for real.
• Handlers are reported as notified but the service is not restarted. Whether the new config is actually loadable is untested.
• Anything that depends on a file an earlier task would have created reports a change that will not happen, or fails.

A clean check-mode run is a necessary gate. It is not a rehearsal.

grep -nE 'ansible\.builtin\.(command|shell|raw|script)' site.yml roles/*/tasks/*.yml
grep -rn 'check_mode:' site.yml roles/*/tasks/*.yml

That first grep is the list of tasks check mode did not evaluate. Read it alongside the diff so you know what the diff left out.

Rung 5: A real run against staging, twice

ansible-playbook -i inventories/staging site.yml --diff | tee "run-staging-1.txt"
echo "exit=$?"

Exit codes, verified on 2.21.3:

Unreachable takes precedence. A CI gate that only checks for 2 treats a run with both failures and unreachable hosts as a pass. Check for non-zero, then read the recap.

Now the second run, which is the part that gets skipped:

ansible-playbook -i inventories/staging site.yml --diff | tee "run-staging-2.txt"
grep -E 'changed=[1-9]' "run-staging-2.txt"

That grep must produce nothing. Any host still reporting changes on the second run contains a non-idempotent task, and in production that means a change - often a service restart - on every scheduled converge forever.

What a staging run cannot prove: that production is like staging. Different data volumes, different existing configuration, different kernel, different hand-edits accumulated over years. That is rung 6.

Rung 6: Check mode against one production canary

ansible-playbook -i inventories/production site.yml \
--limit web01.example.com --check --diff | tee "check-canary.txt"

Then compare the two diffs directly:

diff <(grep -E '^[+-]' "check-staging.txt") \
   <(grep -E '^[+-]' "check-canary.txt") | head -40

Differences are expected - hostnames, addresses, sizes. What you are looking for is a structural difference: a file that changes in production and not in staging, a service that stops in one and not the other, a package version that only production is missing.

A large structural difference is not a reason to be more careful with this playbook. It is a reason to escalate, because it means staging does not represent production and every test anyone runs there is worth less than they think.

Rung 7: Real run on the canary, then hold

ansible-playbook -i inventories/production site.yml \
--limit web01.example.com --diff | tee "run-canary.txt"

Point of no return. One production host has now changed.

# A real request, not a port check
curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' \
http://192.0.2.11:8080/healthz

# Logs from the last few minutes
ansible web01.example.com -b -m command \
-a 'journalctl -u nginx --since "5 min ago" --no-pager -p warning' -o

# Idempotence on the real host
ansible-playbook -i inventories/production site.yml \
--limit web01.example.com --diff

A port check proves a process is listening. It does not prove the application behind it can answer, and a config that loads but points at a dead upstream passes every check except the one that asks for a response.

Hold before proceeding to the rest of the fleet. How long depends on the traffic pattern - long enough for the service to have done its normal work at least once.

Record what you did

For the change record:

• Which rungs were run, and against what.
• The host count from rung 3.
• Anything the check-mode diff did not predict that the real run did.
• The idempotence result on staging and on the canary.
• Any structural difference between the staging and production diffs.

The most valuable line is the second-to-last one. A gap between what check mode predicted and what happened is the thing that tells the next person how much to trust rung 4 for this playbook.

Rollback

Rungs 1 to 4 and 6 have nothing to roll back.

# Files the playbook manages, from the backups its tasks took
ansible web01.example.com -b -m find \
-a 'paths=/etc/nginx/conf.d patterns="*.conf.*"' -o

# Restore the chosen backup, validate, reload
ansible web01.example.com -b -m copy \
-a 'src=/etc/nginx/conf.d/default.conf.REPLACE_ME dest=/etc/nginx/conf.d/default.conf remote_src=true'
ansible web01.example.com -b -m command -a 'nginx -t'
ansible web01.example.com -b -m systemd_service -a 'name=nginx state=reloaded'

Do not leave the canary half-converged while you investigate. Either finish the change or reverse it. A host in an intermediate state that nobody is tracking is how a configuration incident gets discovered three weeks later by someone else.

Common patterns

Escalation

Escalate when:

• The production check diff differs structurally from staging.
• The diff touches something owned by another team.
• The playbook cannot be tested against staging at all.
• Idempotence fails and there is pressure to proceed anyway.