The sequence before a production run · Ansible

Each of the previous six lessons produced a control with a stated limit. This one puts them in order, because the ordering is where most of the value is: the rungs are arranged so that each one is cheaper than the next and rules out a class of mistake that would otherwise waste the next one’s time.

#	Rung	Catches	Blind to
1	`--syntax-check`	YAML errors, unresolvable module names, missing roles and static imports	Module options, variables, template sources, dynamic includes
2	`ansible-lint`	Deprecated constructs, non-determinism, missing file modes, shell where a module exists	Anything that depends on a value; anything computed at run time
3	`--list-hosts`	The wrong target set, a pattern that matched more than intended	What would change; whether the hosts are reachable
4	`--check` on one canary	Connectivity, authentication, escalation, guard evaluation, template resolution, a change count	Unsupported modules, sequential dependencies, environmental preconditions, everything second-order
5	`--check --diff` on one canary	The exact content of predicted file changes	Whether that content is correct or the service accepts it
6	Real run under `--limit` to the canary	Everything the earlier rungs are blind to, on one host	Behaviour at scale, concurrency, load

The third column is the one to read. Every rung above is satisfiable without establishing safety, and the reason to write the blind spots down is that a ladder whose limits are undocumented gets treated as complete by whoever inherits it.

The sequence, as an actual procedure

Read-only / Saferungs 1 to 3 - free, connect to nothing— Runs on any workstation against production inventory without touching a host. Safe to run at any time, which is the point.

set -euo pipefail

PLAY=site.yml
INV=inventories/production/hosts
CANARY=web01.example.com

# 1. It compiles.
ansible-playbook -i "$INV" "$PLAY" --syntax-check

# 2. It is not obviously wrong. Fail on the target profile.
ansible-lint "$PLAY"

# 3. It targets who you think. READ THIS OUTPUT - do not just check the
#    exit code, which is 0 for any resolvable pattern including a wrong one.
ansible-playbook -i "$INV" "$PLAY" --list-hosts
ansible-playbook -i "$INV" "$PLAY" --list-tasks

Read-only / Saferungs 4 and 5 - one host, no changes— Connects to the canary, authenticates, escalates and executes module code there in check mode. Writes nothing.

# 4 and 5 together. One host: diff output for a fleet is unreadable, and
#    the host whose diff differs from the rest is invisible in it.
ansible-playbook -i "$INV" "$PLAY" --limit "$CANARY" --check --diff

Service impact possiblerung 6 - the canary is a real change— This changes one production host. Everything before it was preparation; this is the first rung that produces evidence about the change rather than about the playbook.

ansible-playbook -i "$INV" "$PLAY" --limit "$CANARY"

# Then verify the outcome directly, on the host, not from Ansible output:
#   the service is running, the port answers, the log is clean, the
#   dependent system still works. Ansible reporting ok means the module
#   was satisfied, not that the service works.

The stop conditions

A rung is not “run it and look at the exit code”. Each has something to read and a condition that stops the sequence.

1. --syntax-check. Stop on any non-zero exit. Note that 120 means the inventory did not parse and 1 usually means a referenced file or role is missing — neither is a problem in the playbook body, and knowing that saves twenty minutes.

2. ansible-lint. Stop on a new finding in a file this change touches. Pre-existing findings elsewhere are debt, not a stop condition — that distinction is what makes the linter adoptable at all.

3. --list-hosts. Stop if the count differs from what you expected, in either direction. Fewer is as alarming as more: a pattern that matched twelve hosts when you expected fourteen means two are missing from the inventory, and the run will report complete success while leaving them behind.

4 and 5. --check --diff on the canary. Three separate reads:

The skipped count, against the modules you expect to be skipped. A surprise here means something is not being evaluated and you do not know what.
The diff hunks, line by line. This is a code review of the rendered output and it is the reason the rung exists.
The first task whose success depends on a previous task’s real effect. Everything after it is reported against a world that will not exist.

Stop if the diff contains a line you did not intend, or if the check run ended earlier than the play does.

6. The canary run. Stop unless the outcome verifies on the host. Not changed=3 in the recap: the service is running, the port answers, the dependent thing still works. This distinction — module satisfied versus service working — is the subject of the testing part, and it is the one that separates a canary from a slower deployment.

Every rung can be satisfied without being read

This is the failure mode of the whole procedure, and it is not hypothetical: it is what happens to any checklist after the twentieth time.

--syntax-check exits 0 and nobody looks at what it did not check.
ansible-lint exits 0 because the skip_list grew.
--list-hosts exits 0 for any pattern that resolves, including one resolving to the wrong fleet, and the number is never compared to an expectation.
--check exits 0 with skipped=6 and the changed column is the only one read.
--diff prints forty lines that get scrolled past.

Every rung passed. Nothing was examined. The remedy is that each rung’s output should produce a number or a sentence written down — hosts targeted, tasks evaluated, files changed, lines changed. A rung that produces nothing to record is a rung that will be skipped in spirit while being run in fact.

What the change record should carry

The output of the sequence is a paragraph, and it is a much better artefact than a screenshot of a green run:

Playbook:      site.yml @ 4f2a91c
Inventory:     inventories/production/hosts
Target:        webservers (14 hosts, confirmed with --list-hosts)
Static gates:  syntax-check pass; ansible-lint pass at profile 'safety'
Dry run:       --check --diff --limit web01.example.com
               18 of 22 tasks evaluated; 4 command tasks skipped
               (no check-mode support); 3 files predicted to change,
               diffs reviewed
Not validated: the 4 command tasks; anything after task 15, which
               depends on the package install
Canary:        web01.example.com, real run, service verified on host
Rollout:       --limit 'webservers:!web01.example.com', serial 4

Two lines in that are unusual and both are the point. “18 of 22 tasks evaluated” replaces “check mode passed” with something a reviewer can weigh. “Not validated” is a section most change records do not have, and it is the one that would have prevented the incidents in the previous lessons.

Why the order is cost-ascending, and where it stops being free

The boundary in the ladder is between rungs 3 and 4, and it is sharper than the table suggests.

Rungs 1 to 3 are handled by the CLI before PlaybookExecutor starts. They parse files, resolve the inventory and print. No socket is opened. They can be run against production inventory from a laptop, in a hook, on every commit, by anyone, with no authorisation, at no risk.

Rung 4 is a real run. It opens SSH to every targeted host, authenticates, escalates if the task escalates, writes a temporary directory, transfers module code and executes it. The only difference from a production run is a flag in the module arguments that each module is trusted to honour.

That is why the canary limit belongs on rung 4 rather than rung 6. --check against two hundred production hosts is two hundred authenticated sessions and a load spike, for output nobody can read. The sequence puts --limit in the moment the ladder stops being free, not the moment it stops being safe.

The second reason for the ordering is arithmetic. Rung 1 costs a second, rung 6 costs a change window and a rollback plan. Every mistake caught by a cheaper rung is a mistake that did not consume an expensive one — and on an inherited repository, most of the mistakes are in the first three rungs, which is exactly why they are worth automating and exactly why they are not sufficient.

What happens when the change window is 30 minutes and the sequence takes 40?

The sequence gets truncated, and which end you truncate from is the decision worth making in advance rather than at 22:50.

The rungs are not equally compressible. Rungs 1 to 3 cost seconds and can never be the reason you ran out of time; if they were skipped, it was for a reason other than time. Rung 5’s diff review is the one that takes human minutes and it is the most valuable minute in the list. Rung 6 — the canary and its verification — is the only rung that produces evidence about the change rather than about the playbook.

Which gives the priority order under pressure:

Move rungs 1 to 3 out of the window entirely. They belong in CI, on every commit. If they are being run during a change window, something is misconfigured upstream.
Do the canary. One host, real run, verified on the host. If only one rung fits, it is this one.
Do the diff review on the canary before it. Reading the diff costs two minutes and is the last chance to see a wrong value before it is on a machine.
Reduce the rollout, not the verification. A change window that cannot fit the whole fleet after a proper canary should deploy fewer hosts, not verify less. Half a fleet correctly changed is a normal state. A whole fleet changed on the strength of a skipped canary is an incident waiting for its trigger.

The pressure to skip verification always presents itself as pressure to finish the rollout. Those are separable, and separating them is the decision.

Knowledge check

Why this matters

The ladder is not ceremony, and the way to tell is that each rung has a different blind spot. A sequence of checks that all fail on the same class of defect is ceremony; a sequence where rung 2 catches what rung 1 cannot and rung 6 catches what neither can is a design.

The part that survives longest is the third column of that first table. An engineer who can say what --syntax-check does not check, and what a green --check does not prove, will read a validation report for what it omits. An engineer who cannot has a pipeline of green ticks and no picture of the residual risk.

That picture is the deliverable. Not a green run — a sentence that says what was validated, what was not, and on how many hosts. Every incident in this part happened to somebody who had run the checks and had not written that sentence.

The ladder, with its blind spots

The sequence, as an actual procedure

The stop conditions

What the change record should carry

Knowledge check