Anti-pattern: automation nobody tests, and automation nobody dares run · Ansible

This is one anti-pattern with two ends, and the second follows from the first with enough reliability that they belong in the same lesson.

The first end: no lint, no --syntax-check, no check-mode run, no Molecule scenario, no review. Production is the first test environment.

The second end, which is where it goes: the automation has broken things often enough that nobody wants to run it, so hosts get fixed by hand. That is drift with extra steps, plus a repository that now lies about the estate.

End one: production as the first test environment

Read-only / Safethe five layers, cheapest first - each takes seconds— READ-ONLY: all five only read or simulate. The first four require no infrastructure at all.

# 1. Parse. Catches YAML errors, undefined roles, bad task keywords.
ansible-playbook site.yml --syntax-check

# 2. Lint. Catches anti-patterns from this whole part, by rule.
ansible-lint --profile production

# 3. Inventory resolution. Proves the targeting is what you think.
ansible-inventory -i inventory/ --graph
ansible-playbook site.yml --limit app_prod --list-hosts

# 4. Check mode with diff, against a canary.
ansible-playbook site.yml --limit canary --check --diff

# 5. Idempotence. The second run should report changed=0.
ansible-playbook site.yml --limit canary
ansible-playbook site.yml --limit canary

Five commands. Nothing to install beyond ansible-lint, no test infrastructure, no container runtime. Every one of them catches a class of defect that would otherwise be caught by a managed node.

Molecule sits above these: a role tested in a container, converged, verified and converged again, without a managed node. Part XXVI covers it. It is the most valuable layer and the most expensive one, which is why it belongs after the five above rather than instead of them.

The specific omission: no validation of what gets written

Configuration changethe difference validate: makes— CONFIGURATION: writes a config file. With validate, a file that fails the check is never installed - the module writes to a temporary path, runs the validator, and only then moves it into place.

# Without: a broken config is written, then the handler restarts
# the service into it.
- name: Install the nginx configuration
ansible.builtin.template:
  src: nginx.conf.j2
  dest: /etc/nginx/nginx.conf
notify: Restart nginx

# With: the file is validated before it replaces the live one.
- name: Install the nginx configuration
ansible.builtin.template:
  src: nginx.conf.j2
  dest: /etc/nginx/nginx.conf
  validate: 'nginx -t -c %s'
notify: Restart nginx

validate: is a per-task control that costs one line and converts “write a broken config and restart into it” into “fail before touching the live file”. Combined with serial, the failure stops at host one. Part XVII covers the render-check-activate pattern in full.

End two: automation nobody dares run

The endpoint has a recognisable set of symptoms. If several of these are familiar, this is where the repository is.

The playbook is run only by one person, who "knows how it behaves".
Runs are always scoped with --limit to a handful of hosts, never to a group, because a wider run is felt to be risky.
There is a step in the runbook that says "check with the team before running site.yml".
Hosts are being fixed by hand during incidents, with the intention of putting it in the playbook afterwards - which happens sometimes.
Nobody has run the full playbook against the full fleet in more than six months.
A new engineer is told which playbooks are safe and which are not, verbally.

Now the repository lies, and that is worse than having none

While hosts are being fixed by hand, the repository stops describing the estate. It continues to look authoritative — it is in version control, it has commit history, it is what anyone would consult.

The failures that follow are all failures of misplaced trust:

A rebuilt host is wrong. Someone provisions a replacement from the playbook and gets a host missing eighteen months of manual fixes, because the playbook does not know about them.

Incident diagnosis starts from a false premise. “The config should be X, the repository says so” — and it is not X, because somebody changed it by hand in 2025.

Compliance evidence is fabricated in good faith. The repository is offered as evidence of what the estate runs, and it is not.

Drift detection is meaningless. A check-mode run against a manually diverged fleet reports enormous drift, so the report becomes noise and gets ignored — which returns you to the failure in lesson 5 by a different route.

Having no repository at all is a known-unknown. A repository nobody has run for a year is an unknown-unknown wearing the clothes of a control.

Getting out

The instinct is to run the playbook against the fleet and let it converge. That is the one move guaranteed to confirm everyone’s fears.

Measure the divergence before changing anything. --check --diff against one host per group, read the diffs, and write down what has drifted. This is read-only and it is the whole basis for what follows.
Triage each difference: a manual fix that should be in the repository, a manual fix that was wrong, or a genuine per-host exception that belongs in inventory as data.
Codify the first category. Every manual fix that should be automation becomes a commit. The repository starts describing reality again, one difference at a time.
Declare the third category. Exceptions belong in group_vars or a documented exception group, with a reason and a review date, per Part XXXVI.
Only then converge, and do it with staging: one canary host, then a small group, then the tier. The first honest run after a long gap is the most dangerous run in this course, and it should be treated like a migration.
Add the five testing layers before the second convergence, so the gap does not reopen.

The order matters. Codifying before converging is what turns a frightening fleet-wide change into a series of reviewable commits whose effect is already known, because the diff was read first.

Under the hood

Why does untested automation reliably become feared automation, rather than simply staying untested?

Because of an asymmetry in how the two kinds of outcome accumulate.

A successful run produces nothing memorable — the hosts were already converging, the recap looked normal, nobody thought about it. A failed run produces an incident, a postmortem, a story, and a durable association between the playbook and the outage.

So the evidence a team holds about its own automation is systematically biased toward failure, regardless of the actual success rate. Ten thousand uneventful task executions leave no trace; one broken template that reached three hundred hosts is remembered for years and is retold to every new joiner.

The behavioural response is rational given that evidence: reduce exposure. Run with a tighter --limit. Have the person who knows it best do it. Fix this one host by hand instead. Each decision is individually sensible and each one reduces the rate at which the automation is exercised — which reduces the rate at which defects are found, which increases the chance that the next full run breaks something, which produces another memorable failure.

That is the loop, and it is self-reinforcing. Testing breaks it at the input: the five layers move defect discovery off production entirely, so the memorable-failure rate drops and the evidence base slowly changes. Staging breaks it at the output: a defect that reaches one canary host instead of three hundred produces a small correction rather than a story.

Which is why this lesson closes the part. Almost every other anti-pattern here is a way of making a failure larger, later, or harder to see — and this one is the mechanism by which the accumulated failures turn a working tool into one that nobody uses.

The part, as a review checklist

Each anti-pattern with the incident class it produces. This is the summary artefact — usable directly on a pull request or a repository review.

Anti-pattern	Incident class	Corrected form taught in
`shell` everywhere	False assurance — `--check` proves nothing, reruns re-perform	Parts VIII, IX, XII
Secrets in Git	Irrevocable leak, remediated by fleet-wide rotation	Part XXI
No limit, no canary	Fleet-wide outage with no healthy population left	Parts XXX, XXXI
`ignore_errors` everywhere	False assurance — success reported during an outage	Part XXIII
Inaccurate `changed`	Restart storms, then silent drift	Parts XII, XVI
Inventory logic in `when`	Unreviewable targeting; check mode generalises to nothing	Parts V, XV
`latest` everywhere	Unreproducible estate; Part L becomes unachievable	Parts XXVII, XL
Nobody tests, nobody dares run	Silent drift and a repository that lies	Parts XXV, XXVI

Five distinct incident classes across eight anti-patterns: false assurance, irrevocable leak, fleet-wide outage, unreproducible estate, and silent drift. Ranked by blast radius, the leak and the outage come first — but false assurance is the one that enables the others, because it is what stops you noticing.

What happens if the estate is too diverged to converge safely at all?

It happens: hundreds of hosts, years of manual work, and a --check diff so large that triage would take months.

Converging is off the table. Three approaches, and most large recoveries use all three.

Reduce scope until it is tractable. Do not fix the estate; fix one group. Pick the tier with the least divergence, get it converged and tested, and keep it that way. One group that genuinely matches its repository is worth more than a plan covering everything, because it establishes that the state is achievable and gives you a working reference.

Rebuild rather than converge, where you can. For stateless hosts, provisioning a replacement from the playbook and cutting over is often faster and much safer than reconciling a diverged one — and it proves the playbook is complete, because a host built from it either works or does not. This is the phoenix-server approach, and it turns an open-ended reconciliation into a bounded exercise with a pass/fail result.

Narrow what the automation claims. A playbook that manages three files correctly is better than one that claims to manage forty and manages none. Remove the tasks that no longer reflect reality, so what remains is true, and grow the scope back deliberately. The repository becomes smaller and honest, which is a much better starting point than large and wrong.

Accept a documented cut-off for the rest. Hosts that will not be reconciled get recorded as unmanaged, with a reason and a decommission or rebuild date. That is not defeat — it is the difference between an estate with a known unmanaged set and an estate where nobody can tell which hosts are which.

The failure mode to avoid is the six-month project to fix everything, which is abandoned at month four with nothing converged. Small, complete and true beats large and aspirational, every time.

Knowledge check

Why this matters

This part has catalogued eight ways automation goes wrong. Seven of them are defects in what was written. This one is a defect in what was never done, which is why it is the hardest to raise and the easiest to defer.

The loop is worth stating once more because it is the mechanism behind most of the estates this course is written for. Untested automation produces memorable failures. Memorable failures produce caution. Caution produces less exercise, which produces fewer defects found, which produces the next memorable failure. Somewhere in that loop, people start fixing hosts by hand — sensibly, under pressure — and the repository quietly stops being true.

The end state is the worst configuration available: an estate with drift nobody is tracking, a repository everybody trusts and nobody has run, and a team that has the tooling for all of this and has stopped using it.

Getting out is not heroic. It is --check --diff against one host per group, reading the output, and turning what you find into commits. It is five read-only commands added to CI. It is one canary before the group. Every one of those is an afternoon, and each one makes the next run slightly less frightening than the last — which is the only direction that compounds.