The layered testing model: what each rung actually proves · Ansible

Testing automation is not one activity. It is a ladder of layers, each of which is cheap relative to the one above it and blind to something the one above it can see.

Most teams build one or two rungs, then reason as though they had built the ladder. That is the failure this part exists to prevent, and the place it goes wrong first is the very bottom, where a command that reports success is routinely mistaken for a test.

The ladder

#	Layer	Typical cost per run	What it can catch	What it structurally cannot
1	`--syntax-check`	under a second	the file is not loadable as a playbook	anything about what the tasks do
2	`yamllint`	under a second	text-level YAML defects: duplicate keys, tabs, `truthy` values	anything requiring knowledge of Ansible
3	`ansible-lint`	seconds	risky and non-idempotent patterns, deprecations, naming, FQCN	whether the values are correct
4	schema and argument validation	seconds	a task that names an option the module does not have	whether the option value is right for this host
5	disposable integration (Molecule)	minutes	the role converges, is idempotent, and reaches its declared outcome	anything the container does not model
6	staging	tens of minutes	integration with real services, real init system, real network	anything about production data and load
7	canary in production	one host’s worth of risk	everything, on one host	nothing — and that is the point

Rungs 1 to 4 are static: nothing is executed against a host. Rungs 5 to 7 are dynamic, and each one buys fidelity by giving up speed.

The reason to know the ladder as a ladder is triage. When a defect reaches production, the question afterwards is not “should we test more?” — it is which is the lowest rung that could have caught this, and why did it not run there? That question has an answer. “We should have tested more” does not.

Rung 1 is almost empty, and this is the surprise

ansible-playbook --syntax-check is the first thing everyone reaches for, and its name flatters it. Here is a playbook with two defects that would fail on the first host it touched:

Read-only / Safebroken.yml— state: presnt is not a valid state for the package module, and onwer is not an option of the copy module.

---
- name: Deploy the web tier
hosts: web
gather_facts: false
tasks:
  - name: Install the package
    ansible.builtin.package:
      name: nginx
      state: presnt

  - name: Write the config
    ansible.builtin.copy:
      src: nginx.conf
      dest: /etc/nginx/nginx.conf
      onwer: root

Read-only / Safethe syntax check is happy— Executed against ansible-core 2.21.3. Exit code 0.

$ ansible-playbook -i inventory.ini --syntax-check broken.yml

playbook: broken.yml

That is the entire output, and the exit code is 0.

--syntax-check loads the playbook, parses the YAML, and confirms that the resulting structure is a playbook: plays with hosts, task lists that are lists, keywords that exist. It does not import modules and it does not validate their arguments, because argument validation is something the module itself performs, on the target, at execution time.

So a green --syntax-check means this file can be loaded. It is worth running — an unloadable playbook that reaches CI wastes an agent slot — but it is not a test of the automation. Treating it as one is how a pipeline ends up with a green badge and no coverage.

Rung 3: ansible-lint reads Ansible, not YAML

ansible-lint is the first rung that knows what a task is. The course targets ansible-lint 26.x; the output below was produced by 26.6.0 running against ansible-core 2.21.3.

Here is a role with the defects people actually write:

Read-only / Saferoles/webserver/tasks/main.yml— Four tasks, deliberately written the way a role looks before anyone has linted it.

---
- name: install nginx
package:
  name: nginx
  state: latest

- name: Deploy config
ansible.builtin.copy:
  src: nginx.conf
  dest: /etc/nginx/nginx.conf
  mode: 644

- ansible.builtin.shell: cat /etc/passwd | grep -c bash

- name: Reload nginx
ansible.builtin.command: systemctl reload nginx

Read-only / Safeansible-lint 26.6.0— Executed. Exit code 2. Nine violations across three files, at the default profile.

$ ansible-lint --offline --show-relpath -f full .

WARNING  Listing 9 violation(s) that are fatal

# Rule Violation Summary

1 command-instead-of-module profile:basic tags:command-shell,idiom
1 name profile:basic tags:idiom
1 name profile:basic tags:idiom
1 package-latest profile:basic tags:idempotency
1 risky-octal profile:basic tags:formatting
1 risky-shell-pipe profile:basic tags:command-shell
2 no-changed-when profile:basic tags:command-shell,idempotency
1 fqcn profile:basic tags:formatting

Failed: 9 failure(s), 0 warning(s) in 3 files processed of 4 encountered.
Last profile that met the validation criteria was 'min'.

name[casing]: All names should start with an uppercase letter.
roles/webserver/tasks/main.yml:2:9 Task/Handler: install nginx

package-latest: Package installs should not use latest.
roles/webserver/tasks/main.yml:2 Task/Handler: install nginx

fqcn[action-core]: Use FQCN for builtin module actions (package).
roles/webserver/tasks/main.yml:3:3 Use `ansible.builtin.package` or `ansible.legacy.package` instead.

risky-octal: `mode: 644` should have a string value with leading zero `mode: "01204"` or use symbolic mode.
roles/webserver/tasks/main.yml:7 Task/Handler: Deploy config

name[missing]: All tasks should be named.
roles/webserver/tasks/main.yml:13 Task/Handler: shell cat /etc/passwd | grep -c bash

no-changed-when: Commands should not change things if nothing needs doing.
roles/webserver/tasks/main.yml:13 Task/Handler: shell cat /etc/passwd | grep -c bash

risky-shell-pipe: Shells that use pipes should set the pipefail option.
roles/webserver/tasks/main.yml:13 Task/Handler: shell cat /etc/passwd | grep -c bash

command-instead-of-module: systemctl used in place of systemd module
roles/webserver/tasks/main.yml:15 Task/Handler: Reload nginx

no-changed-when: Commands should not change things if nothing needs doing.
roles/webserver/tasks/main.yml:15 Task/Handler: Reload nginx

Every one of those has an operational reason behind it. Reading them as a style checklist is how a team ends up with a skip_list and no benefit.

risky-octal is the one worth reading twice. The message says mode: 644 should be mode: "01204". That is not a typo in the linter. YAML parsed the unquoted 644 as the decimal integer six hundred and forty-four, and Ansible will apply it as a mode — which in octal is 01204, the setuid bit plus permissions nobody intended. The linter is telling you what the file currently means, and it is not what it looks like.

package-latest is tagged idempotency. state: latest means the task’s result depends on what a mirror published this morning, so the same code produces ok today and changed tomorrow with no commit in between. That destroys the second-run signal covered in the next lesson, and it makes a rollback ambiguous — you cannot roll back to “latest”.

no-changed-when appears twice, on both the shell and the command task. Those modules cannot know whether they changed anything, so they report changed unconditionally. Two unfenced commands are a permanent noise floor of two on every host, forever.

risky-shell-pipe is about set -o pipefail. Without it, the exit status of cat /etc/passwd | grep -c bash is grep’s alone. A failure in the first command of the pipeline is silently discarded and the task reports success.

fqcn[action-core] and name[casing] / name[missing] look cosmetic and are not. FQCN is about which module actually resolves on the controller that runs the change, which can differ between your laptop and the CI runner. Task names are what the recap, the callback output and the audit log contain: an unnamed task appears in the log as its own command line, so an incident review reads shell cat /etc/passwd | grep -c bash instead of the reason anybody ran it.

Profiles are the dial, and turning it changes the rule set

ansible-lint ships six named profiles, each extending the one before:

Profile	Adds	Reasonable for
`min`	only the rules that prevent fatal load errors	a repository you have just inherited
`basic`	naming, FQCN, deprecations, schema, `yaml`	the default working target
`moderate`	`name[template]`, `name[casing]`, readability rules	a repo with more than one author
`safety`	`latest`, `package-latest`, `risky-*`	anything that touches production
`shared`	`galaxy`, `meta-*`, `no-changed-when`, `no-handler`	roles other teams consume
`production`	`fqcn`, `sanity`, AAP certification rules	published or certified content

The important mechanic: selecting a lower profile removes rules from the run rather than downgrading them. With --profile min the same role above reports nothing at all:

Read-only / Safethe same nine defects, at --profile min— Executed. Exit code 0. The defects have not been fixed; they have been excluded from evaluation.

$ ansible-lint --offline --profile min -f brief .

Passed: 0 failure(s), 0 warning(s) in 3 files processed of 4 encountered. Profile 'min' was required, but 'production' profile passed.

Read that second sentence carefully, because it is the trap. Nothing about the role changed. ansible-lint evaluated only the rules in the min profile, none of them failed, and it then reported the highest profile satisfied by the rules it actually ran — which is production. A role with nine fatal violations at the default profile has just produced a line containing the word “production” next to the word “passed”.

Lowering the profile is a legitimate thing to do on day one with a large legacy repository: a gate that fails on nine thousand violations is a gate everybody disables. It is not a legitimate thing to leave in place without a plan to raise it, and the plan should be in the repository, not in someone’s head.

A suppression without a reason is a defect the next person inherits

ansible-lint gives you three ways to silence a rule, and they differ in how much they tell the next reader:

# noqa rule-id on the task — narrow, visible at the point of the decision, and the right choice when the exception is genuine.
.ansible-lint-ignore — a file of path rule-id pairs. Useful for bulk debt on an inherited repository, and --generate-ignore writes one for every current violation.
skip_list in the config — repository-wide. This one turns the rule off for code nobody has written yet.

The rule is not which mechanism you use. It is that every suppression carries a sentence saying why, next to the suppression. A bare # noqa no-changed-when says “someone decided this was fine”; it does not say whether they had a reason or were in a hurry, and six months later nobody can tell the difference. # noqa no-changed-when — the vendor CLI is idempotent internally and has no query mode can be checked, argued with, and eventually deleted.

--generate-ignore is a debt ledger, not a fix. Writing one is fine. Writing one and not shrinking it is how a repository ends up lint-clean and unlinted.

Rung 4: the gap the static layers leave open

Return to the playbook with state: presnt and onwer: root. The syntax check passed it. What does the linter say?

Read-only / Safeansible-lint on the same broken playbook— Executed against ansible-lint 26.6.0 with its default configuration. Exit code 2 — but not for either of the two defects that will actually fail.

$ ansible-lint --offline --show-relpath -f brief broken.yml

WARNING  Listing 1 violation(s) that are fatal

# Rule Violation Summary

1 risky-file-permissions profile:safety tags:unpredictability

Failed: 1 failure(s), 0 warning(s) in 1 files processed of 1 encountered.
Last profile that met the validation criteria was 'moderate'. Rating: 2/5 star

risky-file-permissions: File permissions unset or incorrect.
broken.yml:11 Task/Handler: Write the config

It found a real defect — the copy task sets no mode, so the file’s permissions depend on the remote umask — and it said nothing about either misspelling. Module argument validation lives in ansible-lint’s args rule, which is tagged experimental and therefore in the default warn list rather than the failing set.

The two defects are caught at the moment the module runs, and not before:

Read-only / Safethe run finds it immediately— Executed. A read-only stat task with a misspelled option, run against localhost, showing the argument-validation error shape.

$ ansible-playbook typo.yml

TASK [Stat a file with a misspelled option] ************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Unsupported parameters for (ansible.builtin.stat) module: folow. Supported parameters include: checksum_algorithm, follow, get_attributes, get_checksum, get_mime, get_selinux_context, path (attr, attributes, checksum, checksum_algo, dest, mime, mime-type, mime_type, name)."}

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

The error message is excellent. The problem is where it arrives: on a host, mid-run, after the play has already made whatever changes came before it. On a fleet with serial: 25, that is twenty-five hosts into a change nobody wanted to start.

This is the argument for rung 5. A disposable host that the play converges end to end, before the fleet does, moves that error from production into a container that costs nothing to throw away.

Under the hood

The layering is not an arbitrary convention — it follows how much of Ansible each tool is willing to instantiate.

--syntax-check runs the playbook through the parser and the play/task data structures. It never reaches the module.

ansible-lint builds on ansible-compat and can go further: it invokes the syntax check itself as the syntax-check rule, applies JSON Schema validation to files it recognises (schema rule), and inspects task data structurally. Argument validation, though, needs the module’s DOCUMENTATION and its AnsibleModule argument spec, resolved through the same collection path the run will use — which is why the args rule is harder than it looks and is still marked experimental.

At run time, the argument spec is enforced inside the module process, on the target, by AnsibleModule.__init__. That is the earliest point at which anything has both the module code and the resolved parameter values in the same place. No static tool has both, and no static tool ever fully will.

Where a given defect belongs

The discipline is to push each defect class down to the cheapest rung that can catch it, and to be honest about the ones that cannot be pushed.

Defect	Lowest rung that catches it
tab in the indentation	1 — the file will not parse
duplicate key in `group_vars`	2 — `yamllint`, `key-duplicates`
`mode: 644` meaning `01204`	3 — `risky-octal`
`shell` used where `systemd` exists	3 — `command-instead-of-module`
`state: presnt`	5 — the module rejects it when it runs
role is not idempotent	5 — the second converge reports `changed`
service task succeeds but service is dead	5, only if the scenario verifies the outcome
role assumes systemd as PID 1	6 — a VM, because the container never had one
config reload drops in-flight requests	7 — real traffic
the change is correct but the wrong hosts were targeted	7 — and only if someone is watching

The last three rows are the shape of this part. Every layer below them is worth building, and none of them reaches the incident.