Anatomy of a run, start to finish

This is the spine of the course. Almost every confusion you will meet later — why template reads a file from the controller, why a lookup does not run on the target, why check mode depends on which module you used, why a failed host stops appearing in later tasks — is answered by knowing this sequence and, critically, which machine each step happens on.

The sequence

flowchart TB
  A["1. Locate ansible.cfg<br/>controller"] --> B["2. Parse inventory<br/>controller"]
  B --> C["3. Resolve variables<br/>controller"]
  C --> D["4. Select hosts for the play<br/>controller"]
  D --> E["5. Open connections<br/>controller to node"]
  E --> F["6. Gather facts<br/>runs on node"]
  F --> G["7. For each task"]
  G --> H["7a. Render templates<br/>controller"]
  H --> I["7b. Build module payload<br/>controller"]
  I --> J["7c. Transfer and execute<br/>node"]
  J --> K["7d. Read JSON result<br/>controller"]
  K --> L["7e. Evaluate changed / failed<br/>controller"]
  L --> G
  L --> M["8. Flush handlers<br/>node"]
  M --> N["9. Aggregate the recap<br/>controller"]

Count the stages that happen on the controller: seven of eleven. That ratio is the point of the diagram. Ansible is a controller-side program that occasionally causes something to execute elsewhere, not a distributed system.

Stage by stage

1. Locate the configuration — controller

Before anything else, Ansible finds one ansible.cfg. It checks ANSIBLE_CONFIG, then the current directory, then ~/.ansible.cfg, then /etc/ansible/ansible.cfg, and uses the first one it finds.

The others are not merged. They are ignored entirely. This is the single most common cause of a setting that appears not to apply, and it is why ansible --version prints the config file it selected.

2. Parse the inventory — controller

Inventory sources are read and expanded into hosts, groups, and the implicit all and ungrouped groups. Dynamic inventory plugins run here, on the controller, which means an inventory script that queries a cloud API does so before any managed node is contacted.

The output of this stage is the set of hosts that exist as far as this run is concerned. A host absent here is invisible to everything that follows — not skipped, invisible.

3. Resolve variables — controller

Variables from every source are collected and precedence applied: inventory variables, group and host variable files, role defaults, play variables, --extra-vars, and the rest. An entire part of this course covers the ordering.

What matters architecturally is that this happens on the controller, before connection, using data the controller holds.

4. Select the hosts for the play — controller

The play’s hosts: pattern is applied to the parsed inventory, then --limit narrows it further. The result is the play’s host list — the blast radius, computed and fixed before a single connection opens.

That it is computable before anything happens is the basis of every safety mechanism in the course. You can ask for it without running anything:

$ ansible-playbook -i inv.ini reach.yml --list-hosts

playbook: reach.yml

play #1 (all): Prove reachability requires a connection	TAGS: []
  pattern: ['all']
  hosts (3):
    db01
    web02
    web01

5. Open connections — controller reaching out

The connection plugin — ssh by default — establishes a connection per host, up to forks at a time. The default forks is 5, which surprises people running against three hundred hosts and wondering why it takes so long.

A host that cannot be connected to is marked unreachable and dropped from the remainder of the play. It is not failed; the distinction is tracked separately and reported separately, and Part I’s shell-loop lesson explains why that separation is valuable.

6. Gather facts — executes on the node

Unless gather_facts: false, Ansible runs the setup module on each host. This is a real module execution with the full transfer-and-run cycle, and its result is a large structure of facts about the host — distribution, interfaces, memory, mounts, kernel.

Facts are used by later tasks and by templates. They are gathered afresh every run, because with the default memory cache plugin nothing survives the process.

Fact gathering is the single largest fixed cost of a run against a large fleet, which is why gather_facts: false on plays that do not need facts is a real optimisation and not a micro-optimisation.

7. For each task, in order

Now the loop. With the default linear strategy, every host completes task 1 before any host starts task 2.

That is a deliberate design choice and it has direct operational consequences. A slow host holds up the whole fleet at each step. In exchange, the fleet stays in a consistent state between tasks, which is what makes ordering guarantees meaningful — you know that when task 5 begins, task 4 has finished everywhere.

7a. Render templates — controller

Jinja2 expressions in task arguments are evaluated on the controller, using controller-side variables and facts already gathered. A template task reads its source file from the controller filesystem and renders it in controller memory.

This is the answer to “why does template look for the file on my machine”. It has to: the rendering happens there, and the managed node has no access to your repository.

7b. Build the module payload — controller

The module and everything it imports are assembled into a self-contained Python payload, with the task’s arguments embedded. The next lesson takes this apart in detail.

7c. Transfer and execute — node

The payload is written to a temporary directory on the node and run by the node’s Python interpreter. This is the only stage where your task actually does anything to the host.

7d. Read the JSON result — controller

The module prints a single JSON object to stdout. The controller reads and parses it.

This is the contract, and it is worth stating explicitly: a module communicates with the controller by printing JSON. Everything the controller knows about what happened on that host came through this channel.

7e. Evaluate the result — controller

The controller now decides what the result means:

• Was the task successful? failed_when, if present, overrides the module’s own opinion.
• Did it change anything? changed_when, if present, overrides the module’s changed.
• Should the task have run at all? when was evaluated before the task, on the controller.
• Does anything need notifying? notify queues a handler if the task reported changed.
• Should the value be kept? register stores the parsed object.

Every one of these is controller-side logic applied to a JSON document. The managed node has no idea any of it is happening, which is why changed_when can make a task report unchanged without altering what the task did on the host.

8. Flush handlers — node

At the end of the tasks section — and after pre_tasks and roles individually — queued handlers run. A handler notified by twelve tasks runs once. Handlers run in the order they are defined, not the order they were notified.

Handlers are ordinary tasks, so they go through 7a to 7e like everything else.

9. Aggregate the recap — controller

The controller prints the per-host summary:

$ ansible-playbook -i inv.ini reach.yml

PLAY RECAP *********************************************************************
db01                       : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
web01                      : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
web02                      : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

Read the columns as answers to distinct questions. ok and changed tell you what happened; unreachable tells you which hosts you know nothing about; failed tells you which hosts reported a problem; skipped tells you where a when evaluated false.

The recap is the only durable record the run produces. There is no database — if the output was a terminal that has since closed, this is gone.

Reading a real run

Here is one task, at -vvv, executed against ansible-core 2.21.3. Every line maps to a stage above.

$ ansible-playbook -i localhost, local.yml -vvv

TASK [Ping the local Python] ***************************************************
task path: /home/ops/estate/local.yml:6
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: ops
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/ops/.ansible/tmp `"&& mkdir "` echo /home/ops/.ansible/tmp/ansible-tmp-1786481665.26-4003164-70804194833275 `" && echo ansible-tmp-... )'
<localhost> Attempting python interpreter discovery
<localhost> EXEC /bin/sh -c 'echo FOUND; command -v '"'"'python3.14'"'"'; ... ; echo ENDFOUND'
Using module file /opt/estate/venv/lib/python3.14/site-packages/ansible/modules/ping.py
<localhost> PUT /home/ops/.ansible/tmp/ansible-local-4003157fj1vadaa/tmpoicft4o4 TO /home/ops/.ansible/tmp/ansible-tmp-1786481665.26-4003164-70804194833275/AnsiballZ_ping.py
<localhost> EXEC /bin/sh -c 'chmod u+rwx /home/ops/.ansible/tmp/ansible-tmp-.../ /home/ops/.ansible/tmp/ansible-tmp-.../AnsiballZ_ping.py'
<localhost> EXEC /bin/sh -c '/usr/bin/python3.14 /home/ops/.ansible/tmp/ansible-tmp-.../AnsiballZ_ping.py'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/ops/.ansible/tmp/ansible-tmp-.../ > /dev/null 2>&1'
ok: [localhost] => {
  "ansible_facts": {
      "discovered_interpreter_python": "/usr/bin/python3.14"
  },
  "changed": false,
  "ping": "pong"
}

Six operations for one trivial task:

1. mkdir the temporary directory, with umask 77 so only the remote user can read it.
2. Interpreter discovery, once per host per run.
3. Using module file — the controller selecting the module source from its own filesystem. Stage 7b.
4. PUT — the payload transferred to the node. Stage 7c.
5. chmod u+rwx then execute with the discovered interpreter.
6. rm -f -r — the temporary directory removed. Nothing persists, which is the agentless property from lesson 1, visible in the trace.

Then the JSON result, which the controller parses and evaluates.

Where a failure actually happened

The practical payoff of this lesson is being able to place a failure at a stage. Nearly every diagnosis gets faster once you can.

Note how few of these are module problems. Most failures that present as module failures are stage 5, 6 or 7c problems wearing a module’s name, and the reason they are hard to diagnose without this model is that the error message names the last thing the controller was doing rather than the thing that broke.

Knowledge check