Lookups execute on the controller · Ansible

lookup('file', '/etc/passwd') reads the controller’s /etc/passwd. Not the managed node’s. Every time, for every host in the play, regardless of what the play targets.

Part VIII said this in passing while explaining action plugins. It is repeated here, in a lesson of its own, because saying it does not work. People read it, agree with it, and then write a play that assumes the opposite six months later — because the expression sits inside a task that is targeting a remote host, and everything about how it is written suggests it happens there.

So this lesson does not assert it. It proves it.

The demonstration

An inventory of two hosts in the documentation range 192.0.2.0/24. These addresses are reserved for documentation and are not routable — nothing answers on them, by design.

Read-only / Safeinv.ini

[web]
web01.example.com ansible_host=192.0.2.11
web02.example.com ansible_host=192.0.2.12

A file on the controller, containing a string that identifies it:

Read-only / Safemotd.txt, on the controller

controller-copy

And a play with two tasks. The first uses a lookup. The second uses slurp, which is the module for reading a file from a managed node:

Read-only / Safeproof.yml

- name: Prove the hosts are unreachable
hosts: web
gather_facts: false
tasks:
  - name: A lookup, which needs no connection
    ansible.builtin.debug:
      msg: "{{ lookup('ansible.builtin.file', 'motd.txt') }}"

  - name: A slurp, which does
    ansible.builtin.slurp:
      src: /etc/motd
    register: remote_motd

  - name: What the target actually had
    ansible.builtin.debug:
      msg: "{{ remote_motd.content | b64decode }}"

Read-only / Saferun it— Runs the play against two hosts in the documentation range. The lookup task succeeds; the slurp task cannot connect.

$ ansible-playbook -i inv.ini proof.yml

TASK [A lookup, which needs no connection] *************************************
ok: [web01.example.com] => {
  "msg": "controller-copy"
}
ok: [web02.example.com] => {
  "msg": "controller-copy"
}

TASK [A slurp, which does] *****************************************************
fatal: [web01.example.com]: UNREACHABLE! => {"changed": false, "msg": "Task failed: Failed to connect to the host via ssh: ssh: connect to host 192.0.2.11 port 22: Connection timed out", "unreachable": true}
fatal: [web02.example.com]: UNREACHABLE! => {"changed": false, "msg": "Task failed: Failed to connect to the host via ssh: ssh: connect to host 192.0.2.12 port 22: Connection timed out", "unreachable": true}

PLAY RECAP *********************************************************************
web01.example.com          : ok=1    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
web02.example.com          : ok=1    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

Read those two task results together, because the pair is the argument:

The lookup task reported ok for both hosts and returned controller-copy — the content of a file on the machine running ansible-playbook.
The very next task, against the very same hosts, could not open a TCP connection to either of them.

The hosts do not exist. There is no SSH daemon, no route, nothing. And the lookup still produced data, because it never tried to go there.

Note the second detail, which matters as much: both hosts returned the identical string. A lookup does not vary per host, because the per-host thing — the host — is not involved.

The failure this causes deploys the wrong file everywhere and looks like it worked

The classic version:

- name: Deploy the app config
  ansible.builtin.copy:
    content: "{{ lookup('ansible.builtin.file', '/opt/app/config.json') }}"
    dest: /opt/app/config.json

The author’s intent was “read the existing config on each host and write it back with a change”. What it does is read /opt/app/config.json on the controller and write that same content to every host in the play.

Every symptom points away from the cause:

The task reports changed on the hosts where the content differed, which reads as “it worked and there was drift”.
All hosts end up with identical content, which reads as “convergence achieved”.
The content is plausible — it came from a real config file, just the wrong machine’s.
On the controller, where somebody will inevitably test it, it is correct, because there the controller’s copy is the target’s copy.

The signature to recognise: every host got identical content, and the content matches one particular machine. That machine is the controller.

Which one do you need?

You want to read	On the controller	On the target
A file’s contents	`lookup('ansible.builtin.file', path)`	`ansible.builtin.slurp` + `b64decode`, or `ansible.builtin.fetch`
A file’s metadata	`lookup('ansible.builtin.fileglob', pattern)`	`ansible.builtin.stat`
Command output	`lookup('ansible.builtin.pipe', cmd)`	`ansible.builtin.command` + `register`
An environment variable	`lookup('ansible.builtin.env', name)`	`ansible_env` fact, after fact gathering
Lines of a file	`lookup('ansible.builtin.lines', cmd)`	`slurp` then split, or `command`

The remote form is always a module plus register, and that is the tell. A module executes on the target; a lookup does not. If your expression has no module in it, it did not go anywhere.

Read-only / Safethe remote read, written correctly

- name: Read the config from the target
ansible.builtin.slurp:
  src: /opt/app/config.json
register: target_config

- name: Use it
ansible.builtin.debug:
  msg: "{{ target_config.content | b64decode | from_json }}"

slurp returns base64 — always, for every file, because it has to survive JSON transport intact — so b64decode is not optional. That mandatory decode step is a useful marker in review: an expression with b64decode in it read the target; one without it did not.

Reviewing for this

Two questions on any expression containing a lookup:

1. Where does this data live? Repository, controller environment, controller-reachable service — a lookup is right. On the managed node — you need a module.

2. Would identical output on every host be correct? A lookup produces the same value for every host in the play unless one of its arguments is templated from a host variable. If the requirement is per-host data, the expression is wrong.

The second question catches this one, which is subtler because it looks per-host:

Read-only / Safetemplated argument: still a controller-side read

# The path varies per host. The read does not: this opens
# /opt/configs/web01.example.com.json ON THE CONTROLLER.
- ansible.builtin.debug:
  msg: "{{ lookup('ansible.builtin.file', '/opt/configs/' + inventory_hostname + '.json') }}"

The argument is templated per host, so the output differs per host, so it looks like a remote read. It is not. It reads a controller-directory of per-host files — which is a perfectly good pattern when that is what you meant, and a confusing bug when it is not.

Why it works this way, and why it cannot work otherwise

Templating happens on the controller. Every {{ }} in a task is resolved into a concrete value before anything is sent anywhere, because what gets sent is the resolved module arguments — the module on the far end receives JSON, not Jinja.

A lookup is a plugin invoked during that templating pass. It runs in the ansible-playbook process, in Python, with the controller’s filesystem, environment and network. There is no connection open at that point for many tasks, and for the first task of a play there may never have been one.

Which is exactly what the demonstration showed: the lookup completed for two hosts that could not be contacted at all.

Could it be otherwise? Not without changing what templating is. To evaluate a lookup on the target, Ansible would have to ship the un-rendered expression to the host, along with the Jinja environment, every variable in scope, and the lookup plugin itself, then execute it there — which is a different execution model, and one where a templating error becomes a remote failure and secrets in scope go over the wire.

The current design keeps expression evaluation on the machine you control and sends only concrete values. slurp and fetch exist because the other thing is also sometimes needed, and they are modules precisely because modules are the things that run over there.

What happens if the file exists on the targets but not the controller?

You get an error that names a path which demonstrably exists — on the machine you are looking at.

An unhandled exception occurred while running the lookup plugin
'ansible.builtin.file'. Error was a <class 'ansible.errors.AnsibleError'>,
original message: could not locate file in lookup: /opt/app/config.json

The diagnosis goes wrong because the obvious check confirms the wrong thing. You SSH to web01, run ls -l /opt/app/config.json, see it there, and conclude Ansible is broken or that permissions are at fault. People then add become: true to the task, which changes nothing, because the read is not happening on that host at all.

The check that resolves it is ls on the controller. If the file is absent there, the message was accurate and the expression was wrong.

The failure is at least loud. Its silent sibling — the file existing on both machines with different content — is the one from the callout above, and it does not error at all.

The demonstration

Which one do you need?

Reviewing for this

Knowledge check