Finding out what a variable actually is

There is no command that tells you where a variable value came from.

That sentence is the whole lesson. Ansible will tell you what a variable resolved to on a given host, quickly and precisely. It will not tell you which of the twenty-two sources supplied it, because — as the previous lesson’s Under the hood section explained — the provenance is applied during the merge and then discarded. Nothing stores it.

So diagnosis is a process of elimination, not a lookup. You establish what the value is, you establish what each layer contributes, and you find the responsible source by narrowing. The sequence below does that in four steps, cheapest first, and every step is read-only.

Step 1: find the definitions

Covered in lesson 1, repeated here because skipping it is the commonest mistake. Before running any Ansible command, find out how many candidate definitions exist:

$ grep -rn 'webapp_worker_count' --include='*.yml' .

./inventory/group_vars/all.yml:2:webapp_worker_count: 4
./inventory/group_vars/prod.yml:1:webapp_worker_count: 16
./inventory/host_vars/web-02.example.com.yml:1:webapp_worker_count: 2
./roles/webapp/defaults/main.yml:2:webapp_worker_count: 1

Four candidates. If this returns one result, you are usually finished already: there is nothing for precedence to arbitrate, and your problem is elsewhere — a typo in the name, a when: that skipped the task, or a template that never used the value.

Widen the pattern if you find nothing. A variable can be constructed rather than written ("{{ prefix }}_count"), set from a loop, or built by set_fact, none of which a literal grep for the final name will catch.

Step 2: ask the inventory

ansible-inventory --host renders exactly what the inventory contributes for one host, with the group merge already resolved:

$ ansible-inventory --host web-02.example.com

{
  "ansible_connection": "local",
  "webapp_listen_port": 443,
  "webapp_log_level": "warn",
  "webapp_worker_count": 2
}

The group view is often more useful, because it shows the whole hierarchy and which level each value came from:

$ ansible-inventory --graph --vars

@all:
|--@ungrouped:
|--@prod:
|  |--@web:
|  |  |--web-01.example.com
|  |  |  |--{webapp_listen_port = 443}
|  |  |  |--{webapp_log_level = warn}
|  |  |  |--{webapp_worker_count = 16}
|  |  |--web-02.example.com
|  |  |  |--{webapp_listen_port = 443}
|  |  |  |--{webapp_log_level = warn}
|  |  |  |--{webapp_worker_count = 2}
|  |  |--{webapp_listen_port = 443}
|  |--{webapp_log_level = warn}
|  |--{webapp_worker_count = 16}
|--{webapp_listen_port = 8080}
|--{webapp_worker_count = 4}

That output is genuinely diagnostic. Read it from the bottom: all supplies webapp_worker_count = 4, the prod group raises it to 16, and web-02 alone drops it to 2. You can see the layering, which is as close to provenance as the tooling gets.

Step 3: ask a host what it resolved

This is the authoritative answer for inventory-plus-facts, and it is one line:

$ ansible web-02.example.com -m ansible.builtin.debug -a 'var=webapp_worker_count'

web-02.example.com | SUCCESS => {
  "webapp_worker_count": 2
}

Two things about the var= form are worth knowing before you rely on it.

First, it takes a name, not a template. Write var=webapp_worker_count, not var={{ webapp_worker_count }}. The braces form works by accident in some cases and produces confusing output in others.

Second, an undefined variable does not fail:

$ ansible web-02.example.com -m ansible.builtin.debug -a 'var=webapp_typo_name'

[WARNING]: Encountered 1 template error.
error 1 - 'webapp_typo_name' is undefined
Origin: <CLI option '-a'>

webapp_typo_name

web-02.example.com | SUCCESS => {
  "webapp_typo_name": "<< error 1 - 'webapp_typo_name' is undefined >>"
}

SUCCESS, exit code 0, and the failure is a string inside the result. If you are checking a variable from a script, test the value, not the exit code.

Step 4: look at everything the host resolved

When you do not know which variable is wrong — the symptom is a wrong rendered config, not a wrong named value — dump the host’s whole resolved set. hostvars[inventory_hostname] is that set, and it is large, so filter it:

- name: inspect a host's resolved variables
  hosts: web-02.example.com
  gather_facts: false
  tasks:
    - name: every webapp_ variable this host resolved
      ansible.builtin.debug:
        msg: >-
          {{ hostvars[inventory_hostname] | dict2items
             | selectattr('key', 'search', '^webapp_') | items2dict }}

$ ansible-playbook inspect.yml

TASK [every webapp_ variable this host resolved] ********************************
ok: [web-02.example.com] => {
  "msg": {
      "webapp_listen_port": 443,
      "webapp_log_level": "warn",
      "webapp_worker_count": 2
  }
}

Placement matters. Put this task where the problem is: inside the role if the role is misbehaving, after the set_fact if a fact is suspected, in pre_tasks if you want the state before any role runs. The value changes as the play proceeds, so a dump from the wrong place answers a question you did not ask.

Two related patterns are worth knowing. hostvars['other-host'] reads another host’s variables, which is how a load-balancer play learns the addresses of its backends. And -vvv on any run prints the full task arguments after templating, which shows what a module actually received rather than what the YAML said.

The trap: the tools do not all stand in the same place

This one is worth a section of its own, because it makes two of the four steps above lie to you, and it does so silently.

“Playbook group_vars/” — entries 5, 7 and 10 of the precedence table — means the directory containing the playbook file. Not the repository root, and not your shell’s working directory.

But ansible-inventory and ad-hoc ansible do not have a playbook, so they use the current working directory instead. Run them from somewhere else and they load a different set of files.

Consider a repository with the variable defined twice, once beside the inventory and once beside the playbook at the repository root:

estate/
├── ansible.cfg
├── group_vars/
│   └── db.yml          db_primary_host: db-staging-01.example.com
├── inventory/
│   ├── production.ini
│   └── group_vars/
│       └── db.yml      db_primary_host: db-prod-01.example.com
└── site.yml

The same command gives two different answers depending on where you stand:

$ ansible-inventory -i inventory/production.ini --host db-prod-01.example.com

    "db_primary_host": "db-staging-01.example.com"

$ cd /tmp && ansible-inventory -i /srv/estate/inventory/production.ini --host db-prod-01.example.com

    "db_primary_host": "db-prod-01.example.com"

ansible-playbook is unaffected by either — it uses the playbook’s own directory, and reports db-staging-01.example.com from both locations.

The consequences are worth stating plainly:

• Diagnose from the same directory the playbook is run from. If a scheduled job runs ansible-playbook /srv/estate/site.yml, the relevant directory is /srv/estate.
• A playbook moved one directory deeper silently changes which variables apply. Moving site.yml into playbooks/ makes the repository-root group_vars/ stop applying entirely. This was confirmed by execution: the value fell back to the inventory copy with no warning of any kind. The same rule governs roles/, which at least fails loudly with “the role was not found”.
• The cleanest defence is to have no playbook-adjacent group_vars/ at all. Keep them beside the inventory, and the whole class of problem disappears — including the incident in lesson 8.

The sequence, condensed

1. grep -rn '<name>' --include='*.yml' . — how many definitions exist?
2. ansible-inventory --graph --vars — what does the inventory layer contribute?
3. ansible <host> -m ansible.builtin.debug -a 'var=<name>' — what does the host resolve?
4. A debug of filtered hostvars at the point of failure — what does the play see there?

Run all four from the directory the playbook runs from. Every one is read-only, and the whole sequence takes under two minutes on an estate you have never seen before.

Knowledge check