Finding out which variable won · Ansible

A template rendered 8080 where you expected 8443. Twenty-two places in the repository could have set that value, and the precedence order has twenty-two entries.

The instinct is to read the precedence table. The faster route is to ask the host what it resolved and work backwards, because the answer to “what is the value” is one task away and the answer to “which of these twenty-two files applies here” requires knowing which files exist for this host in the first place.

`var` and `msg` are not interchangeable

debug takes either. They behave differently, and the difference trips people every time they switch between them.

var: takes a variable name. No braces. Ansible looks the name up and prints the value with its type preserved.
msg: takes a string, which you template. Everything becomes text.

Read-only / Safevar, and the templated-var mistake — executed on ansible-core 2.21.3— Four debug tasks against a local connection. The second one is the mistake, and on 2.21.3 it fails outright rather than doing something confusing. Nothing on the machine is modified.

$ ansible-playbook -i inv.ini dv.yml

TASK [var takes a name, not an expression] *************************************
ok: [lh] => {
  "app_port": 8443
}

TASK [var with a templated value is a common mistake] **************************
fatal: [lh]: FAILED! => {"msg": "argument 'var' is of type int and we were unable
to convert to _check_type_str_no_conversion: '8443' is not a string and conversion
is not allowed"}

var: "{{ app_port }}" templates first, so var receives the value 8443 and then tries to look up a variable called 8443. On 2.21.3 that fails with a type error rather than silently printing something unhelpful, which is a genuine improvement — the same mistake on older releases produced output that looked almost right.

Use var: when you want to see structure and type. Use msg: when you want a sentence. If you find yourself writing var: "{{ ... }}", you wanted msg:.

The four-step trace

The procedure, in the order that resolves the most cases fastest.

1. Ask the host what it resolved

Read-only / Safeone host, one variable— An ad-hoc debug against a single host. debug is a controller-side action plugin, so this works even for a host that is unreachable, and it changes nothing either way.

ansible -i inventory/prod app-047.example.com \
-m ansible.builtin.debug -a 'var=app_port'

2. Compare it across the group

If one host disagrees with the others, the source is host-specific. If they all agree and all disagree with what you expected, the source is higher up.

Read-only / Safethe same variable across every host in the group— One line of output per host, easy to sort and count. Read-only.

ansible -i inventory/prod appservers \
-m ansible.builtin.debug -a 'var=app_port' -o \
| sed -E 's/.*"app_port": (.*)$/\1/' | sort | uniq -c

3. Ask which groups the host is in

Read-only / Safegroup membership and the group_vars files it implies— group_names lists the groups the current host belongs to, which is the list of group_vars files that could have contributed. Controller-side; changes nothing.

- name: What could have set this
hosts: app-047.example.com
gather_facts: false
tasks:
  - name: The groups this host is in
    ansible.builtin.debug:
      var: group_names

  - name: The inventory files that were parsed
    ansible.builtin.debug:
      var: ansible_inventory_sources

group_names is the shortlist. A host in all, appservers, prod and eu_west can only have picked up a group variable from one of those four group_vars files, and the precedence between them is a much smaller question than the full table.

4. Look at everything, once

When the shortlist does not explain it, dump the lot and read.

Read-only / Safeevery variable the host resolved, written where you can grep it— hostvars for a single host is large — often several hundred keys with facts gathered. Writing it to a file on the controller and grepping is faster than scrolling. The copy is delegated to localhost, so nothing is written on the managed node.

- name: Dump one host resolved variables
hosts: app-047.example.com
gather_facts: true
tasks:
  - name: Write the resolved variable set to the controller
    ansible.builtin.copy:
      content: "{{ hostvars[inventory_hostname] | to_nice_yaml }}"
      dest: './artifacts/{{ inventory_hostname }}-vars.yaml'
      mode: '0600'
    delegate_to: localhost

Mode 0600 is not decoration. That dump contains every variable the host resolved, which includes anything Vault decrypted for the play.

`ansible_facts` versus the injected `ansible_` variables

Two names for the same data, and the difference matters when you are reading someone else’s play.

Read-only / Safefact injection is on by default — read from ansible-config on 2.21.3— ansible-config list prints the shipped default and the documentation for each setting. Nothing executes against a host.

$ ansible-config list | grep -A 8 '^INJECT_FACTS_AS_VARS'

INJECT_FACTS_AS_VARS:
default: true
description:
- Facts are available inside the `ansible_facts` variable, this setting also pushes
  them as their own vars in the main namespace.
- Unlike inside the `ansible_facts` dictionary where the prefix `ansible_` is removed
  from fact names, these will have the exact names that are returned by the module.

So ansible_facts['os_family'] and ansible_os_family are the same fact, reached two ways. The dictionary form is canonical; the injected form exists for compatibility and is what most older material uses.

Three consequences for debugging:

The injected names live in the ordinary variable namespace, so an inventory variable called ansible_os_family collides with the fact. The ansible_facts form does not have that problem, which is the argument for preferring it.

A play written against injected names breaks if injection is turned off. Estates do turn it off, precisely to avoid the collision above. If a play works on one controller and not another, INJECT_FACTS_AS_VARS belongs on the list of things to compare.

Not everything named ansible_ is a fact. ansible_host, ansible_user, ansible_port and ansible_connection are connection variables set from the inventory, not gathered from the host. They will not appear in ansible_facts, and looking for them there is a common dead end.

`ansible-config dump --only-changed` is the first question

Two hundred and nineteen configuration settings exist. About six of them differ on your controller, and those six explain a large share of “it behaves differently here”.

Read-only / Safeonly the settings that are not at their default — executed on 2.21.3— Each line names the setting, where the value came from, and the value. Reads configuration only.

$ ANSIBLE_FORKS=25 ANSIBLE_HOST_KEY_CHECKING=False ansible-config dump --only-changed

CONFIG_FILE() = None
DEFAULT_FORKS(env: ANSIBLE_FORKS) = 25
HOST_KEY_CHECKING(env: ANSIBLE_HOST_KEY_CHECKING) = False

The parenthetical is the valuable part. (env: ANSIBLE_FORKS) says the value came from the environment, not from ansible.cfg — so it is set in a shell profile, a CI job definition or a wrapper script, and it will not be found by reading the repository.

Under the hood

Why is there no tool that prints where a variable came from?

Because by the time a variable has a value, the information has been discarded. Variable resolution builds a dictionary by merging sources in precedence order — inventory, group vars, host vars, play vars, role defaults, facts, set_fact, extra-vars — and each merge overwrites the key. The final dictionary holds values, not provenance.

ansible-inventory --host can attribute the inventory-level sources because it re-runs that stage in isolation and can report what each plugin produced. Nothing equivalent exists for the play-level layers, because they are constructed during execution and several of them (set_fact, registered results, facts) do not exist until tasks have already run.

This is why the practical procedure is the one in this lesson — establish the value, then narrow the candidate sources by group membership, then read — rather than a single command. It is also a design argument for the variable-layering discipline the course teaches earlier: an estate where a given variable is only ever set in one layer does not need provenance, because the layer is the answer.

What happens if the variable is defined but the play still uses a default?

Then the definition is not visible from where it is being read, and there are three usual reasons.

Scope. A variable set with set_fact in one play is available to later plays for that host, but a variable in a play-level vars block belongs to that play only. Role defaults are the lowest precedence of anything, so almost any other definition beats them — which is what they are for, and also why a value set in defaults/main.yml and expected to apply everywhere frequently does not.

Timing. A variable used in hosts: or in a vars_files path is evaluated before the play begins, so it cannot come from a set_fact in the same play or from a fact gathered during it.

Name. app_port and app_Port are different variables, and so are app_port and app-port — the second is not a valid Jinja identifier and must be reached via vars['app-port'], which is a good reason not to create it.

Print the variable with var: immediately before the task that uses it. If it prints correctly there and the task still behaves as though it is unset, the problem is in the task rather than in the variable, and the next lesson on templating errors is where to go.

Knowledge check

Why this matters

Variable problems produce the most disproportionate debugging sessions in Ansible, because the symptom appears a long way from the cause. A service listens on the wrong port; the port came from a template; the template read a variable; the variable was set in a group_vars file for a group the host joined through a children relationship somebody added last month.

Every step of that chain is simple. Finding the chain is not, and the reason is that no tool reports provenance — the value exists, the history does not.

So the skill is a search strategy rather than a command. Establish the value, compare across hosts to find out whether it is host-specific, reduce the candidate sources by group membership, then read. Four steps, each cheap, in an order that discards most of the search space early. The alternative — reading the precedence table and then reading the repository — is how a five-minute question becomes an afternoon.