low riskinformational~35 min

Runbook: Troubleshoot variable precedence

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The variable name is exact, including any prefix - a near-miss name is a different variable and resolves independently
· The host and the play where the wrong value appears are both identified; precedence is per host and per play
· The value that was expected and the value that appeared are both written down
· It is established whether the run used any -e extra variables, because those beat everything below them

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Measure what the task actually sees, at the point in the play where it matters
2Measure what inventory alone resolves, and note that this is a different question
3Enumerate every place in the repository that sets this name
4Compare the two measurements: if they differ, the value comes from a non-inventory layer
5Walk the candidate layers in precedence order from the top, stopping at the first that sets it
6Check for the near-miss cases: a similar name, a group both hosts are in, a role default with no prefix
7Confirm the diagnosis by removing or changing the suspected source and re-measuring
8Fix by consolidating to one layer rather than by adding an override at a higher one

4 · Verification

Confirm the procedure actually fixed the problem.

✓A debug task at the point of use reports the expected value on the affected host
✓The same measurement on an unaffected host still reports its own correct value - the fix did not flatten everyone
✓Exactly one layer in the repository now sets this variable for this host, or the layering is deliberate and documented
✓A check-mode run of the real playbook renders the intended configuration

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Measurement changes nothing; only the fix needs rolling back
↶If the fix was a repository change, git revert restores the previous resolution
↶If the fix was an -e override applied to a live run, remove it - an extra variable is not a fix, it is a way of not finding the cause
↶If a variable was deleted from a layer, confirm no other host depended on it before considering the change complete
↶Re-measure on a sampled host from every affected group after any rollback

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the repository owner if the same variable is set in three or more layers - that is a design problem and patching one layer moves the bug rather than fixing it
· Escalate if the wrong value was applied to production hosts before it was noticed; the configuration rollback runbook applies, not this one
· Escalate if a role sets an unprefixed variable that collides across roles - fixing it changes the role interface and needs a release

Ansible has twenty-two variable precedence levels. Nobody holds them in their head, and trying to reason your way from the table to the answer is slower and less reliable than measuring.

This runbook is a measurement procedure. Three commands establish where the value is coming from; the precedence table is only consulted at the end, to confirm what the measurements already told you.

When to use this runbook

A template rendered a value nobody expected.
A variable is correct on one host in a group and wrong on another.
A change to group_vars had no visible effect.
A role behaves differently in two environments with “the same” configuration.

Blast radius

None. Every measurement step is read-only. The fix is a repository change and goes through the normal review path.

The two questions people conflate

Before any command: there are two different questions and two different tools, and mixing them up is most of the confusion in this area.

Question	Tool	Includes
What does inventory resolve for this host?	`ansible-inventory --host`	host_vars, group_vars, inventory-defined vars
What does the task see at this point?	a `debug` task in the play	all of the above plus play vars, vars_files, role defaults and vars, set_fact, registered results, extra vars

ansible-inventory --host answers the narrower question

Verified on 2.21.3. With app_port: 8082 in host_vars and app_port: 8083 in the play’s vars: block:

$ ansible-inventory -i inv.yml --host web01.example.com
{
    "app_port": 8082
}

$ ansible-playbook -i inv.yml prec.yml
    "msg": "app_port=8083"

Both outputs are correct answers to different questions. Using the first to debug the second is how an afternoon disappears - the file you are staring at really does say 8082, and the task really does see 8083.

Measure the task first. Then use ansible-inventory to work out whether the answer came from inventory at all.

Step 1: Measure what the task sees

Read-only / Safedebug at the point of use

- name: What is the effective value here?
ansible.builtin.debug:
  msg: >-
    app_port={{ app_port | default('UNDEFINED') }}
    type={{ app_port | type_debug }}
tags: [always]

Insert it immediately before the task that produced the wrong result - not at the top of the play. Variables change during a play: set_fact runs, include_vars loads, a role’s defaults come into scope when the role starts. A value measured at the top of the play is not necessarily the value the task saw.

Read-only / Safeor without editing the playbook

ansible web01.example.com -m debug -a 'var=app_port' -o
ansible web01.example.com -m debug -a 'var=hostvars[inventory_hostname]' -o | head -40

Note the type_debug in the first snippet. A surprising number of these incidents are not precedence at all but type: port: 8080 is an integer, port: "8080" is a string, and a comparison between them fails. Verified on 2.21.3 - '8080' | type_debug reports str, and when: "'8080' > 80" fails with:

A 'when' expression failed: Error rendering expression:
'>' not supported between instances of 'str' and 'int'

Step 2: Measure what inventory alone says

Read-only / Safeansible-inventory --host

ansible-inventory -i inventories/production --host web01.example.com

# And which groups this host is in, since group_vars follow membership
ansible-inventory -i inventories/production --graph --vars \
| grep -A5 'web01.example.com'

Now compare:

Task sees	Inventory says	Conclusion
Same value	Same value	The value comes from inventory. Step 3 finds which file.
Different	Different	A higher-precedence layer is overriding: play vars, extra vars, set_fact, role vars. Go to Step 4.
Defined	Not present at all	It comes from a role default, a play var, a vars_file, or `set_fact`. Go to Step 4.

Step 3: Enumerate every place that sets the name

Read-only / Safegrep the repository

grep -rn --include='*.yml' --include='*.yaml' -E '^\s*app_port\s*:' . \
--exclude-dir=.git | sort

Read that list against precedence. The layers that matter, high to low, in the form they usually appear in a repository:

extra vars (-e)                       beats everything
task vars
block vars
role vars (roles/<name>/vars/main.yml)
set_fact / registered vars
play vars_files
play vars
host_vars
group_vars/<child group>
group_vars/<parent group>
group_vars/all
role defaults (roles/<name>/defaults/main.yml)   lowest

Verified on 2.21.3, with the same variable set in four places:

group_vars/all.yml            app_port: 8080
group_vars/web.yml            app_port: 8081
host_vars/web01.example.com   app_port: 8082
play vars:                    app_port: 8083

with play vars present   -> 8083
with -e app_port=9999    -> 9999
without play vars        -> 8082   (host_vars beats both group_vars files)

Step 4: Walk the layers from the top

Start at the highest layer and stop at the first one that sets the name. Walking upwards from group_vars means reading five files that are all being overridden.

Read-only / Safewas there an extra var

# The run log, the CI job definition, the scheduler entry - anywhere -e can hide
grep -rn '\-e ' .gitlab-ci.yml Makefile scripts/ 2>/dev/null | grep -i app_port
history | grep -- '-e app_port'

Read-only / Saferole vars versus role defaults

# vars/ is high precedence - it cannot be overridden from inventory
find roles -path '*/vars/main.yml' -exec grep -Hn 'app_port' {} +

# defaults/ is the lowest - it is the role's overridable interface
find roles -path '*/defaults/main.yml' -exec grep -Hn 'app_port' {} +

A tunable in roles/<name>/vars/main.yml is the second most common cause after extra vars, and it is the more confusing one: the operator sets the value in group_vars, the change is reviewed and merged, and nothing happens. vars/ outranks group_vars; defaults/ does not. That asymmetry is the reason defaults/ exists.

Step 5: Check the near misses

Three things that look like precedence and are not:

Read-only / Safenear-miss names

# A similar-but-different name
grep -rn --include='*.yml' -E 'app_port|app_listen_port|port' \
inventories/ roles/ --exclude-dir=.git | grep -v '^Binary'

A near-miss name. app_port and app_listen_port are unrelated variables. The template uses one; everyone has been editing the other.
Group membership, not group_vars. The host is in a group you did not expect, so a group_vars file you were not reading applies. ansible-inventory --graph answers this.
Two groups at the same depth. When a host is in two groups that both set the name and neither is a child of the other, resolution falls back to alphabetical order of the group names. That is defined behaviour and a terrible thing to depend on: renaming a group changes the value.

Read-only / Safewhich groups is this host in

ansible-inventory -i inventories/production --graph \
| grep -B10 'web01.example.com'
ansible web01.example.com -m debug -a 'var=group_names' -o

group_names is the definitive list, resolved, as the task sees it.

Step 6: Confirm by changing one thing

A diagnosis you have not tested is a theory.

Read-only / Safeprove it

# Comment out the suspected source on a branch, then re-measure
ansible web01.example.com -m debug -a 'var=app_port' -o

# Or prove the opposite direction: override at a known-higher layer
ansible web01.example.com -m debug -a 'var=app_port' -e app_port=7777 -o

If removing the suspected line changes the value, you found it. If it does not, the value is coming from somewhere else and the search continues at the next layer down.

Step 7: Fix by consolidating, not by overriding

The tempting fix is to set the value at a higher layer so it wins. That works, and it makes the next occurrence worse: now the variable is set in four places instead of three, and the next person has one more file to find.

Situation	Fix
Same value in several layers	Delete all but one. Choose the layer that matches the value’s scope.
A tunable in `roles/*/vars/`	Move it to `roles/*/defaults/` so inventory can override it
An `-e` in a CI job or scheduler	Move it into `group_vars` for the environment
Two groups at the same depth both set it	Make one a child of the other, or move the value to host_vars
Unprefixed role variable colliding across roles	Prefix it with the role name - this changes the role interface, so it needs a release

Which layer to choose: match the layer to the scope of the value. A value that is true for every host in an environment belongs in that environment’s group_vars/all. A value true for one host belongs in host_vars. A value that is a property of the role belongs in the role’s defaults.

Verification

Read-only / Safeverify the fix

# The affected host now sees the intended value
ansible web01.example.com -m debug -a 'var=app_port' -o

# An unaffected host still sees ITS correct value
ansible db01.example.com -m debug -a 'var=app_port' -o

# Only one place sets it now
grep -rn --include='*.yml' -E '^\s*app_port\s*:' . --exclude-dir=.git

# And the real playbook renders correctly
ansible-playbook -i inventories/production site.yml \
--limit web01.example.com --check --diff

The second check is the one that catches an over-correction. Deleting a group_vars entry to fix one host frequently changes the value for forty others, and check mode against one host will not show it.

Common patterns

Symptom	Likely cause	Resolution
The file says X, the task sees Y	A higher layer sets it - usually `-e` or `roles/*/vars/`	Measure the task, then walk layers from the top
`group_vars` change has no effect	The value is in `roles/*/vars/`, which outranks it	Move it to `defaults/`
Correct on most hosts, wrong on one	That host has a `host_vars` file	`ansible-inventory --host`
Value changes when a group is renamed	Two same-depth groups; alphabetical resolution	Restructure so one is a child, or move the value
Comparison fails though the value looks right	Type, not precedence - quoted number is a string	`type_debug`; quote or unquote deliberately
`ansible-inventory --host` disagrees with reality	It only reports inventory layers	Use a debug task for the full picture
Variable is undefined only inside a role	A role default with a different prefix	Read the role’s `defaults/main.yml`
Everyone agrees on the repository, runs disagree	An `-e` in the invocation	Read the CI job or the scheduler entry

Escalation

Escalate when:

The same variable is set in three or more layers. Patching one moves the bug.
The wrong value already reached production hosts. That is the configuration rollback runbook.
A role sets an unprefixed variable that collides. Fixing it is an interface change and needs a release.

Why this matters

The precedence table is documentation, not a debugging tool. Reading it tells you what could be happening; two commands tell you what is.

The habit worth building is starting with the measurement rather than the theory - a debug task at the point of use, then ansible-inventory --host - because the gap between those two answers is itself the diagnosis. Equal means inventory; different means something above inventory, and there are only a handful of candidates.

The other habit is fixing downwards. Every one of these incidents can be resolved in thirty seconds by setting the value at a higher layer, and every such fix adds one more place for the next person to look. The slow fix - find the layers, delete all but one - is the only one that makes the next occurrence less likely rather than more.