Lab: The same variable in six places — find the winner, then prove it · Lab · Ansible

Objective

By the end of this lab you will have determined the relative precedence of six variable layers by experiment — removing one layer at a time and observing what surfaces — rather than by consulting a table you will misremember under pressure. You will also have found the case where ansible-inventory --host, the obvious diagnostic, reports a value the playbook will not use.

Architecture

One inventory, one host, one variable defined six times.

repo/
├── inventories/prod/
│   ├── hosts.yml
│   ├── group_vars/web.yml       app_port: 8000   (inventory group_vars)
│   └── host_vars/node1.yml      app_port: 8002   (inventory host_vars)
├── group_vars/web.yml           app_port: 8001   (playbook group_vars)
├── host_vars/node1.yml          app_port: 8003   (playbook host_vars)
├── roles/app/
│   ├── defaults/main.yml        app_port: 8004   (role defaults)
│   └── vars/main.yml            app_port: 8005   (role vars)
└── site.yml

The port numbers encode nothing — they are labels, chosen so that the value in the output tells you which file it came from without a second lookup. ansible_connection: local on the group means the lab needs no managed nodes.

Requirements

A controller with ansible-core 2.21.x. All values below were observed on 2.21.3.
No SSH, no credentials, no managed nodes. The play runs debug only.
A place to write predictions before running commands. As with the targeting lab, running first and rationalising afterwards teaches nothing.

Scenario

Production is serving on the wrong port. The team’s group_vars/web.yml clearly says app_port: 8001, three people have read it, and the service is not on 8001. Somebody suggests the file is not being loaded. Somebody else suggests a caching problem. Your job is to find where the value actually comes from, and to build the two-command procedure that answers this question in future without a discussion.

Tasks

Task 1: Build the six-layer repository

WORKDIR="$HOME/ansible-precedence-lab"
mkdir -p "$WORKDIR"/{inventories/prod/{group_vars,host_vars},group_vars,host_vars,roles/app/{defaults,vars,tasks}}
cd "$WORKDIR"

# Confirm nothing is redirecting inventory or roles_path
ansible-config dump --only-changed

Write inventories/prod/hosts.yml:

web:
  hosts:
    node1:
  vars:
    ansible_connection: local

Now seed all six layers. Each value is deliberately distinct:

cd "$HOME/ansible-precedence-lab"

echo 'app_port: 8000' > inventories/prod/group_vars/web.yml
echo 'app_port: 8001' > group_vars/web.yml
echo 'app_port: 8002' > inventories/prod/host_vars/node1.yml
echo 'app_port: 8003' > host_vars/node1.yml
echo 'app_port: 8004' > roles/app/defaults/main.yml
echo 'app_port: 8005' > roles/app/vars/main.yml

Write roles/app/tasks/main.yml:

- name: Report the value the role sees
  ansible.builtin.debug:
    msg: "role sees {{ app_port }}"

And two playbooks. site.yml uses no role, so it isolates the four inventory and playbook layers:

# site.yml
- name: Layers without a role
  hosts: web
  gather_facts: false
  tasks:
    - name: Report
      ansible.builtin.debug:
        msg: "no role: {{ app_port }}"

# site-role.yml
- name: Layers including the role
  hosts: web
  gather_facts: false
  roles:
    - app

Task 2: Predict, then peel

Write down your prediction for each row before running anything. Then run site.yml, delete the layer that won, and run again.

Step	Layers still present	Predicted	Actual
1	all four non-role layers
2	after deleting `host_vars/node1.yml`
3	after deleting `inventories/prod/host_vars/node1.yml`
4	after deleting `group_vars/web.yml`

Run step 1:

Read-only / Safecontroller— A debug-only play. No connection, no change.

$ ansible-playbook -i inventories/prod/hosts.yml site.yml

TASK [Report] ******************************************************************
ok: [node1] => {
  "msg": "no role: 8003"
}

8003 is host_vars/node1.yml — the playbook-adjacent host_vars, not the inventory-adjacent one. Peel it away:

rm host_vars/node1.yml
ansible-playbook -i inventories/prod/hosts.yml site.yml | grep msg

Then the next, and the next:

rm inventories/prod/host_vars/node1.yml
ansible-playbook -i inventories/prod/hosts.yml site.yml | grep msg

rm group_vars/web.yml
ansible-playbook -i inventories/prod/hosts.yml site.yml | grep msg

The observed sequence is 8003, 8002, 8001, 8000, which orders the four layers, highest first:

playbook-adjacent host_vars/
inventory-adjacent host_vars/
playbook-adjacent group_vars/
inventory-adjacent group_vars/

Restore the three files you deleted before continuing:

echo 'app_port: 8001' > group_vars/web.yml
echo 'app_port: 8002' > inventories/prod/host_vars/node1.yml
echo 'app_port: 8003' > host_vars/node1.yml

Task 3: Add the role and watch the answer invert

Run the role-based playbook:

Read-only / Safecontroller— Same inventory, same host, role added

$ ansible-playbook -i inventories/prod/hosts.yml site-role.yml

TASK [app : Report the value the role sees] *************************************
ok: [node1] => {
  "msg": "role sees 8005"
}

8005 is roles/app/vars/main.yml. Every one of the four layers you just ranked has been overridden by a file inside the role.

This is why role vars are almost always wrong

roles/<name>/vars/main.yml sits above inventory, above group_vars, above host_vars and above play vars. A value there cannot be overridden by any of the mechanisms an operator would reach for. The only things that beat it are a role parameter, set_fact, and --extra-vars.

Prove it:

ansible-playbook -i inventories/prod/hosts.yml site-role.yml -e app_port=9999

That reports 9999. Nothing short of the command line got through.

The consequence for role design is blunt: defaults/main.yml is the role’s interface and vars/main.yml is the role’s internals. Anything a consumer might reasonably want to change belongs in defaults, where every normal layer outranks it. A tunable in vars/ is not a tunable — it is a constant that produces a support ticket.

Confirm the two directions of the same fact:

# Role defaults lose to inventory: delete vars/main.yml and re-run
mv roles/app/vars/main.yml /tmp/role-vars-main.yml
ansible-playbook -i inventories/prod/hosts.yml site-role.yml | grep msg

# Put it back
mv /tmp/role-vars-main.yml roles/app/vars/main.yml

With vars/main.yml gone the role reports 8003 — the playbook host_vars — because defaults/main.yml at 8004 is the lowest layer of all and loses to everything.

Task 4: The diagnostic that lies

Now the part that costs people an afternoon. ansible-inventory --host is the obvious way to ask “what value does this host have”. Run it from inside the repository:

Read-only / Safecontroller — cwd is the repo root

$ ansible-inventory -i inventories/prod/hosts.yml --host node1

{
  "ansible_connection": "local",
  "app_port": 8003
}

Now run exactly the same command from one directory up:

Read-only / Safecontroller — cwd is the parent directory— Same inventory, same host, different answer

$ cd .. && ansible-inventory -i ansible-precedence-lab/inventories/prod/hosts.yml --host node1

{
  "ansible_connection": "local",
  "app_port": 8002
}

The fix is to say which playbook directory you mean:

REPO="$HOME/ansible-precedence-lab"

ansible-inventory -i "$REPO/inventories/prod/hosts.yml" \
  --playbook-dir "$REPO" --host node1

That reports 8003 from anywhere.

The two-command procedure

For “what value will this host actually get”, use both and compare:

REPO="$HOME/ansible-precedence-lab"
INVENTORY="$REPO/inventories/prod/hosts.yml"

# What the inventory layers alone provide
ansible-inventory -i "$INVENTORY" --host node1

# What the play will actually see, role layers included
ansible-playbook -i "$INVENTORY" "$REPO/site-role.yml" \
  -e 'ansible_check_mode_note=diagnostic' --check | grep -A2 'Report'

If they disagree, the difference is a playbook-adjacent or role-level layer, and you now know exactly which four files to look in.

Task 5: Find the value in a repository you did not write

The point of the exercise. Have a colleague — or a shell one-liner — hide a seventh definition somewhere, then find it without reading every file:

cd "$HOME/ansible-precedence-lab"

# Every definition of the variable, with its file and line
grep -rn --include='*.yml' --include='*.yaml' 'app_port' . | sort

Read-only / Safecontroller— The whole search space for one variable, in one command

$ grep -rn --include='*.yml' 'app_port' . | sort

./group_vars/web.yml:1:app_port: 8001
./host_vars/node1.yml:1:app_port: 8003
./inventories/prod/group_vars/web.yml:1:app_port: 8000
./inventories/prod/host_vars/node1.yml:1:app_port: 8002
./roles/app/defaults/main.yml:1:app_port: 8004
./roles/app/vars/main.yml:1:app_port: 8005
./roles/app/tasks/main.yml:3:    msg: "role sees {{ app_port }}"

Six definitions and one use. Rank them with the ladder you built in Tasks 2 and 3, and you have the answer without running anything.

What grep will not find

Three sources do not appear in a file grep and must be checked separately:

--extra-vars on the command line or in a CI job definition,
an ANSIBLE_-prefixed environment variable, for the settings that have one,
set_fact results and registered variables, which are computed at runtime.

set_fact also outranks role vars, and unlike --extra-vars it leaves no trace on the command line. Add a pre_tasks block to site-role.yml and re-run:

- name: Layers including the role
  hosts: web
  gather_facts: false
  pre_tasks:
    - name: A fact set earlier in the play
      ansible.builtin.set_fact:
        app_port: 7777
  roles:
    - app

The role reports 7777, not the 8005 from its own vars/main.yml. A set_fact three plays earlier in an imported playbook produces exactly this and appears in no grep for a literal value.

For the runtime sources, the reliable tool is -vvv on the actual run, or a temporary debug: var=app_port task placed immediately before the use.

Task 6: Write your ladder

In ladder.md, write the precedence order you observed, highest first, with the value you saw at each step as evidence. Add one sentence per layer saying what it is for. A layer whose purpose you cannot state in one sentence is a layer your repository should probably not be using.

Validation

ansible-playbook -i inventories/prod/hosts.yml site.yml reports 8003 with all layers present, and 8002, 8001, 8000 as you delete each winner.
ansible-playbook -i inventories/prod/hosts.yml site-role.yml reports 8005.
The same command with -e app_port=9999 reports 9999.
With roles/app/vars/main.yml moved aside, site-role.yml reports 8003, not 8004 — role defaults are the lowest layer, not a middle one.
ansible-inventory -i inventories/prod/hosts.yml --host node1 reports 8003 from the repo root and 8002 from the parent directory.
The same command with --playbook-dir "$HOME/ansible-precedence-lab" reports 8003 regardless of where it is run from.
ladder.md orders six layers with an observed value beside each.

Expected Outcome

ansible-precedence-lab/
├── group_vars/web.yml
├── host_vars/node1.yml
├── inventories/prod/
│   ├── group_vars/web.yml
│   ├── host_vars/node1.yml
│   └── hosts.yml
├── ladder.md
├── roles/app/
│   ├── defaults/main.yml
│   ├── tasks/main.yml
│   └── vars/main.yml
├── site-role.yml
└── site.yml

You can order six precedence layers from memory because you watched each one surface. You know that ansible-inventory --host needs --playbook-dir to tell the truth about a repository, and you know that a variable in roles/*/vars/ cannot be overridden by anything an operator would try first.

Troubleshooting

Every run reports the same value no matter what you delete. An --extra-vars is set somewhere — check your shell history and any wrapper script. Extra vars beat all six layers and produce exactly this symptom.

group_vars/web.yml has no effect at all. The filename must match the group name exactly. web.yml applies to the group web; webs.yml applies to a group called webs, which does not exist, and Ansible does not warn about a group_vars file that matches nothing.

A group_vars directory instead of a file behaves differently. group_vars/web/ containing several files is valid and all of them load, merged in filename order. That is a useful pattern and a confusing one to debug — grep -rn finds them, a glance at the directory listing may not.

Role defaults appear to win. Then the layer you expected to beat them is not being loaded at all. Confirm with ansible-inventory --list --playbook-dir, and check for a typo in the group name.

ansible-inventory --host shows a variable the play does not have. The reverse of Task 4: you ran it from a directory containing an unrelated host_vars/. Always pass --playbook-dir explicitly when diagnosing.

Cleanup

Everything this lab created lives inside one directory and one temporary file. No system configuration was changed, no host was contacted, and no privilege was escalated.

Step 1. Confirm nothing escaped the working directory:

cd "$HOME/ansible-precedence-lab"
ansible --version | grep 'config file'
ls -la /tmp/role-vars-main.yml 2>/dev/null || echo 'no leftover role vars file'

If /tmp/role-vars-main.yml still exists, Task 3 was interrupted before the mv back. Restore it before deleting anything:

[ -f /tmp/role-vars-main.yml ] && \
  mv /tmp/role-vars-main.yml "$HOME/ansible-precedence-lab/roles/app/vars/main.yml"

Step 2. Keep the ladder — it is the deliverable and you will refer to it:

mkdir -p "$HOME/ansible-lab-deliverables"
cp -a "$HOME/ansible-precedence-lab/ladder.md" \
      "$HOME/ansible-lab-deliverables/precedence-ladder.md"

Step 3. Remove the working directory by absolute path:

rm -rf deletes the whole lab directory, including anything you added

The next block removes ~/ansible-precedence-lab and everything under it, recursively and without confirmation. If you saved notes, an inventory you want to keep, or a playbook you extended, copy it out first — the step above does that for the lab’s own deliverables and knows nothing about yours.

The path is written in full and anchored to $HOME deliberately. A relative rm -rf run from the wrong directory is how people delete work they meant to keep, and the shell gives no second chance.

Confirm what you are about to delete before you delete it:

ls -la "$HOME/ansible-precedence-lab"

rm -rf "$HOME/ansible-precedence-lab"

Under the hood

Ansible builds a variable dictionary per host per task, and it builds it by applying sources in a fixed order, each update()ing the dictionary the previous one produced. Precedence is therefore not a comparison — it is simply who writes last. There is no conflict resolution and no warning, because from the code’s point of view nothing conflicted: one dictionary key was assigned several times.

The playbook-versus-inventory distinction comes from two separate VariableManager sources. Inventory-adjacent group_vars/host_vars are found by the inventory plugin, relative to the inventory source. The playbook-adjacent ones are found by the play, relative to basedir — the directory containing the playbook file. Both are loaded, the playbook set is applied later, and that is the entire mechanism behind “playbook beats inventory”.

ansible-inventory instantiates the inventory but has no play, so its basedir defaults to the process working directory. --playbook-dir sets it explicitly, which is why it changes the answer.

Role vars/main.yml is applied near the very end of the sequence, immediately before task-level vars and the runtime sources. That placement is deliberate: vars/ is meant to hold values the role’s own tasks depend on and that a consumer must not accidentally break — OS-specific package names, internal paths, computed constants. It becomes a defect only when it is used for something a consumer legitimately needs to change.

What You Learned

You derived the ladder by peeling layers, and it came out playbook host_vars, inventory host_vars, playbook group_vars, inventory group_vars — with role vars above all four and role defaults below all four.
roles/*/vars/main.yml is unreachable from inventory. Only a role parameter, set_fact or --extra-vars overrides it, which is why everything a consumer might tune belongs in defaults/.
ansible-inventory --host answers differently depending on your working directory. You reproduced 8003-versus-8002 from the same command, and fixed it with --playbook-dir.
A grep -rn for the variable name is a complete search of the file layers, and an incomplete search overall: extra vars, environment and set_fact do not appear in it.
Nothing warns you. Six definitions, one winner, zero diagnostics. The only defence is a repository convention about where variables are allowed to live, and the ladder you just wrote is the argument for it.

Lab: The same variable in six places — find the winner, then prove it

Objectives

Prerequisites