A layering rule you can hold in your head

This is the prescriptive lesson of the part. Everything before it described how Ansible behaves; this one says what to do about it.

The goal is a specific, testable property:

A new engineer, given the repository and no explanation, can predict what any variable will resolve to on any host — without consulting the precedence table.

If that holds, precedence never arbitrates anything, and the incidents this part exists to prevent cannot occur. If it does not hold, you have an estate whose behaviour is only discoverable by running it, and you will eventually discover it in production.

Three layers get you there.

The three layers

Everything else — play vars, vars_files, role params, set_fact, include_vars, -e — is an escape hatch. Not forbidden, but each use needs a reason that survives being asked about in review.

That is the whole rule. Three layers, three owners, three review paths, and a default answer of “no” for everything else.

Layer 1: role defaults are the contract

Every variable a role reads has exactly one default, in that role’s defaults/main.yml. No exceptions, including variables you expect to always be overridden.

# roles/webapp/defaults/main.yml - the complete interface
webapp_listen_port: 80
webapp_worker_count: 1
webapp_log_level: info

The reason to list every variable, even ones that are always set elsewhere, is that this file is the role’s documentation. A reader who wants to know what the role can be configured with reads one file. A variable that has no entry here is a variable that will fail at templating time on the one host where nobody remembered to set it.

Defaults should be conservative. webapp_worker_count: 1 is the right default; it is safe everywhere and obviously not a production value. A default of 16 is a production value that will silently apply on hosts that were never configured, which is exactly the failure mode a default is supposed to prevent.

Layer 2: group_vars carries policy

This is where the overwhelming majority of an estate’s configuration lives, keyed by group:

inventory/
├── production.ini
└── group_vars/
    ├── all.yml           estate-wide baselines
    ├── prod.yml          production policy
    ├── staging.yml       staging policy
    ├── web.yml           what it means to be a web server
    └── db.yml            what it means to be a database server

Two dimensions, and keeping them separate is what makes the layout readable: environment groups (prod, staging) and function groups (web, db). A host belongs to one of each. prod.yml holds things that are true because it is production; web.yml holds things that are true because it serves HTTP.

When both dimensions set the same variable you have a genuine design question, and the merge rules below decide it — which is precisely why you want to avoid that situation rather than resolve it.

Layer 3: host_vars records exceptions

One file per host, and each one is a statement that this machine differs from its groups:

# inventory/host_vars/web-02.example.com.yml
# web-02 runs on the older 2-core hardware pending replacement in Q4.
# Remove this file when the host is rebuilt.  See CHG-1184.
webapp_worker_count: 2

The comment is not decoration. A host_vars file without a stated reason and a removal condition becomes permanent, and an estate accumulates them until no host matches its group policy any more. That is how snowflakes form under configuration management.

If you find yourself writing the same host_vars file for several hosts, you have discovered a group. Create it.

How groups actually merge

The layout above only works if group merging is predictable. It mostly is, and the parts that are not are worth knowing exactly. Each of the following was established by execution on ansible-core 2.21.3.

A child group beats its parent. With prod as a parent of web, and both defining the variable, the child wins:

$ ansible-playbook play.yml

all + prod(parent) + web(child) all define it   -> from_web_CHILD
only all + prod(parent)                        -> from_prod_PARENT

This is the behaviour you want and it matches the documentation: “A child group’s variables have higher precedence (they override) than a parent group’s variables.” It makes prod a sensible place for broad policy and web a sensible place to specialise it.

Groups at the same depth merge alphabetically, and the last one wins. This is the rule that makes layouts unpredictable:

$ ansible-playbook play.yml

alpha + mike + zulu, host listed alpha-first in the ini  -> from_zulu
alpha + mike only                                       -> from_mike
mike + zulu only                                        -> from_zulu
same two groups, ini order reversed (zulu block first)  -> from_zulu

Note the last line. Reversing the order of the groups in the inventory file changed nothing — the merge is by group name, alphabetically, not by the order you wrote them. Upstream: “By default, Ansible merges groups at the same parent/child level in alphabetical order. Variables from the last group that Ansible loads overwrite variables from the previous groups.”

ansible_group_priority, and the trap in it

There is a control for the sibling case. ansible_group_priority overrides the alphabetical sort for groups at the same level; it defaults to 1, and a larger number means the group is merged later and therefore wins.

$ ansible-playbook play.yml

alpha priority=10 (set in the ini), zulu default priority=1  -> from_alpha

That works. This does not:

$ ansible-playbook play.yml

alpha priority=10 set in group_vars/alpha.yml, not the ini  -> from_zulu

Identical intent, no error, no warning, opposite outcome. Upstream states the constraint: “You can set ansible_group_priority only in an inventory source, not in group_vars/. Ansible uses this variable when it loads the group_vars/ directory.”

The reason is a bootstrapping one — the priority decides the order in which group_vars/ files are read, so it has to be known before any of them are opened. A priority written inside one of those files is read too late to affect anything, including itself.

One more result worth recording, because it is the question people ask next: priority does not beat depth. A parent group with priority 10 still loses to a child group with the default priority of 1. Priority reorders groups within a level, after the parent/child relationship has already been resolved — which is exactly what the documentation says it does, and it is good to have confirmed it.

One home for group_vars

Lesson 3 established that group_vars/ is loaded from two places — beside the inventory and beside the playbook — and that the playbook copy wins.

The rule that follows is short: keep group_vars/ and host_vars/ beside the inventory, and nowhere else.

estate/
├── ansible.cfg
├── site.yml
├── inventory/
│   ├── production.ini
│   ├── group_vars/          <- the only group_vars in the repository
│   └── host_vars/           <- the only host_vars in the repository
└── roles/

This costs nothing and removes an entire class of incident. It also makes the layout survive a playbook being moved into a subdirectory, which otherwise silently drops the playbook-adjacent layer.

Enforce it with a one-line check in CI, because the failure mode is a directory someone adds in good faith:

# fail if a group_vars or host_vars directory exists outside inventory/
find . -type d \( -name group_vars -o -name host_vars \) \
  -not -path './inventory/*' -print -quit | grep -q . && exit 1
exit 0

The test

Before merging a change to the variable layout, ask someone who did not write it to predict a value. Give them a host and a variable name, and let them read the repository.

If they can answer from the layer rule — check the role default, check the host’s groups, check host_vars — the layout is sound. If they have to open the precedence table, work out whether something is inventory-adjacent or playbook-adjacent, or ask how the scheduled job is invoked, then the layout has already failed, and it will fail again during an incident when the person reading it is tired and the stakes are higher.

That test is the whole point of this lesson. It is cheap, it is objective, and it catches the problem while it is still a pull request.

Knowledge check