A precedence incident, worked end to end

This lesson is the part’s exam. Everything in it has appeared already; what is new is the order in which you use it under pressure, and the wrong turns that cost time.

The symptom

A routine configuration run completes cleanly against the production database tier. PLAY RECAP shows no failures, no unreachable hosts.

Twenty minutes later the application team reports that the production application servers are connecting to the staging database. The staging database is read-only from production’s network, so the application is up but every write is failing.

The playbook was not changed. The inventory was not changed. The last commit to the repository was four days ago and touched a different role.

What the operator believes

Before diagnosing, notice the belief that will send you the wrong way. The estate has a group_vars/db.yml beside the inventory, and it is correct:

# inventory/group_vars/db.yml
db_primary_host: db-prod-01.example.com
db_port: 5432

An operator who opens that file sees the right value, concludes the variables are fine, and starts looking at DNS, at the application’s own configuration, or at whether someone changed the database. Those investigations can absorb an hour, and all of them are downstream of a value that was already wrong before the template rendered.

The discipline that avoids this is simple and unnatural under pressure: confirm the value on the host before believing any file.

Step 1: confirm what was actually deployed

Do not read a file. Ask a host.

$ ansible-playbook site.yml

TASK [show what the template would receive] ************************************
ok: [db-prod-01.example.com] => {
  "msg": "primary=db-staging-01.example.com port=5432"
}
ok: [db-prod-02.example.com] => {
  "msg": "primary=db-staging-01.example.com port=5432"
}

PLAY RECAP *********************************************************************
db-prod-01.example.com     : ok=1    changed=0    unreachable=0    failed=0
db-prod-02.example.com     : ok=1    changed=0    unreachable=0    failed=0

Two facts established immediately.

db_primary_host really is resolving to the staging host — this is a variable problem, not a DNS or application problem. And db_port is 5432, which is the value from the file that appears correct. So that file is being loaded; it is not a path problem or a group membership problem. One variable from it is being overridden and another is not.

That second observation is what makes this diagnosable in minutes. A file that is not loaded at all and a file that is loaded but overridden look identical if you only check one variable. Always check a second one.

Step 2: find every definition

$ grep -rn 'db_primary_host' --include='*.yml' .

./inventory/group_vars/db.yml:1:db_primary_host: db-prod-01.example.com
./group_vars/db.yml:1:db_primary_host: db-staging-01.example.com

There it is. Two files, both called group_vars/db.yml, in different directories. One beside the inventory, one at the repository root beside the playbook.

The repository looks like this:

estate/
├── ansible.cfg
├── site.yml
├── group_vars/
│   └── db.yml          db_primary_host: db-staging-01.example.com
├── inventory/
│   ├── production.ini
│   └── group_vars/
│       └── db.yml      db_primary_host: db-prod-01.example.com
└── roles/

Step 3: confirm which one wins

The precedence table answers it. Inventory group_vars/* is entry 6; playbook group_vars/* is entry 7. The playbook-adjacent file wins.

Confirm rather than assume, because “which directory counts as the playbook one” is exactly the detail that is easy to get backwards:

$ ansible db-prod-01.example.com -m ansible.builtin.debug -a 'var=db_primary_host'

db-prod-01.example.com | SUCCESS => {
  "db_primary_host": "db-staging-01.example.com"
}

Confirmed. Diagnosis complete, in three commands, none of which changed anything.

How it got there

Worth reconstructing, because the fix depends on it:

git log --oneline -- group_vars/db.yml

The usual history is that someone needed a staging configuration, copied the estate layout from a tutorial that puts group_vars/ at the repository root, and added the file there. It worked in staging — where the value happened to be correct — and it was reviewed by someone who saw a plausible group_vars/db.yml with a plausible staging value in it.

Nothing about the change looked wrong. The reviewer had no way to see that a second file with the same name already existed one directory down, and no tool told them. That is the property that makes this a recurring incident rather than a one-off mistake.

The delay between the merge and the incident is the other characteristic feature. The file was added days or weeks earlier; nothing happened until the next production run of that play.

The fix, in two parts

Immediate. Delete the repository-root group_vars/db.yml and re-run against production. One host first, then the rest:

$ ansible-playbook site.yml --limit db-prod-01.example.com --check --diff

Then the same without --check, then without --limit. The temptation to skip straight to the full tier is strong once you know the cause; resist it, because the diagnosis is a theory until one host confirms it.

Structural. The immediate fix removes one file. The structural fix removes the possibility.

Apply the rule from lesson 4: group_vars/ and host_vars/ live beside the inventory and nowhere else. Where a genuine staging configuration is needed, it belongs in a second inventory, not a second group_vars directory:

estate/
├── site.yml
├── inventories/
│   ├── production/
│   │   ├── hosts.ini
│   │   └── group_vars/db.yml     db_primary_host: db-prod-01.example.com
│   └── staging/
│       ├── hosts.ini
│       └── group_vars/db.yml     db_primary_host: db-staging-01.example.com
└── roles/

Now the two values cannot collide, because only one inventory is ever loaded in a run. Selecting the environment is -i, which is explicit on every invocation, rather than a precedence rule that is invisible on all of them. Part XXXVII covers environment separation in full.

Prevention. Add the check to CI, because the next person will make the same reasonable mistake:

# fail if a group_vars or host_vars directory exists outside an inventory
find . -type d \( -name group_vars -o -name host_vars \) \
  -not -path './inventories/*' -print -quit | grep -q . && exit 1
exit 0

A five-line check that would have caught this at the pull request, weeks before the incident.

The diagnostic order, generalised

The specific cause here was two group_vars directories. The order that found it works for any precedence surprise:

1. Confirm the symptom is a variable. Ask a host what it resolved. Check a second variable from the same file, to separate “not loaded” from “loaded and overridden”.
2. Find every definition. grep before any Ansible command. This is the step that finds the thing nobody knew existed.
3. Rank them. The precedence table, applied to the definitions you found — from the directory the real run uses.
4. Fix the layout, not the value. An override closes the incident and preserves the cause.
5. Add the check. If a reviewer could not have seen it, a human process will not prevent the recurrence.

Steps 1 to 3 are three commands and take about two minutes on a repository you have never seen. The hour goes to the investigations you start before step 1.

Knowledge check