Break/Fix: production is talking to the staging database and the fix that was deployed to stop it did nothing

Reported symptoms

A data engineer notices that yesterday’s production orders are present in the staging database. Production has been writing to staging for an unknown length of time.

The application configuration on the affected production hosts reads:

db_host = db-staging.example.com

group_vars/production.yml says:

db_host: db-prod.example.com

Six hours ago the platform team already tried to fix this. They added ansible_group_priority: 10 to group_vars/production.yml, reasoning that production should outrank anything else. The change merged, the next converge ran clean, and the value did not move.

Two further details, both of which sent the investigation sideways:

• Only about a third of production hosts are wrong. The rest are fine.
• The wrong ones are the hosts onboarded in the last two months, so the first hypothesis was a provisioning template.

Evidence provided

$ ansible-inventory -i inventory --host web01

{
  "ansible_host": "192.0.2.11",
  "ansible_group_priority": 10,
  "db_host": "db-staging.example.com"
}

$ ansible-playbook -i inventory show-vars.yml --limit web01

TASK [Print db_host] ***********************************************************
ok: [web01] => {
  "db_host": "db-staging.example.com"
}

$ grep -rn 'db_host' group_vars/ host_vars/ inventory/

group_vars/all.yml:3:db_host: db-default.example.com
group_vars/production.yml:2:db_host: db-prod.example.com
group_vars/staging_mirror.yml:2:db_host: db-staging.example.com

$ grep -rn 'ansible_group_priority' inventory/ group_vars/ host_vars/

group_vars/production.yml:3:ansible_group_priority: 10

$ grep -n -A3 'staging_mirror' inventory/10-prod.ini

14:[staging_mirror]
15:web01
16:web07
17:web09

Work the evidence before reading on

The two db_host candidates are not parent and child. Neither group contains the other, so the usual “more specific wins” intuition has nothing to bite on.

1. Sort the two group names alphabetically. Which one comes second?
2. ansible_group_priority is present in the resolved variables for web01. Being present is not the same as being effective. What has to have already happened before Ansible can read a value out of group_vars/production.yml?
3. The unaffected hosts are unaffected because of something structural, not something configured. Check their group membership.

Before continuing: what would you expect to change if you renamed staging_mirror to archive_mirror, and what does your answer tell you about the mechanism?

Root cause

1. Same-depth groups are resolved in name order

Ansible builds a host’s variables by merging every source that applies, in a fixed order, with later sources overwriting earlier ones. Within the group layer the order is by depth first - a child group is applied after its parents, which is why web beats production when web is a child of production.

production and staging_mirror are siblings. Neither is a child of the other, both are direct children of all, and both define db_host. With the depth tie unbroken, the merge order falls back to the group name, alphabetically. staging_mirror sorts after production, so it is applied last, and it wins.

That is why only three hosts are wrong: they are the only members of staging_mirror. It has nothing to do with when they were provisioned, beyond the fact that the data-copy job that needed the group was set up around the same time.

2. ansible_group_priority in group_vars/ is inert

This is the part that cost six hours, because the attempted fix was the right idea in the wrong file.

ansible_group_priority is not an ordinary variable. It is an input to the algorithm that decides the order in which group_vars/ files are merged. To honour a value set inside group_vars/production.yml, Ansible would have to already have decided that group_vars/production.yml outranks group_vars/staging_mirror.yml - which is the very question the setting exists to answer.

So Ansible reads it from the inventory source, where it is available before any group_vars/ file is opened. Set it in group_vars/ and it is loaded as a plain variable: it shows up in ansible-inventory --host output, it templates, it greps, and it changes nothing.

3. Two groups had an opinion about one variable

Underneath both mechanisms sits an ordinary design fault. staging_mirror exists so a nightly data-copy job can find the hosts it should read from. It is an operational tag. Somebody, reasonably enough at the time, put the staging database endpoint in its group_vars so the copy job knew where to write.

From that moment, one variable had two owners, and which one won was decided by alphabetical order.

Resolution

1. Stop production writing to staging. This is a data incident before it is an Ansible incident; involve whoever owns the staging dataset and decide whether the staging data is now contaminated.
2. Pin the correct value at a layer nothing can outrank, as a temporary measure: host_vars/web01.yml for the three affected hosts, or -e db_host=db-prod.example.com for one emergency run. Extra-vars outrank every group source, which is what you want for ten minutes and never as a permanent answer.
3. Re-render and confirm on the host itself. The variable being right is not the same as the file being right; read /etc/app/app.conf on web01.
4. Restart or reload the application so it picks up the corrected endpoint, and confirm from the database side that production addresses have stopped connecting to staging.
5. Now fix the structure. Delete db_host from group_vars/staging_mirror.yml. A group that exists to tag hosts for a copy job should carry no application configuration at all, and the copy job should get its destination from its own configuration.
6. Remove the inert ansible_group_priority from group_vars/production.yml, so nobody later reads it as evidence that priority is being managed.
7. If two groups genuinely must both have an opinion, set the priority in the inventory source and comment it. In an INI inventory that is a [production:vars] stanza; in a YAML inventory it is the group vars: block.
8. Remove the temporary host_vars pins once the structural fix is in, and re-verify. A pin left behind is the next incident, because it also outranks the correct value.

Verification

1. Both resolvers agree on the production value. ansible-inventory -i inventory --host web01 and a debug task run through ansible-playbook both print db-prod.example.com. Agreement across the two paths is the evidence, not either one alone.
2. Every affected group combination is checked, not just the host you debugged. Pick one host from each distinct set of group memberships in the estate and resolve db_host for it.
3. No key is defined twice at the same depth. grep -rn "^db_host:" group_vars/ returns exactly one file.
4. The rendered file on the host is correct. Read /etc/app/app.conf on a production host directly; the run reporting ok is not evidence that the content changed, only that it matches what Ansible intended.
5. The environment guard can fail. Temporarily set db_host to a third value and confirm the pre_tasks assertion refuses the run. A guard that has never rejected anything has not been tested.
6. The database side is quiet. Connections from production source addresses no longer reach the staging instance. This is the only check that is independent of the tooling that caused the fault.
7. The temporary pins are gone. ls host_vars/ shows no leftover emergency overrides, and the values still resolve correctly without them.

Prevention

• One variable, one owner. If you cannot say in one sentence which group is entitled to set a key, the design is the defect and no precedence rule will rescue it.
• Add a CI check that fails when any key appears in more than one same-depth group_vars/ file. Depth-based precedence is understandable; alphabetical tie-breaking is not something anybody should be relying on.
• Keep operational tags free of configuration. Groups that exist for backup jobs, monitoring scrapes or reporting should contain hosts and nothing else.
• Assert the environment inside the play, so a host that resolved a staging endpoint refuses to be configured:

- name: Refuse to configure a production host with a non-production endpoint
  ansible.builtin.assert:
    that:
      - db_host is defined
      - "'staging' not in db_host"
    fail_msg: "{{ inventory_hostname }} resolved db_host={{ db_host }}, which is not a production endpoint."

• Set ansible_group_priority in the inventory source only, and comment it where it is set. Anywhere else it is a variable that looks like a control.
• Verify configuration by rendering, never by reading the source. The question is what the system resolves, and the only way to know is to ask it.