Contradictory membership is an outage waiting

Put a host in a group called patch_now and also in a group called no_patch. Run ansible-inventory --graph. Ansible will show you both memberships without comment, resolve the conflicting variables by a rule you probably do not have in mind, and run successfully.

There is no error. There is no warning. The recap says ok.

This lesson is about that silence, why it is the correct behaviour for a tool and the wrong behaviour for an estate, and what to do instead of hoping.

Ansible does not have a concept of “mutually exclusive”

Groups are sets of hosts. A host can be in any number of them, and nothing in the model says production and staging cannot both contain the same machine — from the tool’s point of view that is no different from a host being in both production and web, which is the design lesson 1 recommended.

So the tool cannot detect the contradiction. Only you know that patch_now and no_patch are opposites, and the only place that knowledge can live is in a check you write.

What Ansible does have is a deterministic rule for resolving the variables that arrive from both.

The merge rule, demonstrated

Two same-level groups, each setting patch_window:

# inventory/production/hosts.yml
all:
  children:
    production:
      hosts:
        web-a1.example.com:
    web:
      hosts:
        web-a1.example.com:

# group_vars/production.yml
patch_window: 'sat-0200'

# group_vars/web.yml
patch_window: 'sun-0400'

$ ansible-inventory -i inventory/production --host web-a1.example.com

{
  "patch_window": "sun-0400"
}

web won. Not because it is more specific, or later in the file, or more important — because w sorts after p.

Now change one thing that has nothing to do with patching. Rename the web group to application, moving group_vars/web.yml to group_vars/application.yml. Same hosts, same values, same everything except the name:

$ ansible-inventory -i inventory/production --host web-a1.example.com

{
  "patch_window": "sat-0200"
}

The maintenance window for a production host changed because somebody renamed a group. Both runs were on ansible-core 2.21.3, and both are correct behaviour.

The rule in full

For completeness, and because the parts interact, here is the resolution order for inventory-sourced variables. All four were confirmed by execution on 2.21.3:

1. Parent groups first, children after. A variable set on production_canary (a child of production) beats the same variable set on production. Verified: a production_canary value of canary-first overrode the parent.
2. Same-level groups merge alphabetically by group name, with the alphabetically last group winning. This is the case above.
3. ansible_group_priority overrides the alphabetical order among same-level groups — with a large caveat, below.
4. host_vars beat every group. Verified: a host_vars entry of wed-0100 won over both groups.

Note that rule 1 and rule 2 combine in an order that surprises people: depth is considered before name, so a deeply nested group beats a shallower one regardless of what letter it starts with.

The contradiction that actually causes incidents

The patch_now versus no_patch case is easy to explain and rare in practice, because it is obviously wrong. The common one is subtler: a host in two environment groups.

all:
  children:
    production:
      hosts:
        web-a1.example.com:
        web-a2.example.com:
    staging:
      hosts:
        web-s1.example.com:
        web-a2.example.com:      # <- also production

# group_vars/production.yml
max_parallel: 1

# group_vars/staging.yml
max_parallel: 10

$ ansible-inventory -i inventory --host web-a2.example.com

{
  "max_parallel": 10,
  "patch_window": "sun-0400"
}

s sorts after p, so the production host inherits the staging parallelism. A rollout designed to change one production machine at a time will change ten. Every tool reports success, because every tool is doing exactly what the data says.

How does a host end up in two environment groups? Almost always by copy-paste: somebody clones a block to add a staging host and leaves a production hostname in it, or a machine is repurposed from staging to production and added to the new group without being removed from the old. Neither of those is a careless person. Both are a Tuesday.

Detecting it

Three tools, in increasing order of how much you should rely on them.

ansible-inventory --graph shows membership, so a host appearing under two groups you know to be exclusive is visible — if you look, and if the fleet is small enough to read.

ansible-inventory --graph --vars shows the resolved values beside each host, which is how you catch the case where membership looks fine but a value is wrong:

$ ansible-inventory -i inventory --graph web --vars

@web:
|--web-a1.example.com
|  |--{max_parallel = 1}
|  |--{patch_window = sun-0400}
|--web-a2.example.com
|  |--{max_parallel = 1}
|  |--{patch_window = sun-0400}
|--web-s1.example.com
|  |--{max_parallel = 10}
|  |--{patch_window = sun-0400}
|--{patch_window = sun-0400}

That output also illustrates the brief version of this part’s whole argument: a group named web that quietly contains both staging and production hosts. Nothing about the group name says so.

A CI assertion is the one to actually rely on, because it does not depend on anybody looking. Declare which families are exclusive, and fail the build on any intersection:

---
- name: Assert that environment groups do not overlap
  hosts: localhost
  connection: local
  gather_facts: false
  vars:
    environments: [production, staging]
  tasks:
    - name: Find hosts present in more than one environment group
      ansible.builtin.set_fact:
        overlap: >-
          {{ groups.get(environments[0], [])
             | intersect(groups.get(environments[1], [])) }}

    - name: Fail when any host belongs to two environment groups
      ansible.builtin.assert:
        that:
          - overlap | length == 0
        fail_msg: 'Hosts in two environment groups: {{ overlap | join(", ") }}'
        success_msg: 'No host belongs to more than one environment group.'

$ ansible-playbook -i inventory exclusive-groups.yml

TASK [Find hosts present in more than one environment group] *******************
ok: [localhost]

TASK [Fail when any host belongs to two environment groups] ********************
fatal: [localhost]: FAILED! => {
  "assertion": "overlap | length == 0",
  "changed": false,
  "evaluated_to": false,
  "msg": "Hosts in two environment groups: web-a2.example.com"
}

The playbook exits 2 on failure, which is the ordinary task-failure exit code and is enough for any CI system to fail the job.

The structural fix

Detection tells you a contradiction exists. The fix is to arrange that it cannot.

Define each variable in exactly one family. If max_parallel is an environment-level decision, it belongs in group_vars/production.yml and group_vars/staging.yml and nowhere else. The moment a role group also sets it, the value depends on merge order and you are back to the alphabet.

Make exclusive families structural where you can. Two environment groups in one inventory can overlap. Two environment groups in separate inventory sources cannot, because they are never loaded together. That is the argument of lesson 2 of this part, arriving from a different direction.

Use children: for genuine specialisation. Where one group really is a refinement of another — production_canary inside production — nest it. Nesting expresses the intent, and rule 1 makes the child win by depth rather than by name, which survives a rename.

Knowledge check