Break/Fix: the deploy landed on four hosts nobody meant to touch

Reported symptoms

At 09:20 the partner-integration team reports that two machines they stood up four days ago have been reconfigured overnight. The application configuration on them is the production web configuration, complete with the production database hostname.

The web team is adamant that nothing changed. They are right:

• The playbook has not been touched in six weeks.
• The command in the pipeline is the same command it has always been.
• The on-call engineer opened inventory/10-prod.ini, found exactly the two web hosts that belong there, and closed it again.
• The run exited 0. Nothing failed, nothing warned, and the pipeline is green.

The only anomaly anybody noticed, after the fact, is that the run summary lists four hosts where it used to list two.

Evidence provided

$ ansible-inventory -i inventory --graph

@all:
|--@ungrouped:
|--@production:
|  |--@web:
|  |  |--web01
|  |  |--web02
|  |  |--canary01
|  |  |--canary02

$ ansible-inventory -i inventory/10-prod.ini --graph

@all:
|--@ungrouped:
|--@production:
|  |--@web:
|  |  |--web01
|  |  |--web02

$ ansible-playbook -i inventory site.yml --list-hosts

playbook: site.yml

play #1 (web): Deploy the web tier	TAGS: []
pattern: ['web']
hosts (4):
web01
canary02
canary01
web02

$ ls -l inventory/; grep -rn '^\[web\]' inventory/

-rw-r--r-- 1 deploy deploy  184 Jun 02 14:11 10-prod.ini
-rw-r--r-- 1 deploy deploy   96 Aug 07 16:52 20-canary.ini
inventory/10-prod.ini:1:[web]
inventory/20-canary.ini:1:[web]

Work the evidence before reading on

Both --graph outputs are correct. That is the whole difficulty: one of them answers the question the reader asked, and the other answers the question the run asked.

1. Compare the two -i arguments character by character. One is a directory. One is a file.
2. grep found [web] twice. What does Ansible do when two inventory sources declare a group with the same name - keep two groups, or one?
3. canary01 and canary02 appear underneath @production in the graph, and nothing in 20-canary.ini mentions production.

Before continuing: which command in your change process would have shown you four hosts before the run, and did anybody run it?

Root cause

1. A directory inventory is one inventory, not several

-i inventory names a directory. Ansible enumerates every parseable file inside it, runs each through the appropriate inventory plugin, and merges the results into a single in-memory inventory. That is the mechanism that makes group_vars/ and host_vars/ directories work, and it is the reason you can split a large estate across 10-eu.ini, 20-us.ini and 30-apac.ini without repeating yourself.

The merge is by name. Two files that both declare [web] are declaring the same group twice, and the host lists are unioned. There is no namespacing, no precedence, and no warning - the second file adds two members to a group that already had two.

2. The membership was inherited upward, silently

10-prod.ini contains:

[production:children]
web

production is a parent of web, so anything that joins web also joins production, and therefore also picks up group_vars/production.yml. The pilot machines did not merely get deployed to. They got the production database hostname, the production TLS material and every other production-only variable, because group membership is transitive and variable inheritance follows it.

3. Nothing about this looks like a change

The commit that caused the incident added one file with four lines and no YAML. It was reviewed as “adding two hosts to the canary inventory”, which is exactly what the author intended and exactly what the diff appeared to say. The word web in a section header does not read as a blast-radius change to a reviewer scanning for playbook edits.

Resolution

1. Stop the schedule. Disable the pipeline trigger before doing anything else, because the play runs hourly and each run reasserts the wrong configuration on the pilot hosts.
2. Establish the true blast radius from the run log, not from the inventory. Every host the play touched is named in the recap; that list is authoritative and the inventory is now a hypothesis.
3. Rename the colliding group in the new file to [web_canary]. If those hosts genuinely need part of the web configuration, add it as an explicit [web_canary:vars] block or a deliberate parent, so the intent is written down rather than inferred from a name.
4. Confirm the shape against the directory: ansible-inventory -i inventory --graph. Two hosts under @web, two under @web_canary, and @production containing only what it should.
5. Confirm the resolved variables for one pilot host: ansible-inventory -i inventory --host canary01. Look for a specific production-only key by name; scanning the JSON for anything that looks wrong is not a check.
6. Treat the two pilot hosts as a configuration incident. Reverse what the play wrote, using the run log to enumerate it, and do it deliberately rather than by running some other playbook at them.
7. Rotate anything the play deployed that counts as a secret. A production credential that has been written to a partner-pilot machine is disclosed, whether or not anybody read it.
8. Add a host-count guard to the play, described in Verification, and only then re-enable the schedule.

Verification

1. The merged inventory has the right shape. ansible-inventory -i inventory --graph shows @web with exactly web01 and web02, and @web_canary as a sibling rather than a member.
2. The play targets two hosts. ansible-playbook -i inventory site.yml --list-hosts reports hosts (2). This is the check that would have caught the original fault and it takes under a second.
3. The pilot hosts no longer inherit production variables. ansible-inventory -i inventory --host canary01 | grep db_host returns nothing. Naming the key matters - a check that greps for the string production would have passed even while the fault was live.
4. A dry run confirms it. ansible-playbook -i inventory site.yml --check --diff produces no output lines mentioning canary01 or canary02.
5. The guard can fail. Set the expected count in the pre-flight assertion to 3, run the play, and confirm it refuses to start. A guard you have never seen reject anything is a comment.
6. The reversal on the pilot hosts is complete. Verify from the hosts themselves - the application configuration, the service unit state and the credentials on disk - not from a subsequent Ansible run reporting ok.
7. Secrets are rotated and the old values are refused. Confirm the retired credential no longer authenticates rather than confirming a new one was issued.

Prevention

• Inspect the inventory with the exact -i argument the run uses. Different argument, different question, and the answers can both be correct.
• Put --list-hosts output in the change record for anything that writes. A diff in the host count between the last change and this one is the single highest-value pre-flight signal in Ansible, and it costs a second.
• Guard the count in the play. A pre_tasks assertion on ansible_play_hosts_all | length converts a silent doubling into a refusal:

- name: Refuse to run against an unexpected number of hosts
  ansible.builtin.assert:
    that: ansible_play_hosts_all | length == expected_host_count
    fail_msg: >-
      Expected {{ expected_host_count }} hosts, inventory resolved
      {{ ansible_play_hosts_all | length }}. Review the inventory before rerunning.

• Name groups for the tier they are, not for the role they resemble. web_canary and web_eu never collide; two files that both say web always will.
• Review inventory commits as changes to production scope. The reviewer’s question is not “are these hosts real” but “which plays now include them”.
• Keep parent-group membership shallow and explicit. Every :children stanza is a variable-inheritance edge, and transitive inheritance is what turned a two-host mistake into a production-credentials mistake.