Break/Fix: the web deploy restarted a queue worker that has never been part of the web tier

Reported symptoms

The Tuesday web deploy ran at 10:00. At 10:14 the events team reports that their queue worker service - stood up eleven days ago, not yet announced, not in any runbook - has lost roughly 40 minutes of queued work.

What happened to it, according to its own logs:

• Its configuration file was overwritten with something referencing an application it does not run.
• It was restarted.
• It was registered with, and then drained from, a load balancer it has never been behind.

The web team’s response is immediate and confident: the deploy has run twice weekly for two years, nothing about it changed, and the worker is not in the web group.

Every part of that is true except the last part, and the last part cannot be checked by reading the repository, because there is no inventory file in the repository to read.

Evidence provided

$ ansible-inventory -i inventory --graph web

@web:
|--web-01
|--web-02
|--web-03
|--webhook-processor-01

$ ansible-playbook -i inventory deploy-web.yml --list-hosts

playbook: deploy-web.yml

play #1 (web): Deploy the web tier	TAGS: []
  pattern: ['web']
  hosts (4):
    web-01
    web-02
    web-03
    webhook-processor-01

$ cat inventory/10-constructed.config.yml

plugin: ansible.builtin.constructed
strict: false
groups:
web: "'web' in instance_name"
db: "'db' in instance_name"
keyed_groups:
- key: instance_tags.Environment
  prefix: env

$ ansible-inventory -i inventory --host webhook-processor-01

{
  "ansible_host": "192.0.2.61",
  "app_config_template": "web-app.conf.j2",
  "instance_name": "webhook-processor-01",
  "lb_pool": "web-pool-a"
}

$ git log --oneline --since=30.days -- inventory/

$ git log --oneline -1 --format='%h %ad %s' --date=short -- inventory/10-constructed.config.yml

4c02e19 2024-09-03 inventory: derive tier groups from the instance name

Work the evidence before reading on

Nothing changed. That is not a figure of speech here - no commit exists in any repository that could explain the difference.

1. Read the groups: expression for web as a string operation, not as a description of intent. What set of names does it match?
2. webhook-processor-01 was created eleven days ago by another team. Was any decision made about the web deploy when that name was chosen?
3. Compare the host count in this deploy’s --list-hosts output with the count in the previous deploy’s change record.

Before continuing: if no file changed, what did change, and where is it recorded?

Root cause

1. The group is computed, not written

The inventory is generated. A plugin enumerates the estate and a groups: block assigns membership by evaluating an expression against each host:

groups:
  web: "'web' in instance_name"

That is a substring test. It matches web-01, web-02, web-03 - and it matches anything else whose name contains those three characters anywhere.

Two years ago that set was exactly the web servers, and the rule was a perfectly reasonable shortcut. It has remained exactly as correct as it was written ever since. What changed is the population it is evaluated against.

2. Naming became scope

webhook-processor-01 was provisioned through the standard pipeline by a team that had no reason to think about the web deploy. The name describes what the service consumes. It contains web because the English word for the events it processes does.

At the moment that instance appeared, the blast radius of the web deploy grew by one host, and every web tier variable applied to it - app_config_template, lb_pool, and whatever else the group carries.

There was no change to review, no approval to seek, and no diff to look at.

3. The deploy behaved perfectly

This is worth stating plainly because it shapes the fix. The playbook did not malfunction. It rendered the web application configuration to the host, restarted the service, added the host to the load balancer pool, drained it, and returned it - which is exactly what it does to a web server, and exactly what the host’s resolved variables told it to do.

Every safeguard in the play operated on the assumption that the inventory was right. None of them can test that assumption, because the inventory is the input that defines what “right” means.

Resolution

1. Repair the worker first. It has a web tier configuration on disk, it was restarted, and it may still be registered somewhere it does not belong; check the load balancer pool membership explicitly rather than assuming the drain step removed it.
2. Establish what else the deploy touched on that host from the run log, and reverse each item deliberately. Do not simply run the playbook owned by the worker team on top and hope it converges over the difference.
3. Freeze the web deploy until the inventory rule is fixed. The next scheduled run will do the same thing again, and the second occurrence will land on a service that is now being watched.
4. Replace the substring test with an exact match against a governed attribute - a role tag assigned at provisioning time, compared for equality, not containment.
5. Survey the whole estate against the new rule before deploying it. Compare the old and new membership for every group in the plugin configuration; a rule that fixes one group may narrow another that was silently relying on the same looseness.
6. Make the tag mandatory at provisioning time so an instance cannot exist without one. A rule that reads a tag no one is required to set will eventually read nothing.
7. Record the approved host count for the deploy and add a guard to the play that refuses to run when the resolved count differs.
8. Tell the other teams. The provisioning pipeline is shared, and the finding that instance names influence deployment scope is something they need before they name the next service.

Verification

1. The group contains exactly the intended hosts. ansible-inventory -i inventory --graph web lists the three web servers and nothing else.
2. The play agrees. ansible-playbook -i inventory deploy-web.yml --list-hosts reports the approved count, and that count is written in the change record for comparison next time.
3. The worker inherits nothing. ansible-inventory -i inventory --host webhook-processor-01 shows no web tier key; check a named key such as lb_pool rather than scanning the output.
4. The guard can fail. Set the approved count one lower and confirm the play refuses to start. This is the check that catches the next silent widening, and it has to be shown to work.
5. No other group changed unexpectedly. Diff the full group membership before and after the rule change, host by host; fixing the web expression must not quietly remove hosts from another group that the loose rule was including.
6. The worker service is healthy. Its own configuration is restored, the service is running, the queue is draining, and it is absent from the web load balancer pool - checked at the load balancer, not from the host.
7. A new instance cannot bypass the tag. Provision a test instance without the role tag and confirm the pipeline rejects it, or that the inventory places it in no tier group at all.

Prevention

• Derive membership from governed attributes only. A tag that provisioning requires is a decision somebody made; a substring of a name is a coincidence.
• Compare for equality, not containment. If a pattern is unavoidable, anchor it at both ends and write a comment explaining what it must not match.
• Record the resolved host count in every change record. With a generated inventory this is the only artefact that makes a scope change visible, because there is no diff to review.
• Guard the count in the play. A pre_tasks assertion on ansible_play_hosts_all | length turns a silent widening into a refusal:

- name: Refuse to deploy to an unapproved number of hosts
  ansible.builtin.assert:
    that: ansible_play_hosts_all | length == approved_host_count
    fail_msg: >-
      Inventory resolved {{ ansible_play_hosts_all | length }} hosts,
      change record approved {{ approved_host_count }}. Re-approve before rerunning.

• Review inventory plugin configuration as a scope document. It is shorter than any playbook and it decides which machines the playbooks are allowed to touch.
• Make unevaluable expressions loud rather than silent, so a rule that has stopped matching fails the inventory instead of quietly producing a smaller group.
• Tell provisioning teams that names affect scope, or better, remove that dependency entirely so the statement stops being true.