Break/Fix: the security patch was signed off as deployed to 60 hosts and reached none of them

Reported symptoms

A vulnerability scan flags 60 application hosts for a package that was patched six weeks ago. The change record for that patch is closed, approved, and links to a green pipeline run as evidence.

Everything checks out on inspection:

• The playbook is correct. The task names the right package and the right state.
• The pipeline is green, and has been green every night for six weeks.
• Running the same playbook by hand against one host patches it in eleven seconds.
• No error appears anywhere in any log.

The only detail anybody eventually finds odd is the duration. The pipeline step takes four seconds. Patching 60 hosts has never taken four seconds.

Evidence provided

$ ansible-playbook -i inventory patch-cve.yml

[WARNING]: Could not match supplied host pattern, ignoring: app_servers

PLAY [Patch the vulnerable package] ********************************************
skipping: no hosts matched

PLAY RECAP *********************************************************************

$ ansible-playbook -i inventory patch-cve.yml >/dev/null 2>&1; echo $?

0

$ ansible-playbook -i inventory patch-cve.yml --list-hosts

playbook: patch-cve.yml

play #1 (app_servers): Patch the vulnerable package	TAGS: []
  pattern: ['app_servers']
  hosts (0):

$ grep -n 'hosts:' patch-cve.yml; ansible-inventory -i inventory --graph | grep -E '@app'

2:  hosts: app_servers
|--@application:

$ git log --oneline -1 --format='%h %ad %s' --date=short -- inventory/

e17b3d4 2026-06-30 inventory: rename tier groups for consistency

$ grep -rln 'app_servers' --include='*.yml' . | head

./patch-cve.yml
./maintenance/rotate-logs.yml

Work the evidence before reading on

There is no error anywhere. That is not because the error was suppressed; there is genuinely nothing that failed.

1. Look at the PLAY RECAP section in the run output. How many host rows does it contain, and what would you expect for 60 hosts?
2. The warning line is present and the exit code is 0. Which of those does the pipeline consume?
3. Four seconds. What work would a run have to do to take longer than that?

Before continuing: what does the pipeline check to decide that a deployment succeeded, and could that check ever have distinguished this run from a real one?

Root cause

1. The group was renamed and this play was missed

An inventory refactor six weeks ago renamed app_servers to application. Most playbooks were updated in the same commit. Two were not, and one of them is the patch play.

That part is an ordinary mistake and it is not the interesting half.

2. A pattern that matches nothing is a warning, not an error

When a host pattern matches no hosts, Ansible warns, prints skipping: no hosts matched, and continues. A play with no hosts runs no tasks, so nothing can fail, so the run ends successfully and the exit code is 0.

This is a defensible design. A playbook containing several plays should not fail because one optional tier is absent from a particular inventory, and treating an empty match as fatal by default would break a great many legitimate playbooks.

The consequence for a pipeline is severe. Every automated signal reads as success:

Only the last two are anomalies, and both are absences. A pipeline detects the presence of failure, not the absence of work.

3. The change record was closed on that evidence

The deployment was signed off by linking a green pipeline run, which is exactly the correct process operating on a signal that could not carry the information required.

Six weeks of nightly runs have since reasserted the same nothing, each one adding to the impression that the patch is deployed and stable.

Resolution

1. Treat the 60 hosts as unpatched for six weeks and raise it as such. The naming error is trivial; the exposure window is not, and it belongs to whoever owns the vulnerability response.
2. Correct the group name in the play, and check the other playbook that grep found. An inventory refactor rarely misses exactly one file.
3. Deploy the patch, and confirm the run now reports 60 host rows in the recap and takes a plausible amount of time.
4. Verify at the hosts. Check the installed package version on every host, and rerun the vulnerability scan; the pipeline reporting success is what got everybody here.
5. Add a host-count assertion to the play, so a future rename produces a refusal rather than a vacuous success.
6. Make the pipeline fail on zero hosts, independently of the play. A guard inside the playbook protects that playbook; a guard in the pipeline protects every playbook it runs.
7. Audit the estate for the same pattern - every pipeline that checks only an exit code, and every play whose target group could have been renamed.
8. Record the host count in the change record from now on, so a future drop from 60 to 0 is visible as a difference rather than as a silence.

Verification

1. The play targets the expected hosts. ansible-playbook ... --list-hosts reports 60, and that number is written into the change record.
2. The pipeline fails on zero hosts. Point a scratch branch at a deliberately wrong group name and confirm the build goes red. This is the check that can fail and its absence is the entire incident.
3. The play refuses on an unexpected count. Set the expected count to 61 and confirm the play stops before doing anything.
4. The hosts are actually patched. Confirm the installed package version on all 60 hosts, and confirm with an independent vulnerability scan rather than with the package database alone.
5. The recap is populated. The successful run shows one row per host, and the duration is consistent with real work.
6. The other playbook is fixed and verified the same way, not merely edited.
7. A grep for the old group name across playbooks, pipeline definitions and documentation returns nothing.

Prevention

• Fail on zero hosts. Assert it in the play so the run refuses rather than completing vacuously:

- name: Refuse to run against an unexpected number of hosts
  ansible.builtin.assert:
    that: ansible_play_hosts_all | length == expected_host_count
    fail_msg: >-
      Expected {{ expected_host_count }} hosts, resolved
      {{ ansible_play_hosts_all | length }}. Check the inventory and the host pattern.

• Guard it in the pipeline too. A step that runs --list-hosts and fails on a count of zero protects every playbook, including ones written after the guard.
• Record the host count in every change record. A deployment that reached 60 hosts last month and 0 this month is a one-line diff if anybody wrote the number down.
• Verify deployments at the target. Package versions, service responses and scans are independent of whether the automation ran at all.
• When renaming a group, grep every playbook, pipeline definition, runbook and document for the old name in the same change. Nothing will fail to remind you.
• Be suspicious of a run that is much faster than usual. Duration is a weak signal and it is often the only one that differs.
• Remember the related silent cases: a when: false everywhere, a --tags value matching nothing, and a group that exists but is empty all produce the same green nothing.