Break/Fix: the same commit that passed CI on Friday removed every firewall rule on Monday

Reported symptoms

Monday 09:10. The scheduled firewall converge runs and removes every rule on 140 hosts that the role does not explicitly manage - including the rules another team maintains for the monitoring network, and the rules the database tier depends on.

The role has run every morning for a year. The last commit touching it was in March.

The confusing part arrives within ten minutes:

• The CI runner reproduces it every time on the exact commit that passed on Friday.
• One engineer reproduces it on their workstation.
• Another engineer cannot reproduce it at all, on the same commit, with the same inventory, against the same staging host.

At that point the working workstation becomes the object of study, on the theory that it is configured differently. It is. Just not in any of the ways being examined.

Evidence provided

$ ansible --version | head -1

ansible [core 2.21.3]

$ ansible-galaxy collection list | grep -i firewall

example.firewall            4.2.0

$ ansible-galaxy collection list | grep -i firewall

example.firewall            4.1.3

$ cat requirements.yml

---
collections:
- name: example.firewall
- name: example.netbase

$ grep -n -A2 'ansible-galaxy' .gitlab-ci.yml

  before_script:
- ansible-galaxy collection install -r requirements.yml --force

$ ansible-doc example.firewall.rule_set 2>/dev/null | grep -A4 'purge_unmanaged'

        purge_unmanaged:
default: true
description: Remove rules present on the host that are not described by
this task. Changed from false in 4.2.0.
type: bool

$ grep -rn 'purge_unmanaged' roles/firewall/

Work the evidence before reading on

The commit is identical everywhere. Two machines behave differently on identical input, so the difference is not in the input.

1. List everything that differs between the two controllers. Start with what ansible --version reports, then with what it does not report.
2. requirements.yml names collections without versions and CI installs from it on every build. What does a build on Monday install that a build on Thursday did not?
3. The role never sets the option that changed. Who chose the value it was using for the last year?

Before continuing: what, exactly, is the version of the software that ran on Friday? Write down everything you would need to reproduce it.

Root cause

1. The controller changed, silently, on every build

requirements.yml names the collection and no version. The CI job installs from that file at the start of every build. Each build therefore resolves to whatever the newest published version happens to be at that moment.

A release published on Friday evening changed the default of an option from false to true. The release is correct, the change is deliberate, and it is in the changelog. Nobody read it, because nobody chose to upgrade - the upgrade happened as a side effect of running a build.

The engineer whose workstation still worked had installed collections some months earlier and had not run ansible-galaxy since. That is the entire difference, and it looked like a workstation configuration difference for an hour.

2. ansible --version does not report the moving part

The command everybody reached for reports ansible-core, the Python version and the configuration file. It says nothing about collection versions, and in a modern Ansible installation most modules live in collections.

So the environment comparison that was performed showed identical results while the environments differed in the only way that mattered. ansible-galaxy collection list is the other half of that comparison and it was not run for an hour.

3. An unset option is a dependency you did not write down

The role never set purge_unmanaged. For a year it inherited false and behaved additively: it ensured its own rules existed and left everything else alone.

Nothing in the role said “this role relies on the default being false”. Nothing could have, because the reliance is invisible - it is expressed by the absence of a line.

When the default moved, the role became destructive without any change to its text, and the first run under the new default removed every rule it did not manage.

Resolution

1. Stop the schedule. The converge runs every morning and will repeat the purge tomorrow.
2. Pin the collection in requirements.yml to the version that was current before the change, and confirm every controller resolves to it.
3. Restore the firewall rules from their source of truth, not from memory and not from a surviving host. Rules maintained by another team have to come from that team.
4. Verify the restored rules from off-host. A rule present in the ruleset and a connection that is actually permitted or refused are different claims, and only the second one matters.
5. Set the option explicitly in the role. Pinning protects you until the next upgrade; an explicit value protects you permanently and documents the intent.
6. Survey every controller. CI runners, workstations and any bastion are now on assorted versions, and more than one of them has configured production this week.
7. Read the changelog between the pinned version and the current one before upgrading, then upgrade as a change of its own with its own test run.
8. Record the full environment in the run artefacts from now on - ansible-core and every collection version - so the next divergence is a diff rather than an investigation.

Verification

1. Every controller reports the same collection version. ansible-galaxy collection list agrees across CI runners and workstations, and each collection appears exactly once with one path.
2. A fresh install changes nothing. Run the install from requirements.yml on a clean controller and confirm it produces the pinned version, not the newest. The check is that reinstalling is a no-op.
3. The role no longer depends on the default. grep -rn purge_unmanaged roles/firewall/ returns the explicit setting, and ansible-doc agreeing with the role is no longer load-bearing.
4. The check can fail. On a scratch controller, install a different version deliberately and confirm the pipeline guard rejects the mismatch before any play runs.
5. The firewall is correct from outside. A connection that should be permitted succeeds and a connection that should be refused is refused, tested from another host rather than from the target.
6. A dry run on one host shows no unexpected removals. --check --diff on the repaired role must not propose deleting anything.
7. The environment is recorded. A completed run leaves an artefact naming ansible-core and every collection version, and two runs can be diffed.

Prevention

• Pin every collection to an exact version. An unpinned dependency reinstalled by CI means the software under test is chosen by the calendar.
• Treat a version bump as a change: its own commit, its own review, its own changelog reading, its own test run. Upgrading as a side effect of a build is how this happens.
• Set explicitly any option that governs what a module does to things it does not manage. Purge, prune, force, replace and state defaults are the ones that turn additive roles authoritative.
• Record the whole environment in run artefacts. ansible --version plus ansible-galaxy collection list is the minimum, and it makes a divergence a two-second diff.
• Rebuild controllers from a declared definition rather than letting them accumulate installs, and make CI and workstations use the same definition.
• Check for duplicate collections across search paths. The same collection at two versions in two paths is a fault waiting for a configuration change to expose it.
• When one commit behaves differently on two machines, compare the environments before rereading the code. The code is provably identical; something else is not.