Skip to main content
RunBook Academy

← All break/fix scenarios in Ansible

advancedconnectivity~40 min

Break/Fix: Ansible firewalled itself off 94 production hosts and the only way back is the console

Reported symptoms

  • ●A firewall hardening play ran and 94 of 200 hosts became permanently unreachable to Ansible
  • ●The hosts are up: they respond to ICMP, serve application traffic, and appear healthy in monitoring
  • ●Only SSH from the automation network is refused
  • ●The run reported success on most of the affected hosts before the connection dropped
  • ●Rerunning the play cannot help, because the play needs SSH to reach the hosts
  • ●The 106 unaffected hosts are the ones the play had not reached when it was stopped

Evidence

  • · `ansible <host> -m ansible.builtin.ping` reports UNREACHABLE with the connection refused or timed out
  • · `ping <host>` succeeds and the application endpoint returns a normal response
  • · The application is serving traffic, so this is a control-plane loss rather than an outage
  • · The rule list rendered by the play has no rule permitting the automation network on port 22
  • · The play sets a default DROP policy on the input chain before the accept rules are applied
  • · The variable that should hold the automation network is absent from the group that the affected hosts belong to
  • · The rendered rule set on an unaffected host, inspected before the play reaches it, would produce the same result
Diagnosis and resolutionclick to reveal

Root cause

The hardening play sets the input chain policy to DROP and then adds accept rules from a list. The list is built from a variable that is defined for one group and not for another, so on hosts in the second group it renders empty of the management rule - and the play had no guard requiring the SSH accept rule to be present before the policy was tightened. On those hosts the DROP policy took effect with nothing permitting the automation network, and the connection Ansible was using was severed by the change Ansible was making. The order is what makes it unrecoverable rather than merely wrong: policy first, accept rules second, with the connection running over the thing being denied. Because the hosts are otherwise healthy - serving traffic, answering ICMP, green in monitoring - the failure presents as a connectivity problem in the automation rather than as an incident on the hosts, which delays the realisation that every one of the 94 needs to be touched physically or through out-of-band management.

Remediation

Recovery requires access that does not depend on SSH. There is no remote fix for a host that is dropping the packets you would use to fix it, so each affected host must be reached through its serial console, its out-of-band management processor, a hypervisor or cloud provider console, or a configuration-management agent that polls outward rather than being connected to. Restore the management accept rule on each host, verify SSH from the automation network before moving on, and only then correct the play. The correction has two parts - the missing variable, and the structural fault that allowed a policy to be tightened without the rule that keeps you connected - and shipping only the first one leaves the trap in place for the next variable that goes missing.

Verification

Ansible reachability is the verification: `ansible <host> -m ansible.builtin.ping` must succeed from the automation network on every recovered host, because it exercises the exact path that was lost. Prove the new guard can fail by running the corrected play against a scratch host with the management variable deliberately undefined and confirming it refuses to start rather than applying a policy. Confirm the rule ordering by reading the rendered rule set on a recovered host and checking that the management accept rule precedes any drop. Then confirm from a second, independent network position that the intended restrictions are actually in force, so the fix has not simply reopened everything.

Prevention

A play that can remove your own access needs three things that this one had none of. First, a guard that refuses to run when the management rule would be absent, asserting on the rendered rule set rather than on the variable. Second, an ordering that never leaves a window in which the policy is restrictive and the accept rule is missing - apply the whole rule set atomically, or add accepts before tightening the policy. Third, a timed rollback: a scheduled revert that fires unless the change is confirmed, so a lockout undoes itself in minutes rather than requiring 94 console sessions. Beyond the play, canary every firewall change to one host and verify reachability from the automation network before the fleet, keep an out-of-band access path documented and tested, and treat firewall changes with the same care as authorisation changes, because both can remove the means of repair.

Reported symptoms

A firewall hardening play runs at 14:00 across 200 hosts. At 14:03 the operator notices that hosts are dropping out of the run as UNREACHABLE and stops it.

By then 94 hosts are unreachable and stay unreachable.

The confusing part is that nothing is down:

  • ping succeeds to every one of them.
  • The application endpoint on each returns a normal response.
  • Monitoring is entirely green; the monitoring collectors reach the hosts on a different network path that is still permitted.
  • The hosts have not rebooted and their uptime is unchanged.

So the first framing is “Ansible has a connectivity problem”, and half an hour goes into the controller, the bastion and the automation network - all of which are healthy.

The 106 hosts that still work are simply the ones the play had not reached.

Evidence provided

Read-only / Safethe automation path is gone
$ ansible app041 -i inventory -m ansible.builtin.ping
app041 | UNREACHABLE! => {
  "changed": false,
  "msg": "Failed to connect to the host via ssh: ssh: connect to host 192.0.2.41 port 22: Connection timed out",
  "unreachable": true
}
Read-only / Safethe host is fine - this is a control-plane loss, not an outage
$ ping -c2 192.0.2.41; curl -s -o /dev/null -w '%{http_code}\n' http://192.0.2.41:8080/health
2 packets transmitted, 2 received, 0% packet loss
200
Read-only / Safethe policy is set first, and the accept list has a default of empty
$ grep -n -B2 -A12 'Harden the input chain' roles/firewall/tasks/main.yml
18:- name: Harden the input chain
19:  ansible.builtin.iptables:
20:    chain: INPUT
21:    policy: DROP
22:
23:- name: Permit the allowed sources
24:  ansible.builtin.iptables:
25:    chain: INPUT
26:    protocol: tcp
27:    destination_port: "22"
28:    source: "{{ item }}"
29:    jump: ACCEPT
30:  loop: "{{ ssh_allowed_sources | default([]) }}"
Read-only / Safedefined for one group only
$ grep -rn 'ssh_allowed_sources' group_vars/
group_vars/web.yml:11:ssh_allowed_sources: ['198.51.100.0/24']
Read-only / Safethe affected hosts are in a group with no such variable
$ ansible-inventory -i inventory --graph app | head -4
@app:
|--app001
|--app002
|--app003
Read-only / Safea host the play had not reached yet - and its group has the same gap
$ ansible app101 -i inventory -b -m ansible.builtin.command -a 'iptables -S INPUT' | head -3
-P INPUT ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

Work the evidence before reading on

The play succeeded. Every task it ran did exactly what it was written to do, on every host it reached.

  1. Read the two tasks in order. What is the state of the host between the first task completing and the second task completing?
  2. ssh_allowed_sources has a default([]). What does the second task do when the list is empty?
  3. Which group are the 94 hosts in, and which group has the variable?

Before continuing: which connection was Ansible using to apply the first task, and what did the first task do to it?

Root cause

1. The policy was tightened before the accept rules existed

The play sets the input chain policy to DROP, then adds accept rules from a list.

Between those two tasks the host drops every inbound packet that no existing rule permits. On these hosts there was no pre-existing rule permitting SSH from the automation network, so the moment the policy changed, the connection Ansible was using was severed - by the task Ansible had just executed.

Ordering is the entire difference between a working hardening play and an unrecoverable one. Accept rules first, then policy, and the window never opens.

2. The accept list rendered empty

ssh_allowed_sources is defined in group_vars/web.yml and nowhere else. The 94 affected hosts are in the app group, which has no such variable.

The loop is guarded by default([]), so on those hosts it iterates over an empty list, adds no rules, reports ok, and moves on without complaint. That default was added at some point to stop the play failing on hosts where the variable was missing, which is precisely the failure that would have prevented this incident.

A default([]) on a list that controls access is the same defect as a default('') on a required template value: it converts a loud error into a silently wrong result.

3. Nothing asserted that the rule survived

There is no guard anywhere in the play requiring that a rule permitting the automation network exists before the policy is tightened. The play trusts that the variable is populated, and the variable is the only thing standing between the play and a fleet-wide lockout.

Resolution

  1. Stop the play if it is still running, and disable any schedule that would run it again. The 106 hosts that still work are only working because the play has not reached them.
  2. Establish the out-of-band path before touching anything. Serial console, management processor, hypervisor or cloud console - whichever exists, confirm it works on one host before planning for 94.
  3. Recover one host first and confirm the recovery procedure end to end, including that SSH from the automation network works afterwards. Do not start a 94-host recovery on an unproven procedure.
  4. Restore the management accept rule on each host through the out-of-band path. Add the accept rule before touching the policy, so the host is never in the locked-out state again during recovery.
  5. Verify reachability from the automation network after each host, not at the end. A host you believe you fixed and cannot reach is indistinguishable from one you have not fixed.
  6. Correct the missing variable, but do not stop there. The variable is the trigger; the structural fault is a play that can tighten a policy without the rule that keeps you connected.
  7. Restructure the play so the rule set is applied atomically, or so accepts precede the policy, and add a guard that asserts the management rule is present in the rendered result.
  8. Add a timed rollback to the play, then canary the corrected play on one host and verify reachability before going anywhere near the fleet.

Verification

  1. Ansible can reach every recovered host. ansible <host> -m ansible.builtin.ping succeeds from the automation network, which is the exact path that was lost and therefore the only meaningful check.
  2. The guard can fail. Run the corrected play against a scratch host with the management variable deliberately undefined and confirm it refuses to start. A guard that has never rejected anything is why this incident happened.
  3. The rule ordering is correct. Read the rendered rule set on a recovered host and confirm the management accept rule precedes any drop or restrictive policy.
  4. The intended restrictions are actually in force. From a network position that should be denied, confirm the connection is refused - a fix that restores access by permitting everything is not a fix.
  5. The timed rollback works. On a scratch host, apply the change and do not confirm it; the host must return to its previous rule set on its own.
  6. The out-of-band path is documented and tested for every host in the estate, including the ones that were not affected. This incident happened to reach 94; the next one may reach all of them.
  7. A canary run on one host is clean and reachable before the fleet is touched, and that step is now part of the runbook rather than a decision somebody makes.

Prevention

  • Guard on the rendered result, not on the input. Assert that a rule permitting the management network exists in what is about to be applied:
- name: Refuse to harden without a management accept rule
  ansible.builtin.assert:
    that:
      - ssh_allowed_sources is defined
      - ssh_allowed_sources | length > 0
      - management_network in ssh_allowed_sources
    fail_msg: >-
      {{ inventory_hostname }} would be hardened with no rule permitting
      {{ management_network }}. Refusing to continue.
  • Never use default([]) on a list that controls access. The failure it prevents is the one you want.
  • Apply rule sets atomically. Render the whole rule set and load it in one operation so no intermediate state exists on the host.
  • Add a timed rollback to any change that can remove your own access. A scheduled revert that fires unless confirmed turns a lockout into a short outage.
  • Canary to one host and verify reachability from the automation network before the fleet. One host is a survivable mistake; 94 is a week.
  • Maintain and test an out-of-band access path for every host, and record it where somebody under pressure will find it. The time to discover that the console password is wrong is not while 94 hosts are unreachable.
  • Treat firewall changes with the same procedure as authorisation changes. Both can remove the means of repair, and both deserve a written pre-flight rather than a review.