Ansible failure modes — partial fleet, lockouts, idempotency breakage, and recovery

Ansible is the right tool for fleet firewall management, but it does not make the work safe. The same playbook that runs successfully 99 times may fail the 100th time in a way that locks an operator out of a production firewall. The discipline is to know the failure modes in advance, to design for recovery rather than diagnose in the moment, and to capture the lessons from each incident so the next playbook is incrementally safer.

This lesson covers the four major failure modes (partial fleet, lockout, idempotency breakage, transient errors), the recovery playbook for each, and the post-incident review that turns incidents into improvements.

The four failure modes

A firewall Ansible run can fail in ways the underlying scripts cannot. The modes:

1. Partial fleet failure. Some firewalls apply the change; others do not. The fleet is now in an inconsistent state — half the firewalls have the new rule, half do not.
2. Lockout. A rule change accidentally blocks the operator’s network from reaching the GUI. The change has been applied; the operator cannot reach the firewall to reverse it.
3. Idempotency breakage. A new collection version or new OPNsense firmware changes the API schema; the playbook re-applies things that have not drifted, in incorrect ways.
4. Transient API errors. The framework was busy (workers saturated), the network blipped, or the firewall was in the middle of a configd reload. The module reports failed but the change may or may not have applied.

Each mode has a different recovery path; the trick is to recognise which mode you are in before applying the wrong recovery.

Partial fleet failure: recognise and recover

Partial fleet failure manifests as a recap with mixed failed=0 and failed=N across hosts, with the same playbook task failing on some hosts but succeeding on others. The causes:

• Different firmware. Some hosts are still on an older OPNsense release whose API does not accept a field the module sends. Those hosts fail validation; the rest succeed.
• Different controller credentials. One host has a key that was rotated and not propagated.
• Network reachability. The controller cannot reach one firewall (a transient routing problem) — unreachable=1.
• Lockout on one host. A rule that succeeded on some firewalls disabled the management network on one.

The recovery depends on cause:

# Get a per-host recap to identify the failed hosts
ansible-playbook -i inv/prod.yml playbooks/all.yml 2>&1 | tee run.log

# Re-run, limited to the failed hosts
ansible-playbook -i inv/prod.yml playbooks/all.yml --limit fw-stage-03,fw-dc2-edge-02

# Verify idempotency on the recovered hosts
ansible-playbook -i inv/prod.yml playbooks/all.yml --limit fw-stage-03,fw-dc2-edge-02 --check

The discipline: after every partial-fleet failure, the operator inspects the per-host state, identifies the cause, and re-runs against only the failed hosts. The good hosts are not re-run; that wastes time and may double-apply.

$ ansible-playbook -i inv/prod.yml playbooks/all.yml 2>&1 | grep -E 'failed=0|failed=' | tee run-recap.log

fw-dc1-edge-01        : ok=44  changed=4  unreachable=0  failed=1  skipped=0
fw-dc1-dmz-01         : ok=44  changed=4  unreachable=0  failed=0  skipped=0
fw-dc2-edge-01        : ok=44  changed=4  unreachable=0  failed=1  skipped=0
fw-dc2-dmz-01         : ok=44  changed=4  unreachable=0  failed=0  skipped=0
fw-stage-03           : ok=20  changed=2  unreachable=0  failed=12 skipped=0
fw-stage-04           : ok=44  changed=4  unreachable=0  failed=0  skipped=0

Lockout: console recovery

A lockout happens when a rule change blocks the management network from reaching the GUI. With SSH on a non-management port also blocked (or never enabled), the operator has only console access: either a physical console (KVM over IP, serial console) or the boot-menu’s factory reset.

The playbook’s management-access discipline:

- name: Pre-flight — management access invariant
  hosts: firewalls
  connection: local
  gather_facts: false
  tasks:
    - name: List the automation rules
      ansibleguy.opnsense.list:
        target: rule
      register: rules

    - name: Fail if this change-set would add a block matching the management network
      ansible.builtin.fail:
        msg: "A rule in this change-set blocks {{ mgmt_network }} — review before proceeding"
      when: >
        desired_rules
        | selectattr('action', 'equalto', 'block')
        | selectattr('destination_net', 'in', [mgmt_network, 'any'])
        | list | length > 0

The check that matters is on what the change-set is about to add, not on what already exists. The anti-lockout rule cannot be removed through this API — it is generated by the firewall rather than stored as an editable rule — so asserting its presence proves nothing. What can lock an operator out is a new block that matches management traffic and sits ahead of whatever was allowing it, and that is a property of the desired state, which the playbook already holds.

For an actual lockout, the recovery is console:

1. SSH to the firewall from a console-attached laptop (if SSH was enabled and not blocked).
2. Web UI from a console-attached laptop via the management interface directly.
3. Serial console or KVM for reboots or boot-menu access.
4. Boot-menu factory reset as the last resort (loses configuration).

The operator who designed the playbook should have configured two access paths so a single change cannot lock out everyone. A playbook that adds a “deny all to GUI” rule with no exception for the operator’s IP is anti-discipline even if the operator can recover with the console.

Idempotency breakage: detect and pin

Idempotency breakage happens when a collection upgrade or OPNsense firmware upgrade changes the API schema in a way the old module does not handle correctly. Symptoms:

• A second-run --check reports changed=1 (the playbook is reapplying something that the firewall says is already there).
• Configd resolver errors in the firewall log after a playbook run.
• Modules that previously ok’d on no-change now changed’d for the same input.

The detection is the daily drift check (lesson 257) — a clean drift check should never see changed. The fix is to pin the collection version (lesson 254) and to validate against the new firmware in a lab before deploying.

The recovery:

# Pin collection
ansible-galaxy collection install ansibleguy.opnsense:1.2.16 -p collections/ --force

# Validate the playbook
ansible-playbook -i inv/lab.yml playbooks/all.yml --check --diff

# Re-run the change playbook against production
ansible-playbook -i inv/prod.yml playbooks/all.yml --limit fw-canary --check

If the canary check shows zero changes, the pin holds; re-run the playbook against the fleet to lock the idempotency back in place. If even the canary check shows changes, the new collection version is incompatible with the playbook; revert the pin and investigate.

Transient API errors: retry, do not re-apply

Transient errors look like failures but recover on retry:

• 503 Service Unavailable from a worker-saturated framework.
• Timeouts from a configd reload in progress.
• HTTP-level resets during a TLS handshake.

The playbook should have retry logic on ok/failed for transient errors:

- name: Ensure the rule, absorbing transient API errors
  ansibleguy.opnsense.rule:
    description: "CHG-2026-1314 outbound"
    match_fields: ['description']
    source_net: ci_runners
    api_retries: 3
    reload: false
  register: result
  retries: 3
  delay: 5
  until: result is succeeded

The modules carry their own retry parameter, api_retries, which retries the connection when the request never reached the firewall. That is the narrower and safer of the two: it covers the case where nothing happened. Ansible’s retries/until wraps the whole task and re-runs the module, which is safe here only because the module is idempotent — it re-reads the rules and does nothing if the rule already exists.

Three retries with 5-second delays absorb transient errors without re-running the playbook from scratch. The discipline: retries on idempotent operations only; non-idempotent operations (raw add calls without a UUID) are not safe to retry.

The recovery from a transient error: let the playbook’s retry logic absorb it. If the retries are exhausted, the playbook failed state will surface; the operator should not re-run blindly. Re-running after a successful-on-second-try adds a false changed to the recap; re-running after a successful-on-third-try and an intervention between tries may double-apply.

Post-incident review (PIR)

Every playbook failure deserves a PIR. The PIR’s output is a change to the playbook (or the inventory, or the runbook) — the next incident is a smaller blast radius. The disciplines:

1. Identify the failure mode. Partial fleet, lockout, idempotency, or transient? The PIR records which.
2. Identify the root cause. Was it a playbook bug, an API schema change, a credentials error, or an operator mistake?
3. Identify what the playbook should have caught. Could a pre-flight over the desired state have detected it before any write?
4. Add a guard. The PIR’s outcome is a new pre-flight assertion, a new retry policy, or a new pinning strategy.

The PIR is short — fifteen minutes — but produces a concrete change. A PIR that ends with “we’ll be more careful next time” without changing the playbook is a missed opportunity; the failure mode is still there waiting for the next run.

Recovery cheatsheet

Summary

• Four failure modes: partial fleet, lockout, idempotency breakage, transient API errors. Each has a different signature and a different recovery.
• Pre-flight checks over the desired state are what prevent lockouts; the anti-lockout rule is a narrow floor, not a guarantee. Console recovery is the backstop.
• Pin collection version and validate against a lab before upgrading either collection or OPNsense firmware. Daily drift check catches regressions.
• Retry only idempotent operations. Transient errors resolve naturally with backoff.
• The PIR’s output is a change to the playbook. Test recovery paths in the lab, not during the incident.