Drift detection with Ansible — comparing desired state to actual state across the estate

A firewall managed by Ansible has a desired state in the playbook. The actual state on the firewall reflects the playbook plus whatever else has touched the firewall since the last run — hand-edits, GUI clicks, an emergency rule added at 03:00, an alias removed by an operator trying to debug a connectivity issue. The difference between desired and actual is drift.

Detecting drift is the precondition for fixing it. Three approaches exist: idempotency as detection (re-running the playbook and reading changed), explicit drift plays that read state without writing, and external observability tools that compare a config snapshot to a known baseline. Each has trade-offs; the right answer is usually a combination — daily explicit drift scans, idempotency-based detection on every scheduled run.

This lesson covers how to detect drift with Ansible, what counts as drift (and what does not), how to run a periodic scan, and how to handle intentional manual changes without losing the playbook’s authority.

Idempotency as drift detection

The simplest drift check is to re-run the playbook in --check mode. The playbook reports changed=1 for any object that does not match the desired state. The operator who runs the playbook nightly gets a free drift report: any firewall that reports changed has drifted.

The pattern:

ansible-playbook -i inv/prod.yml playbooks/firewall-rules.yml --check --ask-vault-pass

Expected output: changed=0 everywhere if the firewall is in the desired state. changed=N on host X means the desired state differs from the actual state on that firewall.

This is the cheapest drift detection — the playbook is the auditor. The cost is small (the run completes in seconds on a typical estate) and the audit comes free with the regular scheduled run.

$ ansible-playbook -i inv/prod.yml playbooks/all.yml --check 2>&1 | tee drift-report-$(date +%F).log | tail -15

PLAY RECAP *********************************************************************
fw-dc1-edge-01        : ok=44  changed=0   unreachable=0  failed=0  skipped=0
fw-dc1-dmz-01         : ok=44  changed=2   unreachable=0  failed=0  skipped=0
fw-dc2-edge-01        : ok=44  changed=0   unreachable=0  failed=0  skipped=0
fw-dc2-dmz-01         : ok=44  changed=0   unreachable=0  failed=0  skipped=0
fw-stage-01           : ok=44  changed=0   unreachable=0  failed=0  skipped=0

Explicit drift detection

Idempotency-based detection works for objects the playbook manages. It does not detect:

• Objects the playbook does not manage. A rule the playbook has no task for is invisible to the playbook’s check. The search returns it; the playbook ignores it.
• Objects on firewalls not in the inventory. A new firewall added directly via the GUI is not in inventory; the playbook never runs against it.
• Out-of-band changes between runs. A manual edit at 02:00 and the next Ansible run at 03:00 reports the edit; the hour between, however, is invisible.

For these cases, an explicit drift detection play that reads the API and reports what is not in the playbook’s desired state is the right tool. The play:

- name: Detect drift on all firewalls
  hosts: firewalls
  connection: local
  gather_facts: false
  tasks:
    - name: List the automation rules
      ansibleguy.opnsense.list:
        target: rule
      register: actual_rules

    - name: Report rules the playbook does not manage
      ansible.builtin.debug:
        msg: >
          Unmanaged rules on {{ inventory_hostname }}: {{
            actual_rules.data
            | rejectattr('description', 'in', playbook_rule_descriptions)
            | map(attribute='description') | list
          }}
      when: >
        actual_rules.data
        | rejectattr('description', 'in', playbook_rule_descriptions)
        | list | length > 0

ansibleguy.opnsense.list is the read-only counterpart to the write modules: target: rule returns the automation rules, target: alias the aliases, and so on for around a hundred object types. Using it rather than a hand-built uri task means the drift check reads the same fields, by the same names, that the write tasks use — so a comparison against the playbook’s desired state is comparing like with like.

The playbook_rule_descriptions variable is the list of rule descriptions the playbook manages. Anything the firewall holds whose description is not in that list is drift.

One limitation is worth stating plainly. This detects rules the playbook does not know about; it does not detect a rule the playbook does know about whose other fields have been edited — a managed rule whose destination someone widened still has its expected description. Catching that is the job of the --check --diff run described above, and the two checks are complementary rather than alternatives.

What counts as drift

Not every difference between desired and actual is a problem. Three categories:

1. Real drift. A rule, alias, or NAT entry that exists on the firewall but not in the playbook. This is the operator-edited or emergency-added change. Real drift is the category that needs remediation.
2. Intentional manual change. A new firewall rule added by the on-call operator to address an incident, with a CHG ticket ID in the description. The playbook does not know about it but the change is correct and traceable. The discipline is to add the rule to the playbook within 24 hours so it does not drift again next time.
3. Out-of-scope configuration. A rule from a feature the playbook does not manage (a temporary test rule, a one-off experimental change). The playbook should not be aware of it because the team has not committed to owning it long-term.

The drift detector should report all three, but only the first is auto-remediable. Categories 2 and 3 require the human review.

Periodic drift scans

A production drift scan runs:

• Daily, at a low-traffic time (e.g. 03:00 local).
• In --check mode, so it reads but does not write.
• With a known-good baseline (the playbook’s variables as the desired state).
• Outputs to a log file or a chat channel, not just stdout.

The cron-equivalent for Ansible is a runner: AWX/Automation Controller with a schedule, or a cron job that calls ansible-playbook. The runner reads the API on schedule, runs the check, and reports.

$ ansible-playbook -i inv/prod.yml playbooks/drift-detect.yml --check 2>&1 | grep -E "drift|TASK|PLAY RECAP" | tee /var/log/opnsense-drift-$(date +%F).log

TASK [report rules not in the playbook] *****************************************
ok: [fw-dc1-edge-01] => { "msg": "No drift on edge-01." }
ok: [fw-dc1-dmz-01] => { "msg": "No drift on dmz-01." }
ok: [fw-dc2-edge-01] => { "msg": "No drift on edge-01." }
ok: [fw-dc2-dmz-01] => { "msg": "No drift on dmz-01." }
ok: [fw-stage-01] => { "msg": "No drift on stage-01." }

PLAY RECAP *********************************************************************
fw-dc1-edge-01    : ok=2   changed=0   unreachable=0  failed=0
fw-dc1-dmz-01     : ok=2   changed=0   unreachable=0  failed=0
fw-dc2-edge-01    : ok=2   changed=0   unreachable=0  failed=0
fw-dc2-dmz-01     : ok=2   changed=0   unreachable=0  failed=0
fw-stage-01       : ok=2   changed=0   unreachable=0  failed=0

Responding to drift

When the scan reports drift, three response paths:

1. Add the rule to the playbook. For an intentional manual change (an emergency rule with a CHG ticket), add the rule to the playbook’s group_vars/ variables, commit, and the next scheduled run will reconcile.
2. Remove the rule from the firewall. For real drift — a rule that should not be there — an ansibleguy.opnsense.rule task with state: absent, matching the rule the same way the managing tasks do, deletes it. Run that only after the drift scan has been reviewed by a human.
3. Document and ignore. For an out-of-scope rule (a temporary test), add it to an ignored_rules list in group_vars/ so the drift scan does not keep reporting it. The discipline: even ignored rules are listed in version control.

The wrong response: re-run the change playbook without understanding the drift. The playbook’s idempotency will correct object states that match its variable, but it will not remove rules it does not know about — that requires explicit state: absent or a remediation play.

The playbook is the source of truth

The discipline that protects the playbook’s authority:

• Manual edits require a follow-up to the playbook. A rule added by hand is added to the playbook within 24 hours. The CHG ticket ID connects them.
• *.bak backups are not the source of truth. The playbook is.
• An emergency change is allowed, but the playbook catches up.

A team that does not keep the playbook up to date will find that the playbook reports drift more often than it applies changes. The drift becomes the steady state. At that point, the playbook is no longer the source of truth — the firewall is — and the playbook’s value has evaporated.

Summary

• Idempotency-based drift detection: re-run the playbook in --check and read the changed count. Free drift reports on every scheduled run.
• Explicit drift detection: read the API and compare to the playbook’s desired state. Catches objects the playbook does not manage.
• Drift is in three categories: real (needs remediation), intentional manual (needs to be added to the playbook), out-of-scope (documented, ignored).
• A daily scan, in --check mode, with output to a log file or chat channel, is the production pattern.
• Drift correction is review-then-act, not act-then-review. The playbook’s authority depends on the team keeping it up to date.