Check mode as a drift detector · Ansible

You already own a drift detector. It is the playbook you use to configure production, run with --check --diff instead of for real.

Every task in it asks a host about a resource and reports whether the host matches. That is precisely what a drift report is. The work is not in building the detector; it is in scheduling it, reading its output correctly, and being honest about what it cannot see.

The scheduled drift run

Read-only / Safethe drift report itself— --check simulates rather than changes; --diff prints what would differ. Both flags together turn a convergence playbook into a comparison report. This is genuinely read-only for modules that support check mode - the exceptions are the subject of the next lesson.

ansible-playbook -i inventory/ site.yml \
--limit env_prod \
--check --diff

Note the severity badge. A --check run is READ-ONLY only if no task in the play carries check_mode: false — a task with that keyword executes for real during a check run. The next lesson deals with that properly; for now, treat “is this play safe to check-run?” as a question you must have answered before scheduling it.

The output is a normal play recap, where changed now means would change:

Read-only / Safea drift report, read as counts— Illustrative shape. In a check run, changed is the number of declarations that do not currently match on that host.

$ ansible-playbook -i inventory/ site.yml --limit env_prod --check --diff

PLAY RECAP *********************************************************************
web014.example.com         : ok=180  changed=0    unreachable=0    failed=0    skipped=6    rescued=0    ignored=0
web015.example.com         : ok=180  changed=0    unreachable=0    failed=0    skipped=6    rescued=0    ignored=0
web016.example.com         : ok=177  changed=3    unreachable=0    failed=0    skipped=6    rescued=0    ignored=0
api047.example.com         : ok=179  changed=1    unreachable=0    failed=0    skipped=6    rescued=0    ignored=0

Illustrative output

Two hosts converged, two not. web016 has three declared attributes that no longer match; api047 has one. That is a drift report, and it took a flag.

The exit code is not the signal

This trips up the first attempt at automating it, every time.

Read-only / Safea check run that found drift exits 0— Executed against ansible-core 2.21.3. The play reported changed=1 in check mode and the process exit status was 0.

$ ansible-playbook -i inventory.ini site.yml --check --diff; echo "exit: $?"

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

exit: 0

changed is not a failure. It never has been, in check mode or out of it, and a scheduled job that tests $? to decide whether the fleet has drifted will report “no drift” forever.

Parsing the recap with what core ships

Being precise about what is available, because the usual advice is wrong for ansible-core.

ansible-doc -t callback -l on 2.21.3 lists exactly five callback plugins: default, junit, minimal, oneline and tree. There is no json stdout callback in core — attempting ANSIBLE_STDOUT_CALLBACK=json produces [ERROR]: Could not load 'json' callback plugin. It lives in a collection, and if you want structured output you have to install one.

With core only, you parse the recap. It is crude and it works:

Read-only / Safedrift report to a machine-readable summary, core only— Runs the check, captures the recap, and emits one line per host with a non-zero would-change count. Exits 1 if any host has drifted, which is the exit code the scheduler should have been getting all along.

set -uo pipefail

REPORT=$(mktemp)
ansible-playbook -i inventory/ site.yml --limit env_prod --check --diff \
| tee "$REPORT"

awk '/PLAY RECAP/{recap=1; next}
   recap && /changed=[1-9]/ {
     host=$1
     for (i=1; i<=NF; i++) if ($i ~ /^changed=/) { split($i, a, "="); c=a[2] }
     printf "drift %s %s\n", host, c
     n++
   }
   END { exit (n > 0 ? 1 : 0) }' "$REPORT"

If you can install collections on the controller, a structured stdout callback is a better foundation than parsing text, and an automation platform gives you per-task results as a product feature. Say which one you are using in the runbook, because the parsing is the fragile part.

A single drift report is a to-do list. A series of them is a health metric, and the series is where the value is.

Three quantities worth recording per run:

Metric	What a rise means
Hosts with `changed > 0`	Drift is spreading across the fleet
Total `changed` across the fleet	Drift is deepening, possibly on few hosts
`changed` count for a specific task	One resource is being fought over — usually a vendor agent, or a genuine exception nobody recorded

The third is the most actionable and the one people never collect. A task that reports “would change” on the same forty hosts every night is not drift being detected; it is drift being observed and ignored, and it is almost always one of three things: a resource something else also manages, a declaration that is wrong, or a legitimate deviation with no mechanism to record it.

What check mode cannot see, and the tools that can

Check mode compares the declared surface. Category-4 drift — changes in areas the automation declares nothing about — is invisible to it by construction.

The complementary technique is to compare the host against something that is not your declaration: the package manager’s own record of what it installed.

Read-only / Safepackage verification on Debian and RPM systems— Both report files that differ from what the package shipped. Read-only. Run them via Ansible with changed_when: false so they never report a change.

# Debian/Ubuntu: md5sum check against the dpkg database
dpkg --verify

# a single package
dpkg --verify nginx-common

# RPM systems: a wider set of attributes
rpm -Va

dpkg --verify is much weaker than rpm -V, and its own man page says so

Read the dpkg man page carefully before relying on this. Two limitations, both quoted from it:

“Currently the only functional check performed is an md5sum verification of the file contents against the stored value in the files database. It will only get checked if the database contains the file md5sum.” So mode, owner and group changes are not detected, and a file with no recorded md5sum is not checked at all.
“This is only an integrity check and should not be considered as any kind of security verification.” An attacker who can change a file can change the database.

rpm -V checks a wider set — size, mode, digest, owner, group, mtime and capabilities — so the same command on the two families gives materially different coverage. debsums is a separate Debian package that covers similar ground to the md5sum check, including configuration files.

None of these is file integrity monitoring. If you need tamper detection, that is a different tool with a different threat model, and this is not it.

Read-only / Safepackage verification as a fleet-wide ad-hoc sweep— changed_when: false keeps the audit honest - this task inspects and never claims to have changed anything. failed_when: false is needed because a non-zero return code here means findings, not an error.

- name: verify installed packages against the package database
hosts: env_prod
gather_facts: true
tasks:
  - name: run package verification
    ansible.builtin.command:
      cmd: dpkg --verify
    register: pkgverify
    changed_when: false
    failed_when: false
    when: ansible_facts['pkg_mgr'] in ['apt']

  - name: report hosts with modified package files
    ansible.builtin.debug:
      msg: "{{ pkgverify.stdout_lines }}"
    when: pkgverify.stdout | default('') | length > 0

Note that this play uses ansible.builtin.command, which means it behaves differently under --check — the next lesson is entirely about that, and this task is one of the cases it covers.

Under the hood

Check mode is not a simulation layer that Ansible wraps around modules. It is a flag passed down to each module, and each module decides what to do with it: compute what it would change, report changed accordingly, and return without acting.

That is why check-mode support is per-module and why the module documentation carries a check_mode attribute — the behaviour is implemented once per module, not once in the engine. ansible-doc shows support: full, partial or none for each.

The practical consequence for a drift report is that its coverage is exactly the set of tasks whose modules implement check mode. A play made of copy, template, file, package and service produces a comprehensive report. A play with a dozen command tasks produces a report with a dozen holes in it, and nothing in the recap distinguishes “matched” from “not examined” — the second appears as skipped, in a column most people do not read.

What happens if the drift report itself drifts?

The scheduled check run executes the playbook as it is today, against the fleet as it is today. When someone changes the playbook, the report changes meaning, and nothing announces it.

Add a declaration and every host reports drift on the following night — not because anything on the hosts changed, but because the declared surface grew. Remove a task and hosts silently stop being checked for that resource. Both look like drift movement, and neither is.

The fix is to record the commit SHA of the content each report was generated from, alongside the counts. A spike in drift that coincides with a content change is a different investigation from a spike that does not, and without the SHA you cannot tell them apart. It is one line in the job and it saves an afternoon roughly once a quarter.

The scheduled drift run

The exit code is not the signal

Parsing the recap with what core ships

Trending it

What check mode cannot see, and the tools that can

Knowledge check