Lab: Produce a drift report from a fleet nobody has been watching

Objective

By the end of this lab you will have taken a fleet with five different manual changes on it and produced a report that names each drifted host, each drifted item, and the exact diff — while not flagging the two deviations that somebody deliberately approved. You will then expire one of those approvals and watch it reappear in the report.

Architecture

Four managed nodes with a convergence role, an exception register on the controller, and an audit play that never changes anything.

controller                                managed nodes
├── roles/baseline/          ─── audit ──▶ node1  clean
├── exceptions.yml                         node2  motd edited, sysctl changed
└── audit.yml  (--check --diff)            node3  package removed
                                           node4  approved exception (not drift)

Requirements

A controller with ansible-core 2.21.x.
Four managed nodes. Configuration and package drift can be reproduced in containers; the sysctl and service portions cannot, so B-nested — real VMs — is the honest mode for a complete run. A container-only run must drop the sysctl item and say so in the report.
SSH key access and become on each node.
No out-of-band access requirement: the audit is --check only and cannot change anything. Task 2, which deliberately introduces the drift, does change things, and Cleanup restores them.

Scenario

Nobody has run configuration management against this fleet for eleven months. It was applied once at build time and then people made changes by hand, some of them for good reasons.

The change board wants to re-enable automated convergence. Their entirely reasonable condition is: show us what would change, host by host, before you change anything. Applying the role blind would revert eleven months of hand-tuning including the two changes that were made deliberately during incidents.

Tasks

Task 1: Capture the pre-drift state

WORKDIR="$HOME/ansible-drift-lab"
mkdir -p "$WORKDIR"/{roles/baseline/{tasks,templates,defaults},reports}
cd "$WORKDIR"

inventory.yml:

fleet:
  hosts:
    node1: {ansible_host: 192.0.2.11}
    node2: {ansible_host: 192.0.2.12}
    node3: {ansible_host: 192.0.2.13}
    node4: {ansible_host: 192.0.2.14}
  vars:
    ansible_user: operator

# capture.yml
- name: Capture every item the audit will inspect
  hosts: fleet
  become: true
  gather_facts: true
  tasks:
    - name: Read the message of the day
      ansible.builtin.slurp:
        src: /etc/motd
      register: motd
      failed_when: false

    - name: Read the current sysctl values
      ansible.builtin.command: "sysctl -n net.ipv4.tcp_syncookies vm.swappiness"
      register: sysctls
      changed_when: false

    - name: List installed packages
      ansible.builtin.package_facts:

    - name: Write the capture to the controller
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          captured: {{ ansible_date_time.iso8601 }}
          motd_b64: {{ motd.content | default('') }}
          sysctls: {{ sysctls.stdout_lines | to_json }}
          rsync_installed: {{ 'rsync' in ansible_facts.packages }}
          chrony_installed: {{ 'chrony' in ansible_facts.packages }}
        dest: "{{ playbook_dir }}/reports/pre-drift-{{ inventory_hostname }}.yml"
        mode: '0600'
      delegate_to: localhost
      become: false

Read-only / Safecontroller— Reads state on four nodes and writes four files on the controller

$ ansible-playbook -i inventory.yml capture.yml

ls -l reports/
cat reports/pre-drift-node2.yml

Task 2: Introduce the drift

# drift.yml
- name: Introduce the drift this lab will detect
  hosts: fleet
  become: true
  gather_facts: false
  tasks:
    - name: node2 - somebody edited the motd by hand
      ansible.builtin.copy:
        content: |
          Welcome. Ask #platform before changing anything on this box.
          -- edited by hand, 2025-11
        dest: /etc/motd
        mode: '0644'
      when: inventory_hostname == 'node2'

    - name: node2 - somebody tuned swappiness during an incident
      ansible.posix.sysctl:
        name: vm.swappiness
        value: '1'
        state: present
        sysctl_set: true
        reload: true
      when: inventory_hostname == 'node2'

    - name: node3 - somebody removed rsync to free space
      ansible.builtin.package:
        name: rsync
        state: absent
      when: inventory_hostname == 'node3'

    - name: node4 - the approved exception
      ansible.builtin.copy:
        content: |
          RESTRICTED SYSTEM - PCI scope
          Access is logged. Approved deviation CHG-4471.
        dest: /etc/motd
        mode: '0644'
      when: inventory_hostname == 'node4'

Configuration changecontroller— Removes a package on node3, sets a sysctl on node2, rewrites two motd files

$ ansible-playbook -i inventory.yml drift.yml

Task 3: Write the baseline role and audit with it

roles/baseline/defaults/main.yml:

---
baseline_motd: |
  Authorised access only. All activity is logged.
  Managed by Ansible. Local edits will be reverted.

baseline_sysctls:
  net.ipv4.tcp_syncookies: '1'
  vm.swappiness: '10'

baseline_packages:
  - rsync
  - chrony

roles/baseline/tasks/main.yml:

- name: Ensure the message of the day matches the baseline
  ansible.builtin.copy:
    content: "{{ baseline_motd }}"
    dest: /etc/motd
    owner: root
    group: root
    mode: '0644'
  tags: [motd]

- name: Ensure kernel parameters match the baseline
  ansible.posix.sysctl:
    name: "{{ item.key }}"
    value: "{{ item.value }}"
    state: present
    sysctl_set: true
    reload: true
  loop: "{{ baseline_sysctls | dict2items }}"
  loop_control:
    label: "{{ item.key }}"
  tags: [sysctl]

- name: Ensure the baseline packages are installed
  ansible.builtin.package:
    name: "{{ baseline_packages }}"
    state: present
  tags: [packages]

audit.yml:

- name: Audit the fleet against the baseline
  hosts: fleet
  become: true
  gather_facts: true
  roles:
    - baseline

Read-only / Safecontroller— Check mode with diff. Predicts changes; makes none.

$ ansible-playbook -i inventory.yml audit.yml --check --diff

TASK [baseline : Ensure the message of the day matches the baseline] ************
ok: [node1]
--- before: /etc/motd
+++ after: /etc/motd
@@ -1,2 +1,2 @@
-Welcome. Ask #platform before changing anything on this box.
--- edited by hand, 2025-11
+Authorised access only. All activity is logged.
+Managed by Ansible. Local edits will be reverted.

changed: [node2]
changed: [node4]

TASK [baseline : Ensure kernel parameters match the baseline] *******************
ok: [node1] => (item=net.ipv4.tcp_syncookies)
ok: [node1] => (item=vm.swappiness)
changed: [node2] => (item=vm.swappiness)

TASK [baseline : Ensure the baseline packages are installed] ********************
ok: [node1]
changed: [node3]

PLAY RECAP *********************************************************************
node1                      : ok=4    changed=0    unreachable=0    failed=0
node2                      : ok=4    changed=2    unreachable=0    failed=0
node3                      : ok=4    changed=1    unreachable=0    failed=0
node4                      : ok=4    changed=1    unreachable=0    failed=0

Illustrative output

That recap is a drift inventory. changed=0 means converged. Every other number is a count of things that differ from the declared state.

Task 4: Capture the drift as data, not as scrollback

A human-readable recap is not a report. Capture the structured result:

cd "$HOME/ansible-drift-lab"

ANSIBLE_STDOUT_CALLBACK=default \
  ansible-playbook -i inventory.yml audit.yml --check --diff \
  > reports/audit-$(date +%Y%m%d-%H%M).txt 2>&1

tail -8 "$(ls -1t reports/audit-*.txt | head -1)"

For machine-readable output you need a structured stdout callback. Find out what this controller actually has before depending on one:

Read-only / Safecontroller— Lists the callback plugins available to this installation

$ ansible-doc -t callback -l

ansible.builtin.default        default Ansible screen output
ansible.builtin.minimal        minimal Ansible screen output
ansible.builtin.oneline        oneline Ansible screen output
ansible.builtin.tree           Save host events to files
...

Illustrative output

A bare ansible-core has almost no callbacks

On a controller with ansible-core and no collections installed, the structured callbacks people reach for — JSON output, JUnit XML, timing profiles — are simply not there. They ship in collections:

ansible-galaxy collection list

An empty result means you have ansible-core alone. Installing a collection to get a callback is a reasonable thing to do, and it is a dependency your controller build has to declare and pin.

Where you cannot add one, build the report inside the play instead — which is what the next task does, needs nothing beyond ansible-core, and gives you exactly the fields you want rather than the whole result tree.

Task 5: Build the report inside the play

The most portable approach registers each task’s result and renders a report from it:

# audit-report.yml
- name: Audit the fleet and produce a drift report
  hosts: fleet
  become: true
  gather_facts: true
  check_mode: true          # this play is ALWAYS an audit, never a change

  vars:
    exceptions_file: "{{ playbook_dir }}/exceptions.yml"

  tasks:
    - name: Load the approved exception register
      ansible.builtin.include_vars:
        file: "{{ exceptions_file }}"
      delegate_to: localhost
      become: false
      run_once: true

    - name: Check the message of the day
      ansible.builtin.copy:
        content: "{{ baseline_motd }}"
        dest: /etc/motd
        owner: root
        group: root
        mode: '0644'
      register: motd_result

    - name: Check the baseline packages
      ansible.builtin.package:
        name: "{{ baseline_packages }}"
        state: present
      register: pkg_result

    - name: Assemble this host's findings
      ansible.builtin.set_fact:
        findings: >-
          {{
            [
              {'item': 'motd',     'drifted': motd_result.changed},
              {'item': 'packages', 'drifted': pkg_result.changed}
            ] | selectattr('drifted') | list
          }}

    - name: Subtract the approved exceptions that are still in date
      ansible.builtin.set_fact:
        unapproved: >-
          {{
            findings | rejectattr('item', 'in', active_exceptions) | list
          }}
      vars:
        active_exceptions: >-
          {{
            (exceptions[inventory_hostname] | default([]))
            | selectattr('expires', 'ge', ansible_date_time.date)
            | map(attribute='item') | list
          }}

    - name: Write this host's report line
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          audited: {{ ansible_date_time.iso8601 }}
          drifted_items: {{ findings | map(attribute='item') | list | to_json }}
          unapproved_drift: {{ unapproved | map(attribute='item') | list | to_json }}
          status: {{ 'DRIFT' if unapproved | length > 0 else 'OK' }}
        dest: "{{ playbook_dir }}/reports/drift-{{ inventory_hostname }}.yml"
        mode: '0644'
      delegate_to: localhost
      become: false
      check_mode: false     # the report must be written even in an audit run

    - name: Fail the audit for hosts with unapproved drift
      ansible.builtin.assert:
        that: unapproved | length == 0
        fail_msg: >-
          {{ inventory_hostname }} has unapproved drift in:
          {{ unapproved | map(attribute='item') | join(', ') }}
        success_msg: "{{ inventory_hostname }} within policy"

exceptions.yml:

---
exceptions:
  node4:
    - item: motd
      reason: "PCI scope banner required by CHG-4471"
      approved_by: security-team
      expires: '2026-12-31'
  node2:
    - item: packages
      reason: "rsync removal pending capacity work, CHG-4502"
      approved_by: platform-lead
      expires: '2026-08-01'

Note the dates. node4’s exception is current; node2’s expired on 1 August 2026.

Task 6: Run the audit and read the report

cd "$HOME/ansible-drift-lab"
ansible-playbook -i inventory.yml audit-report.yml || true

for f in reports/drift-*.yml; do
  echo "--- $f"
  cat "$f"
done

Expected shape:

--- reports/drift-node1.yml
host: node1
drifted_items: []
unapproved_drift: []
status: OK

--- reports/drift-node2.yml
host: node2
drifted_items: ["motd"]
unapproved_drift: ["motd"]
status: DRIFT

--- reports/drift-node4.yml
host: node4
drifted_items: ["motd"]
unapproved_drift: []
status: OK

node4 drifted and is not flagged, because the drift is approved and the approval is in date. That distinction — between different and wrong — is the entire value of the exception register.

Task 7: Expire an exception and watch it reappear

Change node4’s expiry to a past date:

cd "$HOME/ansible-drift-lab"
sed -i "s/expires: '2026-12-31'/expires: '2026-01-31'/" exceptions.yml

ansible-playbook -i inventory.yml audit-report.yml || true
cat reports/drift-node4.yml

status: DRIFT. The deviation did not change; the approval did.

Validation

reports/pre-drift-*.yml exist for all four nodes and were written before drift.yml ran.
ansible-playbook -i inventory.yml audit.yml --check --diff reports changed=0 for node1 and non-zero for node2, node3 and node4.
The diff output for node2’s motd shows the hand-edited text being replaced by the baseline text.
No file on any node changed during the audit — confirm with a post-audit stat comparison against the capture.
audit-report.yml writes four files under reports/, and node4 reports status: OK with a non-empty drifted_items.
After expiring node4’s exception, the same play reports status: DRIFT for node4.
Running audit-report.yml without --check still changes nothing on any managed node, because check_mode: true is set on the play.

Expected Outcome

ansible-drift-lab/
├── audit.yml
├── audit-report.yml
├── capture.yml
├── drift.yml
├── exceptions.yml
├── inventory.yml
├── reports/
│   ├── audit-YYYYMMDD-HHMM.txt
│   ├── drift-node{1..4}.yml
│   └── pre-drift-node{1..4}.yml
└── roles/baseline/{defaults,tasks}/main.yml

A per-host report distinguishing drift from approved deviation, produced by the same role that would remediate it, with an exception register that expires. The managed nodes are exactly as they were before the audit ran.

Troubleshooting

ansible.posix.sysctl not found. It is in the ansible.posix collection, not ansible-core. ansible-galaxy collection install ansible.posix, or substitute a lineinfile on /etc/sysctl.d/ plus a command: sysctl -p — which is less good, and is a fair illustration of why collections matter.

Every host reports drift on the motd. The baseline string and the file differ by a trailing newline. YAML block scalars are precise about this: | keeps one trailing newline, |- strips it, |+ keeps all of them. --diff shows the difference as a \ No newline at end of file marker.

The audit reports changed for a package that is installed. Check whether the role uses state: latest rather than state: present. latest compares against the repository, so a host with an available update reports drift — which is honest but is a different question from “does the baseline package set exist”.

The report file is not written. check_mode: false is missing from the report task, so it is being simulated along with everything else.

selectattr('expires', 'ge', ...) raises a template error. The comparison is between strings, and it works only because ISO dates sort lexicographically. A date written 31/12/2026 breaks it silently by comparing wrongly rather than by failing. Keep dates ISO-8601.

The audit changed something. Then a task in the role has check_mode: false on it, or the module has no check-mode support and the role is doing something with command. Audit the role with the technique from the check-mode lab before trusting it.

Cleanup

The lab removed a package, changed a kernel parameter and rewrote two motd files. All of it is recorded in reports/pre-drift-*.yml and all of it must be restored.

# cleanup.yml
- name: Restore the pre-drift state
  hosts: fleet
  become: true
  gather_facts: false

  tasks:
    - name: Load this host's pre-drift capture
      ansible.builtin.include_vars:
        file: "{{ playbook_dir }}/reports/pre-drift-{{ inventory_hostname }}.yml"
        name: pre
      delegate_to: localhost
      become: false

    - name: Refuse to continue without a capture
      ansible.builtin.assert:
        that: pre.host is defined
        fail_msg: >-
          No pre-drift capture for {{ inventory_hostname }}. Do not guess at
          the original state - restore this host from your own records.

    - name: Restore the original message of the day
      ansible.builtin.copy:
        content: "{{ pre.motd_b64 | b64decode }}"
        dest: /etc/motd
        owner: root
        group: root
        mode: '0644'
      when: pre.motd_b64 | length > 0

    - name: Restore the original sysctl values
      ansible.posix.sysctl:
        name: "{{ item.0 }}"
        value: "{{ item.1 }}"
        state: present
        sysctl_set: true
        reload: true
      loop: "{{ ['net.ipv4.tcp_syncookies', 'vm.swappiness'] | zip(pre.sysctls) | list }}"
      loop_control:
        label: "{{ item.0 }}"

    - name: Reinstall rsync where it was present before the lab
      ansible.builtin.package:
        name: rsync
        state: present
      when: pre.rsync_installed | bool

    - name: Remove rsync where it was absent before the lab
      ansible.builtin.package:
        name: rsync
        state: absent
      when: not (pre.rsync_installed | bool)

Verify against the capture. Note the first step: capture.yml writes to the same filenames, so re-running it to check your work overwrites the evidence you are checking against. Copy first.

cp -a reports reports.pre-cleanup
ansible-playbook -i inventory.yml cleanup.yml
ansible-playbook -i inventory.yml capture.yml

for h in node1 node2 node3 node4; do
  echo "--- $h"
  diff <(grep -vE '^(captured|audited):' reports.pre-cleanup/pre-drift-$h.yml) \
       <(grep -vE '^(captured|audited):' reports/pre-drift-$h.yml) \
    && echo "$h restored"
done

Then remove the working directory:

rm -rf deletes the whole lab directory, including anything you added

The next block removes ~/ansible-drift-lab and everything under it, recursively and without confirmation. If you saved notes, an inventory you want to keep, or a playbook you extended, copy it out first — the step above does that for the lab’s own deliverables and knows nothing about yours.

The path is written in full and anchored to $HOME deliberately. A relative rm -rf run from the wrong directory is how people delete work they meant to keep, and the shell gives no second chance.

Confirm what you are about to delete before you delete it:

ls -la "$HOME/ansible-drift-lab"

mkdir -p "$HOME/ansible-lab-deliverables/drift"
cp -a "$HOME/ansible-drift-lab/exceptions.yml" \
      "$HOME/ansible-drift-lab/audit-report.yml" \
      "$HOME/ansible-lab-deliverables/drift/"

rm -rf "$HOME/ansible-drift-lab"

Under the hood

Check mode makes an audit possible; --diff makes it useful, and the two answer different questions.

changed is a boolean prediction from the module’s comparison. It tells you that current state and declared state differ. diff is a structured {before, after} pair the module returns alongside the result, which the callback renders as a unified diff. Without it you know a host drifted; with it you know how, which is the difference between a report a change board can act on and a list of hostnames.

Modules opt into diff support through the diff_mode attribute, and the support is not universal — ansible-doc command reports diff_mode: support: none, so a command-based check contributes a changed and nothing more.

check_mode: true at play level is worth knowing about as a design tool rather than a debugging flag. It is a declaration that the play is an observer, enforced regardless of invocation, and the only way past it is an explicit per-task check_mode: false that a reviewer can see. A repository where every audit-*.yml carries it has made “somebody ran the audit playbook wrong” impossible rather than unlikely.

What You Learned

The audit and the remediation are the same role. Only the invocation differs, which is what stops the audit drifting from what it audits.
--check gives you changed; --diff gives you the reason. A report without the diff is a list of hostnames.
An exception register turns “different” into “wrong or approved”. node4 drifted and was not flagged; the register, not the host, decided that.
Exceptions must expire. You expired one and watched the same unchanged host move from OK to DRIFT, which is the mechanism that stops a register becoming a permanent divergence list.
check_mode: true on the play makes an audit playbook safe by construction, and check_mode: false on the controller-side report task is the one deliberate exception.
Cleanup restores per host from a per-host capture. Running the baseline role would have normalised four hosts instead of restoring them, which is a different and unapproved change.

Objectives

Prerequisites