Config drift detection - keeping reality aligned with declared state · Linux

Configuration drift is the silent killer of infrastructure. Hosts that started identical end up different; nobody knows when or why. Drift detection catches this.

What is drift?

Drift is any change to a host’s configuration that is not recorded in the desired state:

A user manually edits /etc/ssh/sshd_config.
A package is installed ad-hoc.
A service is enabled but should not be.
A file’s permissions change.

Drift is dangerous because the desired state is no longer true, but the CM system thinks it is.

Detect drift with Ansible

# Check mode: report what would change without making changes
ansible-playbook -i prod site.yml --check --diff

Output:

TASK [Configure nginx]
--- before
+++ after
@@ -1,3 +1,3 @@
 worker_processes 1;
-listen 80;
+listen 8080;

changed: [web01]

The diff shows exactly what is drifted.

Schedule drift checks

# /etc/cron.d/ansible-drift-check
0 2 * * * root install -m 0600 -o root -g root /dev/null /var/log/drift-check.log && /usr/bin/ansible-playbook -i /etc/ansible/prod /etc/ansible/site.yml --check --diff >> /var/log/drift-check.log 2>&1

Daily checks. Review the log weekly.

The install -m 0600 step is not decoration. Root cron runs with umask 022, so a shell redirection creates the log mode 0644 - readable by every local account on the host. The next section explains why that matters.

Add the file to logrotate with the same mode, or the first rotation undoes the fix:

# /etc/logrotate.d/drift-check
/var/log/drift-check.log {
    weekly
    rotate 8
    create 0600 root root
    missingok
}

--diff renders file contents, including secrets

--diff prints the full before-and-after text of every managed file. Any template that renders a database password, an API token, an LDAP bind credential or a private key writes that value in cleartext into whatever you redirect the run into. The file itself may be deployed mode: '0600'; the diff of its contents is not.

Two defences, and you need both:

# 1. Suppress output on every task that renders a secret
- name: deploy app config
  ansible.builtin.template:
    src: app.conf.j2
    dest: /etc/app/app.conf
    mode: '0600'
  no_log: true

# 2. Never let the drift log be world-readable, and never
#    ship it to a shared location without redaction
install -m 0600 -o root -g root /dev/null /var/log/drift-check.log

Without no_log, a nightly drift check copies every templated secret on the host into a log file, every night, for as long as the job runs. See linux-secrets-and-anti-patterns - “secrets in logs” is on that lesson’s leak-path list, and this is how it happens in practice.

Alert on drift

Run the check from the Ansible control node, and drive the alert off the changed count:

#!/usr/bin/env bash
# /usr/local/sbin/drift-check - runs on the control node
set -uo pipefail

out=$(ANSIBLE_STDOUT_CALLBACK=json ansible-playbook \
        -i /etc/ansible/prod /etc/ansible/site.yml --check 2>&1)
rc=$?

if [ "$rc" -ne 0 ]; then
    echo "drift-check FAILED rc=$rc"   # a broken check, not drift
    exit 2
fi

changed=$(printf '%s' "$out" | jq '[.stats[].changed] | add')
if [ "${changed:-0}" -gt 0 ]; then
    echo "DRIFT: $changed resource(s) differ from desired state"
    exit 1        # this is the signal monitoring alerts on
fi
exit 0

Exit 1 means drift, exit 2 means the detector itself is broken. Alert on both - a check that cannot run is not a clean check.

Without the JSON callback, parse the play recap instead:

drifted=$(printf '%s\n' "$out" | awk '/PLAY RECAP/,0' \
  | grep -oP 'changed=\K[0-9]+' | awk '{s+=$1} END{print s+0}')
[ "$drifted" -gt 0 ] && logger -t drift "drift on $drifted host-task(s)"

The exit code does not carry drift

The tempting version of this check registers the playbook result and alerts on the return code:

# WRONG - this alert can never fire
- hosts: all
  tasks:
    - name: Check for drift
      command: ansible-playbook -i prod site.yml --check
      register: drift_check
      failed_when: drift_check.rc != 0

ansible-playbook --check exits 0 even when every task in the run reports changed. Non-zero exit codes mean something else entirely: 2 for failed tasks, 4 for unreachable hosts, 1 for a usage or parse error. Drift lives in the changed= count of the play recap, never in the exit status. So this check reports “clean” exactly when drift exists, and pages only when Ansible itself breaks - the most dangerous shape a control can take, because the cron job succeeds every night and the dashboard stays green.

The second fault is where it runs. hosts: all executing command: ansible-playbook would require Ansible, the inventory and the SSH credentials to be installed on every managed node. Drift checks belong on the control node, targeting hosts, not on the hosts themselves.

Remediate

For critical drift (e.g. SSH config change), auto-remediate:

# Run the playbook for real, not in check mode
ansible-playbook -i prod site.yml --limit web01

For non-critical drift (e.g. cosmetic), log and review: manual decision.

Track drift over time

Store drift reports in a git repo or SIEM:

day=/var/log/drift-history/$(date +%Y%m%d)
install -d -m 0700 -o root -g root "$day"
install -m 0600 -o root -g root /dev/null "$day/$(date +%H%M).log"
ansible-playbook -i /etc/ansible/prod /etc/ansible/site.yml \
  --check --diff >> "$day/$(date +%H%M).log" 2>&1

The permissions carry more weight here than in the nightly job. A history directory keeps every diff indefinitely, so a single unredacted secret stays on disk - and in every host backup taken since - until someone notices. Prefer storing the drift summary (host, task, resource changed) rather than the raw diff, and keep the raw diff only where the secrets have been suppressed with no_log.

Trend analysis: a host with increasing drift over time suggests it is being managed manually, not by CM.

What to check

For each host, check:

Critical config files (/etc/ssh/sshd_config, /etc/pam.d/*, /etc/sudoers).
Installed packages (vs the desired set).
Running services (vs the desired set).
File permissions on sensitive files.
User accounts (no ad-hoc additions).
Sudo rules (no ad-hoc additions).