Detecting drift independently of the CM tool

linux-config-drift-detection builds a detector from the configuration management tool: run the playbook in check mode and count what would change. It is the right first detector and it has one structural limitation — the thing measuring reality is the same thing that defined reality.

A CM dry-run answers “does this host match what the playbook says”. It cannot answer “does this host match what the vendor shipped”, “has this file changed since we last looked”, or “is this host the same as its peers”. Those need different sources of truth, and they find different problems.

Three independent sources of truth

The blind spots are as important as the coverage, and they do not overlap much — which is the argument for having more than one.

Package verification: the Debian side

dpkg --verify (or -V) compares installed files against the metadata recorded when the package was unpacked. With no arguments it checks every installed package.

$ dpkg --verify openssh-server

??5?????? c /etc/ssh/sshd_config
missing     /usr/share/doc/openssh-server/NEWS.Debian.gz

The nine-character result string uses the same layout as rpm -V, which linux-dnf-and-rpm decodes in full: position 2 is mode, position 3 is the digest, . means the check passed and ? means it could not be performed. The c marks a conffile.

Run it fleet-wide and look at the shape of the output rather than the individual lines:

# Every installed package. Slow - schedule it, do not run it ad hoc.
sudo dpkg --verify 2>/dev/null | tee /var/tmp/dpkg-verify.txt

# Just the packages you care about, which is the practical form
for p in openssh-server sudo chrony rsyslog systemd; do
  echo "== $p"
  sudo dpkg --verify "$p" 2>/dev/null
done

# Non-conffile modifications: these are the findings.
# A changed conffile is expected; a changed binary is not.
sudo dpkg --verify 2>/dev/null | awk '$2 != "c"'

That last filter is the whole technique. Configuration files are supposed to be modified — that is what makes them configuration. A modified file that is not a conffile means a package-owned binary, library or unit file has been changed since installation, and there are very few benign explanations.

On the RPM side, the equivalent split:

# Same filter, same reasoning: the file-type column marks c for
# config, so dropping those leaves the unexpected changes
rpm -Va 2>/dev/null | awk '$2 != "c"' | head -40

# Just the security-relevant packages
rpm -V openssh-server sudo chrony

An integrity baseline

Package verification only covers files a package owns. Everything your configuration management wrote, everything an operator created, and everything an attacker dropped is outside it. A file integrity tool closes that gap by comparing against a snapshot you took yourself.

AIDE is the common choice. The workflow is three commands:

# 1. Build the baseline database from the rules in the config
sudo aide --init

# 2. Promote it. The exact paths come from database_in and
#    database_out in aide.conf - read them rather than assuming,
#    because distributions differ and some use a .gz suffix.
grep -E '^database(_in|_out|_new)?\s*=' /etc/aide/aide.conf

# 3. Compare the live filesystem against the baseline
sudo aide --check

Scope the rules deliberately. A baseline covering /var and /tmp produces thousands of expected changes a day, and a report nobody reads is the same as no report:

# /etc/aide/aide.conf - illustrative rule selection
# p permissions, i inode, n link count, u user, g group,
# s size, m mtime, sha256 content digest
Binlib = p+i+n+u+g+s+m+sha256
ConfFiles = p+u+g+s+m+sha256

/usr/bin        Binlib
/usr/sbin       Binlib
/etc            ConfFiles
!/etc/mtab
!/var/log

Peer comparison

The third source of truth costs nothing to build: hosts in the same role should look alike, so compare them to each other.

# Substitute your own values before running:
NODES="web01 web02 web03"

# Hash the same file on every node; a differing hash is drift
for n in $NODES; do
  printf '%-8s ' "$n"
  ssh -o BatchMode=yes "$n" "sha256sum /etc/nginx/nginx.conf" 2>&1
done

# Substitute your own values before running:
A=web01
B=web02

# Diff a whole directory tree between two peers, read-only
diff <(ssh -o BatchMode=yes "$A" "find /etc/nginx -type f -exec sha256sum {} + | sort -k2") \
     <(ssh -o BatchMode=yes "$B" "find /etc/nginx -type f -exec sha256sum {} + | sort -k2")

Knowledge check