Package and version drift across a fleet

Drift programmes concentrate on /etc, because that is where the configuration lives and where the tooling points. But two hosts with byte-identical configuration and different package versions behave differently, and the difference is invisible to every file-based check.

Package drift is also the kind that shows up in a security review rather than an outage: one host still on the vulnerable version, one host that never reboots, one host with an extra repository nobody remembers enabling.

Comparing package sets across hosts

The primitive is a sorted list of name and version, which diffs cleanly:

# Substitute your own values before running:
NODES="web01 web02 web03"
OUT=/var/tmp/pkgdrift

mkdir -p "$OUT"
for n in $NODES; do
  ssh -o BatchMode=yes "$n" \
    "dpkg-query -W -f='\${binary:Package} \${Version}\n' | sort" > "$OUT/$n.txt"
done

# What differs between two peers
diff "$OUT/web01.txt" "$OUT/web02.txt"

# RPM-family equivalent
rpm -qa --qf '%{NAME} %{VERSION}-%{RELEASE}\n' | sort

Three shapes appear in the output, and they mean different things:

Holds: the drift that hides inside patching

A held package is excluded from upgrades. It is a legitimate tool — pinning a database or a kernel while an incompatibility is investigated — and it becomes a defect the moment nobody removes it.

The reason holds are dangerous is that they are invisible to every report that matters. Patching succeeds. The patch run reports no errors. The compliance dashboard is green because the run completed. And the held package stays on the vulnerable version indefinitely.

$ apt-mark showhold

linux-image-generic
postgresql-15
openssl

# RPM-family: dnf versionlock, if the plugin is installed
dnf versionlock list

Repository drift

Where a host gets its packages from is configuration that no package list reveals. A third-party repository enabled on three hosts out of forty means those three receive different versions, on a different schedule, signed by a different key.

# What is configured, including drop-ins and the deb822 format
grep -rhE '^\s*(deb|deb-src|URIs:)' \
  /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null | sort -u

# What apt would actually use, with priorities - this is the
# authoritative answer, because pinning can override the list
apt-cache policy

# Substitute your own values before running:
PKG=nginx

# For one package: which repository would supply it, and at what version
apt-cache policy "$PKG"

apt-cache policy is the one that matters, because a repository can be present in the list and outranked by a pin, or absent from the list and reached through a proxy. linux-third-party-repository-risk covers the pinning mechanics; the drift question here is narrower and answerable in one loop: does every host in this role produce the same apt-cache policy output?

Installed is not running

The last category is the one that catches people in audits. A package upgrade replaces files on disk. It does not replace the code that a long-running process already mapped into memory, and it certainly does not replace a running kernel.

A fleet where the patch run completed on every host can still have twelve hosts running the vulnerable library, because the service was never restarted.

$ sudo needrestart -b -r l

NEEDRESTART-VER: 3.11
NEEDRESTART-KCUR: 7.0.0-27-generic
NEEDRESTART-KEXP: 7.0.0-29-generic
NEEDRESTART-KSTA: 3
NEEDRESTART-SVC: nginx.service
NEEDRESTART-SVC: postgresql@15-main.service

Read the three kernel fields together. KCUR is the running kernel, KEXP is the kernel that would be used after a reboot, and KSTA is the verdict — a value other than the “current kernel is the expected one” state means the host is running older code than it has installed. The SVC lines are services still using replaced libraries.

Simpler checks that need no extra tooling:

# Running kernel versus newest installed kernel
uname -r
dpkg-query -W -f='${Package} ${Version}\n' 'linux-image-*' 2>/dev/null | sort -V | tail -3

# Debian family: set by unattended-upgrades and by kernel packages
ls -l /var/run/reboot-required /var/run/reboot-required.pkgs 2>/dev/null \
  || echo "no reboot flagged"

# Processes still holding deleted library files - the generic form,
# independent of distribution tooling
sudo lsof -nP +L1 2>/dev/null | awk '/\.so/ {print $1, $2}' | sort -u | head

Recording package drift as an inventory, not an alert

Package drift is rarely urgent and almost always cumulative, so the useful form is a table refreshed on a schedule rather than a page in the night:

FLEET PACKAGE DRIFT - role: web - generated 2026-08-11
| Host  | pkgs differing from role baseline | holds | 3rd-party repos | reboot pending |
|-------|-----------------------------------|-------|-----------------|----------------|
| web01 | 0                                 | -     | -               | no             |
| web02 | 0                                 | -     | -               | yes (kernel)   |
| web03 | 4                                 | nginx | nginx-upstream  | no             |

web03 is the finding, and it is a single row rather than an incident: an extra repository, a hold that stops the package being patched, and four version differences that follow from both. None of those would appear in a configuration file diff, and all three have the same root cause.

Knowledge check