How drift accumulates - the eight ways hosts stop matching

linux-drift-remediation covers what to do once drift is found. This lesson is about where it comes from, because the remediation you choose depends entirely on the mechanism — and most teams can name only one of the eight.

The default mental model is that drift means somebody edited a file by hand. That is one source, and in a mature environment it is not the largest.

The eight mechanisms

The right-hand column is the important one. Half of these are invisible to the dry-run that most teams use as their only detection method, which is why linux-detecting-drift-independently exists as a separate lesson.

Build-time divergence

Two hosts built four months apart from “the same” process are not the same host. Between the two builds, the distribution released point updates, the mirror advanced, a base image was rebuilt, a firmware version changed, and a default in the installer moved.

Configuration management converges what it manages, so it papers over this convincingly: both hosts pass every check, and both have different kernels, different library versions, and different values for every setting nobody thought to declare.

Package upgrades rewrite configuration

This is the mechanism that surprises people most, because nobody made a change: an unattended security upgrade ran at 06:00 and the configuration is now different.

Debian-family packages mark configuration files as conffiles. When a package upgrade ships a new version of a conffile that the administrator has modified, dpkg has to choose, and the outcome depends on how the upgrade was invoked:

• Interactively, it prompts, and a human decides.
• Non-interactively with --force-confold, your version is kept and the new upstream file is written alongside as .dpkg-dist.
• Non-interactively with --force-confnew, the upstream version replaces yours and your file is saved as .dpkg-old.

Unattended upgrades run non-interactively, so one of the last two happens silently on every host, every time.

# Which files does the package system consider configuration?
dpkg-query -W -f='${Conffiles}\n' openssh-server

# Leftovers from upgrade decisions - each one is a drift event
sudo find /etc -name '*.dpkg-dist' -o -name '*.dpkg-old' \
  -o -name '*.dpkg-new' -o -name '*.ucf-dist' 2>/dev/null

# RPM-family equivalents
sudo find /etc -name '*.rpmnew' -o -name '*.rpmsave' 2>/dev/null

$ sudo find /etc -name '*.dpkg-dist' -o -name '*.dpkg-old' 2>/dev/null

/etc/ssh/sshd_config.dpkg-dist
/etc/logrotate.d/rsyslog.dpkg-dist
/etc/sysctl.conf.dpkg-old

Each of those files is a decision the package system made on your behalf. A .dpkg-dist means the upstream default has moved away from your file and you have not looked at the difference; a .dpkg-old means your customisation was replaced and is now sitting in a backup file that nothing reads.

New defaults without a changed file

Subtler again: the file did not change and the behaviour did. An upstream project changes a compiled-in default, and the setting you never declared because “the default is fine” is now a different default.

Nothing in /etc is different, so no file-based drift detection sees it, and a CM dry-run reports no changes. The only way this becomes visible is by reading the effective runtime value rather than the configuration file, which is linux-runtime-vs-on-disk-drift.

systemd makes one class of this visible, because it tracks which shipped units have been overridden:

$ systemd-delta --type=extended,overridden

[EXTENDED] /usr/lib/systemd/system/nginx.service -> /etc/systemd/system/nginx.service.d/limits.conf
[OVERRIDDEN] /etc/systemd/system/rsyslog.service -> /usr/lib/systemd/system/rsyslog.service

2 overridden configuration files found.

[OVERRIDDEN] means a file in /etc completely replaces the vendor unit — so vendor improvements to that unit will never reach this host. [EXTENDED] is the healthier pattern: a drop-in that adds to the vendor unit and continues to inherit its changes.

Partial rollouts

A wave-based deployment is drift by design, and correctly so. The problem is not the divergence during the rollout; it is the rollout that stopped.

A canary that was never promoted, a wave that failed and was left alone, a batch aborted by max_fail_percentage and then forgotten — each leaves a subset of the fleet on a different version indefinitely. Six weeks later nobody remembers that the difference was intentional, and the hosts look like unexplained drift.

Drift that configuration management creates

The last mechanism is the one nobody looks for, because it comes from the tool that is supposed to prevent drift.

• Templates with host-specific inputs. A template that renders a value from a fact — memory size, CPU count, an IP — produces a different file per host by design. That is correct, and it means “identical” was never the goal, so a comparison between hosts will always show differences that are not drift.
• Conditional tasks that stopped applying. A task guarded by when: ansible_distribution_major_version == '11' silently stops doing anything after an OS upgrade. The host is no longer configured, CM reports success, and nothing failed.
• Removed tasks leave their effects behind. Deleting a task from a playbook does not undo what it did. The file it created stays, the service it enabled stays enabled, and CM no longer knows either exists.

That third one is the most common and the least intuitive. Configuration management describes what should be present; it has no memory of what it used to make present. Every task ever deleted from your playbooks has left artefacts on every host that ran it, and those artefacts are now unmanaged files — which is the subject of linux-the-limits-of-drift-detection.

Knowledge check