Drift, and what convergence actually means

Part I introduced drift as the reason machines that were built identically stop being identical. This part is about doing something about it, and that starts with being precise about three words that get used loosely.

The vocabulary, tightened

Declared state is what your automation says a host should look like: every attribute that appears as a module argument somewhere in the code that runs against that host. Not what you intend, not what the runbook says — what a task actually asserts.

Actual state is what is on the machine.

Drift is the set of attributes where the two disagree, restricted to attributes that are declared. An attribute nobody declares cannot drift, in the sense that nothing will ever notice.

Convergence is the condition where, for a given host and a given declaration, actual matches declared everywhere. Operationally: a run of that playbook reports changed=0.

The declared surface is the total set of attributes the automation declares — the boundary of what any of this can see.

The limit that governs this whole part

Write it on the wall:

A configuration-management tool reporting “no changes” only proves it checked what it manages.

ok=42 changed=0 is a strong statement — forty-two declarations verified on this host — and it is a statement about forty-two things. A production Linux host has tens of thousands of files, hundreds of packages, dozens of services, a kernel command line, sysctls, firewall rules, cron entries, user accounts and mounted filesystems.

The unmanaged remainder is where drift lives, because it is the only place drift can live undetected. The managed portion gets corrected by the next run. The unmanaged portion accumulates forever.

A clean run is not an audit. It is a clean run.

Classifying any attribute

Every attribute of every host is in one of four states, and the useful thing is that the classification is a decision tree you can walk:

flowchart TD
  A["An attribute of a host<br/>(a file, a package, a service, a sysctl)"] --> B{"Does the automation<br/>declare it?"}
  B -->|"no"| C["UNMANAGED<br/>no run will ever report on it"]
  B -->|"yes"| D{"Does actual match<br/>declared?"}
  D -->|"yes"| E["CONVERGED<br/>the next run reports ok"]
  D -->|"no"| F{"Is there a recorded<br/>exception for this host?"}
  F -->|"yes"| G["APPROVED DEVIATION<br/>expected, owned, with an expiry"]
  F -->|"no"| H["DRIFT<br/>needs a decision, not a reflex"]

Two of those four boxes are the ones people skip.

UNMANAGED is skipped because it produces no output. Nobody builds a report of things they never declared.

APPROVED DEVIATION is skipped because most estates have no mechanism for it, so every legitimate difference shows up as drift, gets “corrected” by a run, gets manually re-applied by the person who needed it, and reappears the next night. Lesson 6 builds the mechanism.

Four sources of drift

They differ in how they arrive and — more importantly — in whether automation can see them at all.

1. Manual change

Somebody edits a file, installs a package, or changes a sysctl by hand. The 02:00 incident fix is the canonical case, and the reason it persists is that the fix worked: nobody who has just stopped an outage feels inclined to go and reconcile it into a role at 03:30.

Detectable if the attribute is declared. Invisible if not — and incident fixes are disproportionately in the unmanaged parts of the system, because those are exactly the parts nobody had thought about before.

2. A failed run leaving partial state

A play that fails at task 30 of 60 leaves a host with the first thirty tasks applied. This is not drift from a previous state; it is a host in a state that no declaration describes — half of a new configuration and half of an old one.

Fully detectable by re-running, which is why “re-run the play” is the right first response to a failed run, and why idempotence is what makes that safe.

3. Upstream change

A package update ships a new default config. A distribution changes a systemd unit. A vendor agent rewrites its own configuration file on restart. Nobody touched the host, and the host changed.

Detectable if declared, and this is the category most likely to hit every host at once, which makes it look like an automation bug rather than drift.

4. Gaps where automation declares nothing

The largest source and the only one that is structurally undetectable. A directory nobody templated, a service nobody declared, a firewall rule somebody added out of band.

Only a comparison against something other than your own declaration — package verification, file integrity, a golden image diff — can find this class. Lesson 2 and lesson 3 are about exactly where that boundary falls.

Idempotent is not converged

These get conflated constantly and they are properties of different things.

Idempotence is a property of the playbook. It means running it twice against an unchanged host produces no change on the second run. You test it in CI, once, on a disposable host. It is either true of the code or it is not.

Convergence is a property of the estate at a moment. It means the declared state and the actual state currently agree on these hosts. It was true last night and may not be true now.

A perfectly idempotent playbook that has not been run for six weeks tells you nothing about the fleet. And a fleet that converged last night has drifted by whatever happened since — which on a busy estate is not nothing.

Measuring the declared surface

You cannot reason about coverage you have not looked at. Two cheap readings:

ansible-playbook -i inventory/ site.yml --limit web014.example.com --list-tasks

ansible-playbook -i inventory/ site.yml --limit web014.example.com \
--list-tasks | grep -c 'TAGS:'

grep -rhoE '(dest|path):[[:space:]]*[^{[:space:]}]+' roles/ \
| sed 's/.*:[[:space:]]*//' | sort -u

A site playbook with 180 tasks that writes 24 files under /etc is a real and useful declared surface. It is also, on a typical host, a small fraction of /etc, and knowing the ratio is what keeps “the fleet is converged” an honest sentence.

Knowledge check