Drift, and how a snowflake is actually made

Ask an operations team whether their ten web servers are identical and you will usually get “yes”. Ask how they know, and the answer changes shape: they were built from the same image, or the same runbook, or by the same person in the same week.

That is a statement about history, not about the current state of the disks. It was true at build time. Nobody has checked since, and the things that would have changed it do not announce themselves.

The gap between “they were built the same” and “they are the same” is drift, and it is the operational problem this whole course exists to address. Ansible is one answer to it. It is not the only one, and this part deliberately does not assume it is the right one for your estate.

The vocabulary

These four words are used precisely throughout the course. Getting them straight now saves ambiguity later, because two of them are routinely used to mean the other two.

Drift is the divergence of a host’s actual state from the state it is supposed to be in. It is a property of a host relative to an intention. If you have no stated intention, you do not have drift — you have an unknown state, which is worse, because you cannot even name the delta.

A snowflake is a host whose current state cannot be reproduced from any written source. Rebuilding it means either copying the disk or reverse-engineering it. The name is Martin Fowler’s, and the point of it is uniqueness rather than badness: a snowflake may be working perfectly. Its defect is that it is not reproducible.

Convergence is the act of moving a host from its actual state toward the intended state. A convergence run reports what differed and what it corrected. This is what a configuration management tool does when it runs; a run that changes nothing is a run that found no divergence.

An exception is a deliberate, recorded difference. Host db01 has vm.swappiness=1 where the rest of the fleet has 60, because the database vendor requires it, and that is written down somewhere the automation reads. An exception is not drift. The distinction matters enormously in practice: a fleet with fifteen recorded exceptions is managed, and a fleet with fifteen unrecorded ones is fifteen snowflakes wearing a uniform.

How a snowflake is actually made

Nobody sits down to create a snowflake. Every mechanism below is a sensible act by a competent person under the constraints of the moment. That is precisely why the problem is durable — you cannot fix it by telling people to be more careful, because nobody was being careless.

The change at 02:00 that never went back

A production incident at 02:00. The on-call engineer finds that nginx is refusing connections because the worker processes have hit the open file limit. They edit /etc/security/limits.conf on the affected host, raise nofile to 65535, restart the service, service recovers, they go back to bed.

Everything about that is correct. The fix was right, it was fast, and it worked.

What did not happen is the second half: the change going back into whatever builds the host. There is a ticket, maybe. It says “raised file limits on web03”. It does not say that web01, web02 and web04 still have the default, and it does not update the build. Six months later nobody remembers, and web03 is now a host that survives a traffic spike its three siblings do not.

The package installed to test something

A developer needs tcpdump to diagnose an intermittent timeout. Installing it on app02 is a two-second decision and entirely reasonable. Removing it afterwards is a decision nobody makes, because nothing prompts it.

app02 now has a package its siblings lack. On its own this is harmless. It matters because of what it implies: the set of installed packages on your hosts is not a set you control, and any statement that begins “our application servers have exactly these packages” is now false in a way you cannot detect without checking every host.

The vendor engineer

A storage vendor’s support engineer joins a screen share to diagnose a latency problem. They set vm.dirty_ratio, adjust a queue scheduler, and possibly add a udev rule. It works. They leave.

The change is now in the running kernel and possibly in /etc/sysctl.d/. Nobody on your team wrote it, nobody on your team can explain it, and there is a real chance nobody on your team knows it is there. Reversing it at the next incident becomes a research project.

The rollout that stopped halfway

A change was pushed to all forty hosts. It reached thirty-one before the deployment tool lost its connection to the remaining nine. Those nine were never revisited, because the outcome recorded was “deployment completed with errors” and the errors scrolled past.

This one is different from the others in an important way: the drift was created by automation. Automation reduces drift caused by humans and introduces a new class caused by partial runs. The course returns to this repeatedly — a run that stops halfway is the normal failure mode, not an exotic one.

The build that was never identical

Two hosts installed three months apart from the same “latest stable” image are not the same host. The package repository moved. A kernel point release landed. A dependency picked up a newer minor version.

No convergence run will remove this, because a convergence run enforces what you declared, and you never declared the version of every transitive dependency. This is the drift that immutable infrastructure addresses well and configuration management addresses poorly — a point that lesson 6 develops rather than glossing.

Drift has a direction and a rate

The mechanisms above are not one-off events. They are a process, and the useful mental model is a rate rather than an incident.

flowchart LR
  subgraph T0["Build day"]
    A1["web01"]
    A2["web02"]
    A3["web03"]
  end
  subgraph T6["Month 6"]
    B1["web01<br/>+ tcpdump"]
    B2["web02<br/>identical"]
    B3["web03<br/>+ nofile 65535"]
  end
  subgraph T12["Month 12"]
    C1["web01<br/>+ tcpdump<br/>+ kernel 6.8"]
    C2["web02<br/>+ kernel 6.8"]
    C3["web03<br/>+ nofile 65535<br/>+ vendor sysctl<br/>kernel 6.5 pinned"]
  end
  A1 --> B1 --> C1
  A2 --> B2 --> C2
  A3 --> B3 --> C3

Three hosts that were identical on build day. By month six one has an extra package and one has a different resource limit. By month twelve the kernel has moved on two of them and been pinned on the third, and web03 now differs from its siblings in three separate dimensions, one of which — the pinned kernel — will make it behave differently under memory pressure.

The important reading is not “web03 is broken”. It probably is not. The important reading is that the divergence is monotonic: nothing in this picture ever brings the hosts back together. There is no force acting in the direction of convergence unless somebody supplies one. Every one of the mechanisms adds difference; none removes it.

That is the whole argument for configuration management stated as cleanly as it can be. Not “manual work is bad” — a convergence mechanism is a restoring force in a system that otherwise has none.

Why it is invisible until an incident

Drift produces no symptom while conditions are normal. That is not a coincidence — it is definitional. If a divergence caused a visible problem, somebody would have fixed it and it would not still be there.

The drift that survives is exactly the drift that is harmless under current load, current traffic patterns, current dependency versions and the current kernel. It becomes visible when one of those changes.

What would it take to prove the hosts match?

This is a more useful question than it looks, because working through it honestly shows why “just check” is not an available answer.

Start with the naive approach. Compare the thing you suspect:

$ for h in web01 web02 web03 web04; do
printf '%-8s ' "$h"
ssh "$h" "grep -h nofile /etc/security/limits.conf /etc/security/limits.d/*.conf 2>/dev/null | tr -s ' '"
done

web01
web02
web03    * soft nofile 65535
web04

That works, and it is worth knowing. But notice what it required: you had to already suspect nofile. The check confirms a hypothesis; it does not discover anything.

The general version — “are these hosts the same?” — is a much larger question. A serious answer needs, at minimum:

Each row is answerable. The problem is that answering them all produces an enormous diff dominated by legitimate differences, and separating those from real drift needs a statement of what is supposed to differ.

Which is the entire point. You cannot detect drift without a declared intended state. The declaration is the hard part and the valuable part; the comparison is comparatively mechanical. Any tool that promises drift detection without asking you to declare intent is comparing hosts to each other, which finds divergence but cannot tell you which host is right.

The failure mode of the fix

It is worth naming this now, because the rest of Part I keeps returning to it: the automation can become the snowflake.

A repository that is applied to production regularly is a description of production. A repository that has not been applied in eight months is a description of what somebody once intended, and the gap between it and reality is unmeasured. Running it at that point is more dangerous than never having written it, because it will confidently revert eight months of undocumented but load-bearing changes.

This is not a hypothetical failure. It is the most common way configuration management goes wrong in practice, and it is why lesson 7 treats “the automation everyone is now afraid to run” as a real cost rather than a joke.

Knowledge check