Snowflakes and the uniqueness budget · Ansible

Automation earns its keep by treating many machines the same way. Every per-host special case erodes that, and the erosion is gradual enough that nobody notices the moment it stops being worth it.

The end state is a repository where the roles are full of hostname conditionals, nobody can predict what a run will do to a given host without reading everything, and the only way to find out what a host is supposed to look like is to run the playbook against it and see.

The failure mode, precisely

- name: tune worker processes
  ansible.builtin.template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    mode: '0644'
  when: inventory_hostname != 'server27'

- name: tune worker processes (server27 variant)
  ansible.builtin.template:
    src: nginx-server27.conf.j2
    dest: /etc/nginx/nginx.conf
    mode: '0644'
  when: inventory_hostname == 'server27'

Each line of that was reasonable when written. Collectively it produces five specific problems.

The reason is gone. Somebody knew why server27 was different in 2023. The code records only that it is.

There is no expiry. The batch job that made server27 special moved to its own host eighteen months ago. The conditional stays forever because deleting it requires knowing it is safe to, and nobody does.

Behaviour is unpredictable without reading everything. A role with n hostname conditionals has up to 2ⁿ distinct behaviours. Answering “what will this do to server27?” means reading every task, not inspecting the host’s data.

It defeats the drift report. The role now declares different things for different hosts based on logic rather than data, so the audit’s notion of correct is buried in the same conditionals.

It spreads. The next person with a special host copies the pattern, because it is visibly the house style.

The alternative is the previous lesson’s mechanism: the value in host_vars, the reason and expiry alongside it, the role unchanged and data-driven. Same behaviour, and every one of the five problems disappears.

Measuring uniqueness

Uniqueness is not a feeling. It is a number you can compute from the inventory, and trending it is what turns “we should avoid snowflakes” into something a team can act on.

Read-only / Safeuniqueness by variable— For each variable in resolved host data, counts how many distinct values it takes across the fleet. Read-only; connects to nothing. Ansible's own ansible_* facts and inventory internals are excluded.

cat > uniqueness.py <<'PY'
import json, sys, collections
SKIP = {'inventory_file', 'inventory_dir', 'group_names', 'groups',
      'omit', 'playbook_dir'}
hv = json.load(sys.stdin)['_meta']['hostvars']
values = collections.defaultdict(set)
for host, vars_ in hv.items():
  for k, v in vars_.items():
      if k.startswith('ansible_') or k in SKIP:
          continue
      values[k].add(json.dumps(v, sort_keys=True))
total = len(hv)
print(f"{'variable':<32}{'distinct':>9}{'hosts':>8}  ratio")
for k, s in sorted(values.items(), key=lambda kv: -len(kv[1])):
  print(f'{k:<32}{len(s):>9}{total:>8}  {len(s)/total:>6.2f}')
PY

ansible-inventory -i inventory/ --list | python3 uniqueness.py

Read-only / Safewhat it prints— Executed against ansible-core 2.21.3 on a four-host demonstration inventory. Two variables have been overridden per host; log_level is uniform.

$ ansible-inventory -i inventory/ --list | python3 uniqueness.py

variable                         distinct   hosts  ratio
nginx_worker_processes                  2       4    0.50
app_tuning_flag                         2       4    0.50
log_level                               1       4    0.25

On a real fleet the interpretation is:

Distinct values	Reading
1	A genuine standard. The best possible outcome
2–5 across thousands of hosts	Environment or class variation. Healthy — and it should live in `group_vars`, not `host_vars`
Dozens	A per-host setting pretending to be a standard. Either it is genuinely per-host and should be documented as such, or the class structure is wrong
Roughly one per host	Not a configuration variable at all. It is data about the host, and it probably belongs in a CMDB or as a derived fact

Two more numbers worth having on the same dashboard: the count of hosts that have any host_vars at all, and the count of config_exceptions records from the previous lesson. Both should be small and both should be flat. A steadily rising line on either is the estate telling you the standard no longer fits.

Gating the pattern in CI

The one-liner wins at 17:40 unless something stops it.

Read-only / Safefail the build on hostname conditionals in roles— Greps role and playbook sources for conditionals keyed on a specific host identity. Read-only. Exits 1 on a hit, which is what makes it a gate rather than a suggestion.

set -uo pipefail

PATTERN='inventory_hostname[[:space:]]*[=!]=|ansible_hostname[[:space:]]*[=!]=|ansible_fqdn[[:space:]]*[=!]='

if grep -rnE "$PATTERN" roles/ playbooks/; then
echo
echo "FAIL: host-identity conditionals found in roles or playbooks."
echo "Express the difference as inventory data with a config_exceptions record."
exit 1
fi

echo "OK: no host-identity conditionals"

Two notes on making that gate survive contact with reality.

Allow the legitimate uses. inventory_hostname is entirely reasonable in a template, in a delegate_to, in a log message, and in run_once selection logic. The pattern above deliberately matches only comparisons against a host identity, which is the construct that encodes a specific host into the code.

Give it an escape hatch with a cost. A marker comment that suppresses the check for one line, plus a rule that the marker must cite a ticket, keeps the gate from being deleted wholesale the first time somebody genuinely needs it. A gate that can only be satisfied or removed will be removed.

When to graduate a host out of the standard

The previous lesson ended on the host with eleven exceptions. Here is the answer.

Three shapes of deliberate difference, and they are not interchangeable:

Situation	Right expression
One host, one or two differences, temporary	`config_exceptions` record with an expiry
Several hosts sharing a constraint	A group with its own `group_vars`
One host differing in many ways, permanently	Its own class: its own group, its own declaration, its own playbook entry

The third is the one people resist, because it feels like giving up. It is not. It is an honest description of what is already true.

A host carrying eleven exceptions is not a web server with adjustments. It is a different kind of machine that has been forced through the web server’s roles by accumulating overrides, and everyone maintaining those roles is paying for the pretence. Giving it its own group and its own declaration makes the roles simpler, makes the drift report meaningful for both populations, and makes the host’s actual requirements visible in one place.

Under the hood

The reason data-driven beats conditional is that Ansible already has a resolution engine for data and does not have one for logic.

A variable’s value is computed by a documented precedence order that ansible-inventory --host and ansible-inventory --graph --vars will show you without running anything. You can ask “what will nginx_worker_processes be on server27?” and get an answer in milliseconds, from a command that connects to nothing.

There is no equivalent for when conditions. They are evaluated at run time, per task, against the full variable context, and the only way to know which branch a host takes is to run the play or to read every task by hand. --list-tasks shows you the tasks; it cannot tell you which ones a when will select.

That asymmetry is the whole argument. Moving a difference from a conditional into inventory does not merely tidy it — it moves it from a part of the system with no introspection into a part with good introspection, and everything downstream (the audit, the drift report, the review, the incident investigation at 03:00) gets easier as a result.

What happens if uniqueness is already high when you start?

It usually is, and treating it as a project to be finished is how the effort dies.

Do not attempt to eliminate it. Attempt to stop it growing, and let it decay.

The sequence that works: turn on the measurement and record today’s number as the baseline, without judging it. Add the CI gate so no new hostname conditionals enter the repository. Require a config_exceptions record for any new host-specific variable. Then, each time somebody touches a role for another reason, convert the conditionals in the part they touched.

Two years of that leaves a repository that is mostly clean, with the remaining special cases concentrated in roles nobody has needed to change — which is a reasonable place for them to be. A three-month cleanup project, by contrast, leaves you with a large risky refactor, no gate, and the same problem again in a year.

Knowledge check

Why this matters

The value of configuration management is a claim about a population: these four hundred machines are configured this way, and here is the evidence. Every special case shrinks the population the claim covers, and enough of them reduce it to nothing while the tooling still runs nightly and still reports success.

What makes this different from a style preference is that it is measurable. Distinct values per variable, hosts carrying host-specific data, exceptions on the register — three numbers, computable from the inventory, trendable over time, and gateable in CI.

An estate that watches those numbers can tell you how much of its automation is still automation. An estate that does not will find out during the incident where somebody says “oh, that host is a bit different” — and discovers that so are eleven others.