Anti-pattern: latest everywhere · Ansible

Configuration management exists to make hosts the same. This anti-pattern makes what “the same” means depend on the date the host was converged.

It enters at four layers, and most estates have it at all four without anyone having chosen it.

Layer 1: `state: latest` in roles

Service impact possiblethe task, and what it actually means— SERVICE-IMPACT: upgrades a package and, with the handler, restarts the service. What version arrives depends on when the task runs.

- name: Install nginx
ansible.builtin.package:
  name: nginx
  state: latest
notify: Restart nginx

state: latest does not mean “install it”. It means “upgrade it to whatever the repository offers, right now”.

The consequences arrive as a set:

The converged state is not defined by the repository. Two hosts converged a week apart hold different versions. Both runs succeeded and both reported convergence.
Every run is a potential upgrade. A play intended to fix a config typo can also bump nginx across three hundred hosts, because the task was in the same role.
The handler fires on the upgrade. So the service restarts, on a run nobody expected to restart anything.
The change is unannounced. No ticket, no window, no review — the upstream maintainer’s release schedule is now your change schedule.

The corrected form is state: present for “ensure it is installed”, and an explicit version where the version matters:

Configuration changepresent, and pinned where it matters— CONFIGURATION: installs if absent. The second form installs an exact version, so convergence means the same thing on every host regardless of date.

- name: Install nginx
ansible.builtin.package:
  name: nginx
  state: present

- name: Install the application at the version this release deploys
ansible.builtin.package:
  name: 'app-server={{ app_server_version }}'
  state: present

Upgrading then becomes a deliberate act: change app_server_version in a commit, review it, and roll it out with the staging from lesson 3. Part XL covers patching as its own scheduled activity, which is where upgrades belong.

Layer 2: unpinned collections

Read-only / Safethree requirements.yml files, only one of which pins— Read-only: file contents. The first two re-resolve on every install; only the third is a specification.

# Unpinned - installs whatever is current today.
collections:
- name: community.general

# A range - narrower, and still not reproducible.
collections:
- name: community.general
  version: '>=12.0.0,<13.0.0'

# Pinned - the same artefact every time.
collections:
- name: community.general
  version: 12.0.0

The failure. A collection release changes a module’s default, adds a parameter, or fixes a bug you were depending on. Nothing in your repository changed, and the fleet’s behaviour did.

The version that makes this concrete: a module gains stricter validation in a minor release, and a playbook that has run nightly for a year starts failing at 02:00 with an error nobody has seen. Your first hypothesis will be that something changed on the hosts. Nothing did.

Layer 3: no `requirements.yml` at all

The version of layer 2 where there is nothing to unpin. Collections were installed by hand, on the controller, over several years.

The failure is the one Part L is built on. The controller is lost, or a second one is built, and there is no definition to install from. The collection set is unrecoverable knowledge, and the rebuild produces a controller that works until the first play that needs the collection somebody installed during an incident in March.

Layer 4: no controller pinning

ansible-core and its Python dependencies installed from whatever the distribution or PyPI currently offers.

The failure is a behaviour difference between controllers, or across time on the same controller. ansible-core minor releases deprecate keywords, change module behaviour and occasionally alter defaults. The porting guides exist precisely because these changes are real.

The estate this produces

Hosts differ by their build date, so "which version of nginx is on the fleet" has as many answers as there were provisioning waves.
Reproducing a bug in staging fails, because staging was built last month and production was built last year.
A rollback target cannot be named. "The previous version" is a different string on different hosts.
Incident timelines have an unexplained variable: something changed, and the change is not in any repository you control.
Compliance evidence is weak. "The estate runs approved versions" is unprovable if the versions were never declared.

This is the snowflake problem from Part I arriving through a different door. Part I described snowflakes made by hand; this is the same divergence produced by automation that ran correctly.

Finding it

Read-only / Safeaudit the four layers— READ-ONLY: greps and version reads only. Run all four - most estates have findings at every layer.

cd /srv/ansible

# Layer 1
grep -rn 'state:\s*latest' --include='*.yml' --include='*.yaml' .

# Layer 2 and 3
test -f requirements.yml && cat requirements.yml || echo 'NO requirements.yml'
ansible-galaxy collection list

# Layer 4
ansible --version | head -1
pip --disable-pip-version-check list 2>/dev/null | grep -i ansible

# The whole picture: what is actually installed versus what is declared.
ansible-lint --profile production

The comparison that matters is between ansible-galaxy collection list and requirements.yml. Every installed collection missing from the declaration is a rebuild failure that has not happened yet.

Under the hood

Why does state: latest feel safe when state: present does not obviously feel safer?

Because it maps onto a genuinely good instinct — keep systems patched — and it appears to automate it. The problem is not the goal but the coupling: it attaches an upgrade to whatever schedule the convergence play runs on, which is a completely different schedule from the one patching should have.

Consider the properties a patching process needs, from Part XL: a maintenance window, an announcement, a staged rollout, a health check between batches, and a rollback plan. Now consider what state: latest in a converge role provides: none of them. The upgrade happens whenever somebody runs the play, on whatever hosts that play targets, with no staging that was not already there for another reason.

There is a second, subtler coupling. state: latest makes the fleet’s state a function of the upstream repository’s state at the moment of the run. Your convergence is now dependent on an external system you do not control, whose contents change without notice, and whose maintainers have no knowledge of your change process. An upstream security release becomes an unannounced production change; so does an upstream regression.

The version that catches teams out is the interaction with handlers. state: latest finding an upgrade returns changed: true, which notifies the restart handler, which restarts the service. A run intended to correct a typo in a config file becomes a fleet-wide package upgrade and service restart, and the recap says changed=300 — which, if the estate also has the anti-pattern from lesson 5, is what it says every night anyway.

The corrected split is to separate the two activities entirely: converge plays install state: present and are safe to run at any time, and patching is its own playbook, on its own schedule, with its own window and staging. Both are then honest about what they do.

What happens if pinning means the estate never gets updated?

That is the real objection, and it is legitimate: a pinned estate with nobody moving the pins is an estate accumulating unpatched vulnerabilities, which is a worse problem than version drift.

Pinning is not freezing. It converts an implicit, continuous, unreviewed update stream into an explicit, periodic, reviewed one — and the second half of that sentence is work somebody has to own.

Schedule the pin bump. Monthly for collections and ansible-core, aligned with your patch cycle for packages. It is a pull request that changes version numbers, reviewed like any other change.

Automate the proposal, not the merge. Dependabot, Renovate or a scheduled job can open the pull request. The value is that a human sees a diff saying “community.general 12.0.0 to 12.1.0” and can read the changelog. The value is lost if it merges itself.

Separate security from feature updates. A CVE fix should move faster than a minor version bump, and saying so in the policy means the fast path exists before you need it.

Test the bump the way you would test any change. The pinned versions go to staging first, then a canary, then the fleet. That is the whole point: pinning is what makes the update a change you can stage, rather than something that has already happened everywhere.

Watch for the pin that never moves. A requirements.yml untouched for three years is its own failure, and Part L lesson 3 describes where it ends — an ansible-core version that is no longer installable, discovered during a rebuild. The metric worth tracking is the age of the oldest pin, not the count of them.

Knowledge check

Why this matters

The purpose of configuration management is that hosts converge to a state the repository describes. This anti-pattern breaks the second half of that sentence: hosts converge to a state the upstream world describes, at the moment they happened to run.

Everything that depends on the repository being the specification degrades together. You cannot reproduce a bug in staging, because staging is a different vintage. You cannot name a rollback target, because the previous version differs per host. You cannot rebuild the controller, because there is no definition to rebuild from. You cannot say what the estate runs, only what it was asked to install.

And unlike most of this part, there is no incident that points at it. The failure presents as an unexplained variable in somebody else’s investigation: the play started failing, the versions do not match, the staging test did not reproduce. Nobody writes “we had no requirements.yml” in a post-incident review, because by then the question has moved on.

The fix is a version number in a file, per layer. It is the cheapest change in this part, and the one whose absence quietly invalidates the whole of Part L.

Layer 1: state: latest in roles