Upgrading Ansible without surprising the fleet · Ansible

Upgrading Ansible feels like maintenance. It is not. It is a change to the tool that executes every task against every host you own, made without touching a single playbook, and its blast radius is the whole fleet.

Treat it accordingly: the same enumeration, the same rehearsal, the same staged rollout you would give a change to the fleet itself. The procedure below is that treatment, and it costs an afternoon.

What actually changes across a minor release

Four categories, in rough order of how much trouble each causes.

Removed features. ansible-core deprecates and then removes on a published schedule. A removal is the only category that fails cleanly: the run stops with an error naming the thing. Unpleasant, but honest.

Changed defaults. The worst category. A default that moves changes behaviour silently — no error, no warning, a different outcome on the fleet. These are exactly what the porting guide exists to enumerate.

Stricter validation. Something that previously produced a warning now fails, or something loose in your YAML is now rejected. Fails cleanly, but often in a place unrelated to what you were changing.

New deprecations. Nothing breaks today. These are the warnings that tell you what the next upgrade will remove, which is why the estates that suppress warnings are the estates for which every upgrade is a crisis.

The managed-node Python floor is a fifth thing, easy to miss because it is not a code change at all. Moving from core 2.19 to 2.21 raises the supported managed-node Python from 3.8 to 3.9. No playbook changed; some hosts left the supported set.

The procedure

Read the porting guide for every version you are crossing, not only the target. Going 2.19 to 2.21 means reading both the 2.20 and the 2.21 guides.
Collect the current deprecation warnings from a full check-mode run. These name what the next release removes, and they are the cheapest predictor you have.
Establish the oldest Python on any managed node, and compare it with the new version managed-node floor.
Install the candidate alongside the current version. Do not replace anything yet.
Syntax-check every playbook in the repository against both versions and diff the results.
Run the estate in check mode against both versions, on staging first, and diff the recaps.
Switch the default on one controller. Leave the old version installed.
Run production change on the new version for a full cycle before removing the old one.

Steps four through six are what make this an upgrade rather than a gamble, so they get the rest of the lesson.

Install alongside

Both install methods from earlier in this part support this directly, which is one of the reasons the course prefers them.

Configuration changea second virtualenv, at a version-named path— Creates an entirely separate environment. The existing controller is untouched and continues to serve scheduled runs while you test.

python3.12 -m venv /opt/estate/venv-2.22
/opt/estate/venv-2.22/bin/pip install 'ansible-core==2.22.0'
/opt/estate/venv-2.22/bin/ansible-galaxy collection install \
  -r /opt/estate/requirements.yml -p /opt/estate/collections-2.22

Configuration changeor with pipx, using a command suffix— Installs a second ansible-core whose commands are named ansible-next, ansible-playbook-next and so on. The unsuffixed commands keep working exactly as before.

pipx install --suffix=-next 'ansible-core==2.22.0'
ansible-next --version

Naming the path after the version rather than after its role — venv-2.22, not venv-new — matters more than it looks. venv-new is accurate for about a month and misleading for years afterwards, and the person debugging a scheduled job at 3 a.m. has no way to tell what is in it.

Prove against both

Two levels of proof, and neither connects to a managed host.

Syntax and structure. --syntax-check parses the playbook, resolves imports and includes, and validates task structure. It catches removed keywords, renamed parameters and anything the new parser rejects.

Read-only / Safeevery playbook, both versions, one diff— Parses each playbook under both installs and compares the results. No connection is made to any host; this is entirely controller-side.

cd /opt/estate
for p in playbooks/*.yml; do
echo "== $p"
/opt/estate/venv/bin/ansible-playbook --syntax-check "$p" 2>&1
done > /tmp/syntax-current.txt

for p in playbooks/*.yml; do
echo "== $p"
/opt/estate/venv-2.22/bin/ansible-playbook --syntax-check "$p" 2>&1
done > /tmp/syntax-candidate.txt

diff /tmp/syntax-current.txt /tmp/syntax-candidate.txt

An empty diff is the outcome you want. Anything else is a list of work to do before the upgrade, produced in a couple of minutes, with no risk to anything.

Targeting. --list-hosts and --list-tasks tell you whether the new version resolves your inventory and your conditionals to the same set of hosts and the same set of tasks. Neither connects.

Read-only / Safewhat this play would do, before it does anything— Lists the plays and their tasks without executing or connecting. Diffing this output between two ansible-core versions catches a changed include or a renamed keyword that syntax checking alone would pass.

$ ansible-playbook --list-tasks playbooks/webtier.yml

playbook: playbooks/webtier.yml

play #1 (webservers): Configure the web tier	TAGS: []
  tasks:
    Install packages	TAGS: [packages]
    Deploy configuration	TAGS: [config]
    Ensure service is running	TAGS: [service]

Illustrative output

Behaviour. Check mode is the closest you get to a rehearsal, and it is the first step in this procedure that touches the fleet at all — it connects, gathers facts, and asks each module what it would change.

Configuration changea check-mode rehearsal against staging, on both versions— Connects to every host in the staging inventory and gathers facts. Modules that support check mode report intended changes without making them - but a module that does not support check mode is skipped rather than simulated, and command or shell tasks are not rehearsed at all unless they carry check_mode: false.

/opt/estate/venv/bin/ansible-playbook -i inventory/staging \
  --check --diff playbooks/site.yml > /tmp/check-current.txt 2>&1

/opt/estate/venv-2.22/bin/ansible-playbook -i inventory/staging \
  --check --diff playbooks/site.yml > /tmp/check-candidate.txt 2>&1

diff /tmp/check-current.txt /tmp/check-candidate.txt

Deprecation warnings are the early-warning system

Every estate that finds upgrades traumatic has the same habit: warnings are noise, so warnings are suppressed.

A deprecation warning names something that will stop working in a specific future release. It arrives one or two releases before the removal, which is deliberately enough time to fix it during ordinary work rather than during an upgrade. Suppressing them does not remove the work; it defers all of it into a single day, later, under pressure.

Configuration changeharvest what the current version is already telling you— A check-mode run against staging, filtered to deprecation notices. This connects to staging hosts. The output is your work list for the next upgrade, available today.

ansible-playbook -i inventory/staging --check playbooks/site.yml 2>&1 \
| grep -i 'DEPRECATION\|will be removed'

Under the hood

ansible-core versions are supported for a defined window, and the window is short by infrastructure standards — releases arrive roughly every six months and older ones stop receiving fixes. An estate that upgrades once every three years is not making a conservative choice; it is accumulating three years of removals to apply in one step, on a version whose porting guide assumes you came from the previous one.

Two consequences follow.

Upgrade little and often. The cost of a single minor bump is mostly fixed — read a guide, build alongside, diff, switch. The cost of crossing four of them at once is superlinear, because the failures interact and you can no longer attribute a broken playbook to a particular change.

The porting guides are written pairwise. Each one describes moving from the immediately preceding version. Skipping releases means reading every guide in between; there is no combined document for 2.19 to 2.22, and reading only the 2.22 guide will miss what 2.20 removed.

Collections have their own release cadences, entirely independent of core. A collection can require a newer core than you run, or drop support for the one you are on. requirements.yml is therefore part of the upgrade, not a separate exercise — and the two-machine manifest test is how you confirm the candidate environment is what you think it is.

What happens if you upgrade in place and it goes wrong?

You discover it during the next scheduled run, which is usually unattended, and your rollback is a reinstall of the old version that you now have to reconstruct.

If the environment is pinned this is recoverable — requirements.txt and requirements.yml from the previous commit rebuild the previous controller, and the two-machine test tells you when you have it back. That is precisely the disaster the pin files exist for.

If it is not pinned, you are reconstructing an environment from memory while production runs are failing. The old version’s exact patch release, the collection versions that went with it, and any library somebody injected are all unknown, and each wrong guess produces a different failure.

This is the whole argument for installing alongside rather than upgrading in place. Rolling back becomes changing which absolute path the cron entry calls — a one-line, instant, fully reversible change — instead of a rebuild. Keep the old environment on disk for at least one full change cycle. A virtualenv is a directory of a few tens of megabytes; the storage is not the constraint, and having it there turns a bad afternoon into a two-minute revert.