A catalogue of rollback patterns, and what each is for · Ansible

“Roll back” is not one operation. It is five, and they differ in what they restore, how long they take, what they cost, and what they silently leave behind. Choosing between them is the work; the previous lesson said the choice must be made in advance, and this lesson is the menu.

The organising question for each pattern is not “how do I do this” but “what failure does this answer, and what does it not touch”.

The reversibility spectrum

Before the patterns, the axis they sit on.

Change	Reversibility	Pattern	Typical cost
Config file content	Full	Restore previous file	Seconds
Service unit content	Full	Revert unit, `daemon_reload`	Seconds
Application artefact	Full	Redeploy previous release	Seconds to minutes
Package version	Partial	Version-pinned reinstall	Minutes, may fail
Kernel	Partial	Boot the previous entry	A reboot
Whole machine state	Full, coarse	Snapshot revert	Minutes, loses newer data
Database schema	None, usually	Down-migration if written, else restore	Tens of minutes, data loss
Deleted data	None	Restore from backup	Hours, data loss
Provider-side secret rotation	None	Rotate forward again	Varies

Read the table top to bottom and the cost climbs faster than the difficulty of the change does. A three-character edit to a config file and a three-character edit to a migration script look identical in a diff and sit at opposite ends of this table.

Pattern 1 — restore the previous configuration file

Answers: a config change that produced bad behaviour. Restores: the file. Nothing else. Cost: seconds, plus a service reload.

The mechanism is backup: true on copy or template, which the documentation describes as “Create a backup file including the timestamp information so you can get the original file back if you somehow clobbered it incorrectly.” The path of that file comes back in the backup_file return value, documented as returned “changed and if backup=yes”.

Configuration changethe forward change, recording its own reverse— Recording backup_file is what makes the restore deterministic. Without it the restore is a guess about which timestamped file is the right one.

- name: Deploy the configuration
ansible.builtin.template:
  src: app.conf.j2
  dest: /etc/app/app.conf
  owner: root
  group: app
  mode: '0640'
  backup: true
register: config_write
notify: reload app

- name: Record the backup path for the reverse procedure
ansible.builtin.copy:
  content: "{{ config_write.backup_file }}"
  dest: /var/lib/app-change/last-config-backup
  mode: '0600'
when: config_write.backup_file is defined

When it is the right choice: the change was a config edit, the service is stateless or nearly so, and the effect is bounded by what the running process does now rather than what it did while misconfigured.

Pattern 2 — reinstall the previous package version

Answers: a package upgrade that broke something. Restores: the binaries and files the package owns. Cost: minutes, and it can fail outright.

Service impact possibleversion-pinned reinstall— Both options verified present in ansible-core 2.21.3 via ansible-doc. Both module documents warn that allow_downgrade can make the module non-idempotent.

# Debian family. apt accepts name=version.
- name: Pin the application package back to the previous release
ansible.builtin.apt:
  name: acme-app=2.4.1-1
  state: present
  allow_downgrade: true

# Red Hat family. dnf accepts an explicit NEVRA.
- name: Pin the application package back to the previous release
ansible.builtin.dnf:
  name: acme-app-2.4.1-1.el9
  state: present
  allow_downgrade: true

Three preconditions have to hold, and the pre-flight play should assert all three rather than discovering them during the window:

The old version is still in a reachable repository. Many internal mirrors keep only the current release. Assert it with a repository query before the window, not after.
The package supports downgrade. Post-install scripts that migrate on-disk state forward frequently have no reverse, so the older binary starts against data it does not understand.
Dependency resolution does not take other packages with it. This is the explicit warning in both module documents: the task “could end up with a set of packages that does not match the complete list of specified packages to install”.

When it is the right choice: the package is self-contained, the old version is definitely available, and you have downgraded it successfully at least once somewhere else.

Pattern 3 — redeploy the previous artefact

Answers: a bad application release. Restores: the running code. Cost: seconds, and it is the most reliable pattern in this list.

This is why release-directory-plus-symlink layouts exist. The old release is still on disk; rolling back is changing which one current points at.

Service impact possibleartefact rollback by symlink— The previous release is already on disk, already unpacked, already the exact bytes that were running an hour ago. Nothing is downloaded and nothing is rebuilt.

- name: Point current at the previous release
ansible.builtin.file:
  src: "/opt/app/releases/{{ rollback_to_release }}"
  dest: /opt/app/current
  state: link
  owner: app
  group: app
notify: restart app

- name: Confirm the application answers after the swap
ansible.builtin.uri:
  url: "http://{{ ansible_host }}:8080/healthz"
  status_code: 200
  timeout: 10
retries: 6
delay: 5
until: rollback_health is succeeded
register: rollback_health

It is reliable because nothing has to be reconstructed. The previous release is bytes on a disk that were serving traffic recently. Contrast pattern 2, where the previous version has to be fetched, resolved and installed by a package manager that may decide otherwise.

Its limits are the same as every code rollback: it does not reverse anything the new release wrote. Rows, files, queue messages, calls to other systems.

Pattern 4 — restore from a snapshot

Answers: a change whose blast radius is the whole machine. Restores: everything on the volume, as of the snapshot. Cost: minutes to revert, plus everything written since.

Available where the platform supports it: a hypervisor snapshot, a cloud volume snapshot, an LVM snapshot taken with lvcreate --snapshot, a ZFS snapshot. Not available on bare metal without volume management, which is the usual reason a plan cannot use it.

When it is the right choice: an irreversible in-place operation on a machine whose durable state lives somewhere else, where the alternative is rebuilding the host.

Pattern 5 — revert the Git commit and re-apply

Answers: a change to declared state, where the declaration is the whole story. Restores: everything the role declares. Cost: one playbook run.

This is the pattern everyone reaches for first, and it is genuinely the right answer for a real class of change: the role manages the complete content of what it manages, the change altered only that content, and nothing outside the role was touched.

Configuration changerevert and re-apply, scoped— The limit must match the forward change. Reverting the commit and running against the whole fleet applies the revert to hosts that were never changed - usually harmless, occasionally not.

# Confirm what the revert would change, without changing it.
ansible-playbook -i inventories/prod site.yml \
--limit 'appservers:&batch_a' --check --diff

# Apply.
ansible-playbook -i inventories/prod site.yml \
--limit 'appservers:&batch_a'

The previous lesson covered why it is insufficient on its own. The short version, restated because it is the failure this catalogue exists to prevent: the old revision makes no statement about state the new revision created, so users, packages, directories, cron entries and migrations added by the change survive the revert untouched, and the recap is green anyway.

When it is the right choice: the diff between the two revisions consists entirely of changed values in files the role already managed, with no additions. That is checkable — read the diff and look for anything that creates something.

Special case — the kernel

Worth its own note because the pattern is different from all five.

The previous kernel is still installed. Rolling back is a boot-entry selection, not a package operation. grub-reboot sets the default boot entry “for the next boot only”, which is exactly the right semantic for a rollback: if the previous kernel also fails to boot, the next reboot returns to the default rather than pinning you into a broken choice.

Service impact possiblekernel rollback is a boot selection— A reboot is required, so this pattern costs an outage for the host regardless of how quickly the command runs.

- name: Select the previous kernel for the next boot only
ansible.builtin.command:
  argv:
    - grub-reboot
    - "{{ previous_boot_entry }}"
changed_when: true

- name: Reboot into it
ansible.builtin.reboot:
  reboot_timeout: 600

- name: Confirm the running kernel is the one intended
ansible.builtin.assert:
  that:
    - ansible_facts['kernel'] is version(bad_kernel_version, '<')
  fail_msg: "Host booted {{ ansible_facts['kernel'] }}, not the previous kernel"

Note the assert. Without it the play reports success for a host that rebooted straight back into the kernel you were rolling back from, because the boot entry string did not match anything.

The changes with no pattern

An honest list. These are not solved by anything above.

Deleted data. Restore from backup, or it is gone.
A forward-only schema migration. A down-migration if someone wrote one and it has been tested; otherwise a database restore.
A rotated secret at the provider. The old value no longer exists. Rotating “back” means rotating forward to a third value, and every consumer has to be updated again.
An email, a webhook, a payment, a DNS propagation. Sent is sent.
A reformatted filesystem, a repartitioned disk, a destroyed volume.
A decommissioned host whose lease was released.

Why unit-file changes need daemon_reload and config-file changes do not

A service unit is a config file with an extra step, and forgetting the step produces a rollback that reports success and does nothing.

systemd parses unit files once and caches the result in memory. Writing /etc/systemd/system/app.service back to its previous content changes the file on disk; the running systemd still holds the parsed version of the file you are rolling back from. A restart after the write restarts the service with the old, cached unit definition — so the change you reverted is still in effect, and the play reports changed for the file write and changed for the restart.

daemon_reload is what makes systemd re-read the unit directory. It is a documented option of ansible.builtin.systemd_service, described as running “daemon-reload before doing any other operations, to make sure systemd has read any changes”:

- name: Restore the previous unit file
  ansible.builtin.copy:
    src: "{{ unit_backup_path }}"
    dest: /etc/systemd/system/app.service
    remote_src: true
    owner: root
    group: root
    mode: '0644'

- name: Reload systemd and restart with the restored unit
  ansible.builtin.systemd_service:
    name: app
    daemon_reload: true
    state: restarted

The failure mode is worth internalising because it generalises: a rollback that restores a file consumed by a long-running process has not finished until the process has re-read it. That is true of systemd units, of nginx config, of sshd, of anything with a reload step. The file write is the visible part and the reload is the part that matters.

What happens if two patterns are needed at once?

That is the normal case for anything bigger than a one-file change, and it introduces the property most reverse procedures get wrong: order.

A release that changed the artefact, the config and the package version needs all three reversed, and the order is not arbitrary. Reversing the config first means the running (new) code reads the old config, which is a combination that has never been tested. Reversing the package first may remove a library the running code depends on.

The usual safe order is: stop or drain the service, reverse in the inverse order of the forward change, then start and validate. That is worth writing into the reverse procedure as numbered steps, because “reverse the three changes” gives the operator six possible orders and only one of them is the one you rehearsed.

The other thing that falls out of ordering: the whole sequence has a duration, and it is the sum of the parts plus the drain. A plan claiming a two-minute rollback for a three-pattern reverse has not been timed.