The reverse is part of the change · Ansible

Rollback is planned before execution, not improvised after it.

That is not a preference. It follows directly from the previous lesson: if Ansible has no undo, then the reverse of a change is something a person builds, and the two moments available for building it are before the window, when the change is understood and nobody is under pressure, or during the incident, when neither is true.

The industry default is the second one. This lesson is about making the first one non-optional.

The rule

A change with no written reverse does not get a maintenance window.

Not “should have one”. Does not get one. The reverse procedure is an entry gate, in the same way a peer review or a --syntax-check is an entry gate, and for the same reason: it catches a class of problem that is cheap to fix now and expensive to fix later.

The rule has a useful side effect that is arguably worth more than the artefact. Writing the reverse forces someone to think through what the change actually does, and that thinking regularly finds a problem in the forward plan. A change whose reverse cannot be written is usually a change that has not been understood.

What the artefact contains

Six fields. A change plan that has all six is executable by someone who did not write it, which is the whole test.

Field	The question it answers	Failure when it is missing
Reverse procedure	Exactly which commands, in order, on which hosts	Improvised at 03:00 from memory
Executor	Who is permitted to run it, and who is on call	Nobody feels authorised to pull the trigger
Duration	How long it takes end to end	The decision is made without knowing the cost
Cost	What the rollback itself breaks or interrupts	The reverse causes a second outage
Trigger criteria	The objective conditions that mean roll back	Debate during the incident
Point of no return	The step after which forward is cheaper than back	Someone attempts a reverse that no longer exists

Reverse procedure

Specific to the point of being boring. Not “restore the config” but the actual invocation, with the actual limit, tested against the actual inventory.

Configuration changereverse procedure, as written in the change plan— The limit is the same host list the forward change used. Tags select only the restore path. The operator does not have to construct anything.

# Step 1 - confirm what will be touched, changing nothing.
ansible-playbook -i inventories/prod site.yml \
--limit 'appservers:&batch_a' \
--tags rollback_config \
--check --diff --list-hosts

# Step 2 - restore the previous configuration.
ansible-playbook -i inventories/prod site.yml \
--limit 'appservers:&batch_a' \
--tags rollback_config

# Step 3 - validate. Exit non-zero means the rollback did not land.
ansible-playbook -i inventories/prod validate.yml \
--limit 'appservers:&batch_a'

Three properties make that usable under pressure: the commands are copyable, the host selection is identical to the forward change so there is nothing to re-derive, and step 3 tells the operator whether it worked rather than leaving them to decide.

The reverse must be tested, in a place where testing it is free

A reverse procedure that has never been executed is a hypothesis with good handwriting.

The cheapest test is the pre-production fleet, where the forward change and the reverse both run before the production window. The second cheapest is a container. Either is enormously better than the first execution being the incident.

The specific failures a rehearsal catches, all of which have shipped in real change plans: the --tags value does not exist so the run does nothing and exits 0; the limit pattern matches an empty set so the run does nothing and exits 0; the restore task depends on a variable defined only in the forward path; the backup file the restore reads is written by a task the rollback tag excludes.

Notice that the first two fail silently and successfully. That is the shape to be afraid of.

Trigger criteria

The criteria must be objective, because the person applying them will be tired, invested in the change, and subject to the sunk-cost pull of “it is probably nearly working”.

Written like this	Or like this
“if performance degrades”	“if p99 latency exceeds 400 ms for 5 consecutive minutes”
“if errors increase”	“if the 5xx rate exceeds 1% of requests”
“if the canary looks unhealthy”	“if the canary fails the validation play”
“if it takes too long”	“if the window is not complete by 03:30”

The left column is not a criterion. It is a request for a judgement call under conditions that make judgement calls bad. The right column can be evaluated by someone who arrived ten minutes ago, and — more importantly — can be enforced mechanically, which is what lesson 5 is about.

The clock criterion deserves particular attention. Almost every change plan omits it, and time is the criterion that fires most often. A change that is still going at the end of the window has failed even if nothing is broken, because the next thing to happen is the business day.

Point of no return

Every change plan should name the step after which rolling back costs more than going forward. This single line prevents more damage than the rest of the document.

Consider a deployment with a forward-only schema migration:

Step	Reverse available	Cost of reversing
1. Deploy new artefact to canary	Yes	Swap the symlink back, 30 s
2. Validate canary	Yes	Swap the symlink back, 30 s
3. Deploy to remaining batches	Yes	Swap symlinks back, 4 min
4. Run the schema migration	Point of no return	Restore the database, 40 min, data loss
5. Enable the new feature flag	Forward only	Fix forward

Before step 4 the reverse is a symlink swap. After step 4 the reverse is a database restore with an RPO measured in minutes of lost writes. Those are not the same decision and the plan must not present them as one line called “rollback”.

Where the artefact lives

In the repository, next to the change, reviewed with the change. Not in a wiki page, not in a ticket comment, not in the head of the person who wrote the role.

changes/
  2026-08-14-app-2.5.0/
    plan.md              # forward steps, timings, approvers
    reverse.md           # the six fields above
    preflight.yml        # asserts run before the window opens
    validate.yml         # asserts run after each batch

Keeping it in the repository buys three things: it is reviewed by the same process as the code, it is versioned alongside the revision it reverses, and it is available to someone who was not in the planning meeting. That last one is the point — the artefact exists for the person who did not write the change.

Making the rollback path a first-class part of the role

The reverse procedure is more likely to be correct if it lives in the same role as the forward change, rather than in a separate playbook somebody wrote once.

Tags are the usual mechanism. A role that manages a config file can carry both directions:

- name: Deploy the new configuration
  ansible.builtin.template:
    src: app.conf.j2
    dest: /etc/app/app.conf
    owner: root
    group: app
    mode: '0640'
    backup: true
  register: config_write
  tags: [change, config]
  notify: reload app

- name: Record where the backup went
  ansible.builtin.copy:
    content: "{{ config_write.backup_file }}\n"
    dest: /var/lib/app-change/last-backup-path
    mode: '0600'
  when: config_write.backup_file is defined
  tags: [change, config]

- name: Read the recorded backup path
  ansible.builtin.slurp:
    src: /var/lib/app-change/last-backup-path
  register: recorded_backup
  tags: [never, rollback_config]

- name: Restore the previous configuration
  ansible.builtin.copy:
    src: "{{ (recorded_backup.content | b64decode) | trim }}"
    dest: /etc/app/app.conf
    remote_src: true
    owner: root
    group: app
    mode: '0640'
  tags: [never, rollback_config]
  notify: reload app

Two details carry the weight.

backup_file is a documented return value of copy and template, returned “changed and if backup=yes”, containing the path of the timestamped copy. Recording it is what makes the restore deterministic instead of a guess about which of the seventeen backup files in /etc/app/ is the right one.

The never tag on the rollback tasks means they are skipped unless the tag is requested explicitly. --tags rollback_config runs them; --tags change does not; a bare ansible-playbook site.yml does not. The rollback path is therefore present in every run of the role — so it is linted, reviewed and syntax-checked — and executed in none of them until asked for. That combination is why it stays correct.

What happens if a change genuinely has no reverse?

Some do not. A destructive data migration, a decommission, an irreversible vendor-side operation. The rule is not that such changes are forbidden.

The rule is that the plan must say so, in those words, and the mitigation moves elsewhere:

A restore becomes the reverse. Not a rollback, a recovery, with its own RTO and its own rehearsal. If restoring takes 40 minutes, the window must contain 40 spare minutes.
The exposure is narrowed. If it cannot be undone, it should touch as few hosts as possible per step, with validation between steps.
The approval bar rises. A change with no reverse should have a different approver from one whose reverse is a symlink swap.
The point of no return moves to the front of the plan, and the go/no-go gate before it becomes the real decision point.

What must not happen is a plan that says “rollback: N/A” with no sentence after it. That reads as an omission and reviewers treat it as one. “Rollback: none. Recovery is a restore from the pre-change snapshot, RTO 40 minutes, tested 2026-07-30” is a plan.

Knowledge check

Why this matters

Change plans are written by people who expect the change to work. That is not a criticism; a person who expected it to fail would not be proposing it. But it produces a document weighted almost entirely toward the forward path, with rollback as a sentence near the bottom that nobody has costed.

The six fields exist to shift that weighting without requiring anybody to be pessimistic. Filling them in is a mechanical exercise. It takes twenty minutes. And it converts the rollback from a vague reassurance into something with a duration, an owner and a price — which means the change board is approving a real thing rather than a word.

The point of no return is the field that repays the effort most. Almost nobody writes it, and it is the one that decides whether the response to a bad change is a symlink swap or a database restore.