high riskservice affecting~75 min

Runbook: Roll back a configuration change

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The set of hosts that actually received the change is established by measurement, not by reading the change record
· The previous state is identified concretely: a backup file on each host, a Git commit, or a package version - not "how it was before"
· Every step of the change has been classified as reversible or not, and the irreversible ones are listed by name
· The rollback has been decided as the right action - a forward fix is sometimes correct and reverting is not automatically safer
· The service owner knows the rollback is happening and what it will do to the service
· Whether reverting will break something that was built on top of the change in the meantime has been checked

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Stop the change from being re-applied: pause scheduled converges and CI, and revert the repository commit
2Measure which hosts hold the new state and which hold the old
3Locate the previous state per host: the backup the task took, or the artefact version
4Validate the restored content BEFORE activating it, using the consuming programs own validator
5Restore in batches, with drain, restore, validate, reload, verify, return - as a rolling change
6Verify per host that the running process has loaded the restored configuration, not just that the file is back
7Confirm the whole scope is in one state at the end
8Reconcile the repository with reality so the next converge does not re-apply the change
9Record the window during which the change was live, and on which hosts

4 · Verification

Confirm the procedure actually fixed the problem.

✓Every host in scope holds the previous configuration on disk, verified by checksum against a known-good reference
✓The running process has loaded it - verified by process start time or the services own config query, not by reading the file
✓The service answers a real request on every host that carries traffic
✓A converge run reports changed=0, proving the repository and the hosts agree
✓No host is left drained and no host is left in the new state
✓The repository no longer contains the change, so the next scheduled run will not re-apply it

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶The rollback of a rollback is re-applying the change, which is the original playbook - so keep the reverted commit rather than deleting the branch
↶Restore-from-backup is only available where the original task used backup: true; if it did not, the previous content may exist only in Git or nowhere
↶POINT OF NO RETURN: any part of the change that migrated data, rotated a credential, called an external system or removed something cannot be reversed by restoring a file
↶A service restart is not reversible in the usual sense - the previous process is gone, so validate the restored config before reloading
↶Do not stop a rollback partway; a fleet split between the change and its reversal is worse than either state

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the service owner before restarting anything that does not support reload
· Escalate immediately if the previous state cannot be located on some hosts - restoring the wrong content is worse than leaving the change in place
· Escalate if something was built on top of the change since it was applied; reverting will break the thing built on it
· Escalate if any host reached an irreversible step - the decision for those hosts is forward-only and needs stating explicitly
· Escalate if the change is a security control; reverting it reopens whatever it closed, and that is a security decision

Rolling back a configuration change is not the change played backwards. It is a new change, with its own blast radius, its own failure modes and its own need for verification - and it is being executed under time pressure by people who are already having a bad day.

Two things make it go wrong. Reverting hosts that never received the change, and assuming that restoring a file is the same as reversing an action. Most of this runbook is about those two.

When to use this runbook

A configuration change is causing a problem and must be reversed.
A change was applied to the wrong hosts.
A change was applied correctly and turns out to be wrong.
A rollout was stopped and the decision was to go backwards.

Blast radius

Every host that received the change - which is not necessarily the set the change record names, and establishing the difference is Step 2.

The rollback itself is service-affecting on every one of those hosts: it rewrites configuration and reloads or restarts a service.

Step 0: Is a rollback the right action?

Not always. Ask before starting:

Situation	Better action
The change is correct but one host reacted badly	Fix that host; do not revert forty
Something has been built on top of the change already	Forward fix; reverting breaks the thing built on it
The change is a security control	Reverting reopens what it closed - that is a security decision
Part of the change is irreversible	Rollback is impossible for the hosts that reached it; forward is the only option
The change is bad and nothing depends on it	Roll back. This runbook.

Reverting is not automatically the safer choice. It is a change, and it inherits every risk that the original change had.

Step 1: Stop it being re-applied

Read-only / Safepause the automation

systemctl list-timers --all --no-pager | grep -i ansible
sudo crontab -l -u ansible 2>/dev/null
pgrep -af ansible-playbook || echo 'nothing running'
echo 'Pause the CI pipeline that runs this playbook'

Configuration changerevert the repository first

cd /srv/automation/repo
git log --oneline -5
git revert --no-edit 9a3f1c2
git push origin main

Revert the repository before touching hosts. A rollback executed against hosts while the repository still contains the change is a rollback that the next scheduled converge undoes - and the second occurrence is much more confusing than the first, because now the change is arriving with no operator behind it.

Use git revert, not a force push. The reverted commit stays in history, which is what you need if the decision changes and the change has to be re-applied.

Step 2: Measure who received it

Read-only / Safedo not trust the change record

# Checksum the managed file across the whole plausible scope
ansible web -b -m stat \
-a 'path=/etc/nginx/conf.d/site.conf checksum_algorithm=sha256' -o \
| tee measured-config.txt

# Group hosts by checksum
grep -oE '"checksum": "[a-f0-9]+"' measured-config.txt | sort | uniq -c

Two checksums means two states, and the counts tell you the split. Build the target list from that:

Read-only / Safebuild the rollback target list

NEWSUM=REPLACE_ME_WITH_THE_NEW_CHECKSUM
grep "$NEWSUM" measured-config.txt \
| grep -oE '^[a-z0-9.-]+\.example\.com' | sort -u > to-rollback.txt
wc -l to-rollback.txt

Step 3: Locate the previous state

Concretely, per host. “How it was before” is not a rollback target.

From the backup the task took

Read-only / Safefind the backups

ansible "$(paste -sd, to-rollback.txt)" -b -m find \
-a 'paths=/etc/nginx/conf.d patterns="site.conf.*" file_type=file' -o \
| python3 -m json.tool | grep -E '"path"|"mtime"' | head -40

backup: true on template and copy writes a timestamped copy beside the original. That is the per-host rollback material, and it is the only source that reflects what that host had - which may differ from what the repository says it had, if the host had drifted.

From the repository

Read-only / Saferender the previous version

cd /srv/automation/repo
git show 9a3f1c2~1:roles/nginx_frontend/templates/site.conf.j2 > /tmp/previous.j2
diff <(git show 9a3f1c2~1:roles/nginx_frontend/templates/site.conf.j2) \
   roles/nginx_frontend/templates/site.conf.j2

After the git revert in Step 1, the repository is the previous version, so re-running the playbook renders the old content. That is the cleanest rollback when it applies - and it only applies if the file is fully managed.

From a package version

Read-only / Safeprevious package version

ansible "$(paste -sd, to-rollback.txt)" -b -m command \
-a 'dnf history list --reverse' -o | tail -5

Step 4: Validate before activating

Read-only / Safevalidate

# nginx: check a specific file without activating it
ansible "$(paste -sd, to-rollback.txt)" -b -m command \
-a 'nginx -t -c /etc/nginx/nginx.conf' -o

# sshd
ansible "$(paste -sd, to-rollback.txt)" -b -m command -a 'sshd -t' -o

# A rendered candidate, before it replaces the live file
ansible "$(paste -sd, to-rollback.txt)" -b -m command \
-a 'nginx -t -c /tmp/candidate.conf' -o

Step 5: Roll back as a rolling change

Service impact possiblerollback.yml

- name: Roll back the site configuration
hosts: "{{ rollback_targets }}"
become: true
serial: "25%"
max_fail_percentage: 0
vars:
  backup_path: /etc/nginx/conf.d/site.conf.REPLACE_ME
tasks:
  - name: The backup we are about to restore exists
    ansible.builtin.stat:
      path: "{{ backup_path }}"
    register: bk
    failed_when: not bk.stat.exists

  - name: Stage the restored content without activating it
    ansible.builtin.copy:
      src: "{{ backup_path }}"
      dest: /etc/nginx/conf.d/site.conf.candidate
      remote_src: true
      owner: root
      group: root
      mode: '0644'

  - name: Drain from the load balancer
    ansible.builtin.uri:
      url: "https://lb.example.com/api/pool/web/{{ inventory_hostname }}/drain"
      method: POST
      status_code: [200, 204]
    delegate_to: localhost

  - name: Activate the restored configuration
    ansible.builtin.copy:
      src: /etc/nginx/conf.d/site.conf.candidate
      dest: /etc/nginx/conf.d/site.conf
      remote_src: true
      owner: root
      group: root
      mode: '0644'
      backup: true

  - name: Configuration is valid before we ask the service to load it
    ansible.builtin.command: nginx -t
    changed_when: false

  - name: Reload the service
    ansible.builtin.systemd_service:
      name: nginx
      state: reloaded

  - name: Service answers a real request
    ansible.builtin.uri:
      url: "http://{{ ansible_host }}:8080/healthz"
      status_code: 200
      return_content: true
    register: health
    retries: 6
    delay: 5
    until: health.status == 200 and 'ok' in health.content

  - name: Return to the load balancer pool
    ansible.builtin.uri:
      url: "https://lb.example.com/api/pool/web/{{ inventory_hostname }}/enable"
      method: POST
      status_code: [200, 204]
    delegate_to: localhost

Read-only / Safeconfirm the target list, then run

LIMIT=$(paste -sd, to-rollback.txt)
ansible-playbook -i inventories/production rollback.yml \
-e "rollback_targets=$LIMIT" --limit "$LIMIT" --list-hosts

ansible-playbook -i inventories/production rollback.yml \
-e "rollback_targets=$LIMIT" --limit "$LIMIT" --diff | tee rollback.log

Note the first task: it fails if the backup is not there. A rollback that silently proceeds with a missing source file is how a host ends up with an empty configuration, and the failure is much better placed before the drain than after the reload.

Note also backup: true on the activation step. It captures the bad configuration before overwriting it, which you will want when someone asks what exactly was wrong with it.

Step 6: Verify that the process loaded it

Read-only / Safefile and process

# The file is back - compare against a known-good checksum
ansible "$LIMIT" -b -m stat \
-a 'path=/etc/nginx/conf.d/site.conf checksum_algorithm=sha256' -o \
| grep -oE '"checksum": "[a-f0-9]+"' | sort | uniq -c

# The process loaded it - config query beats timestamps where available
ansible "$LIMIT" -b -m command -a 'nginx -T' -o | grep -c 'listen 8080'
ansible "$LIMIT" -b -m command \
-a 'systemctl show nginx -p ExecMainStartTimestamp' -o

A restored file with an unreloaded service is exactly the state the failed-handler runbook exists for. Under rollback pressure it is easy to produce: the copy task succeeds, the reload fails or is skipped, and the verification reads the file rather than the process.

Step 7: Confirm one state, and reconcile

Read-only / Safewhole scope in one state

ansible web -b -m stat \
-a 'path=/etc/nginx/conf.d/site.conf checksum_algorithm=sha256' -o \
| grep -oE '"checksum": "[a-f0-9]+"' | sort | uniq -c

# One checksum, count equal to the host count
ansible web --list-hosts | grep -c 'example.com'

# Repository and hosts agree
ansible-playbook -i inventories/production site.yml --limit web --check --diff \
| grep -E 'changed=[1-9]'

That last check is the reconciliation. If the repository was reverted in Step 1 and the hosts were restored in Step 5, a check-mode converge must report no changes. If it reports changes, the two halves disagree - and the next scheduled converge will act on that disagreement without supervision.

Step 8: Record the window

When the change was applied, to which hosts, and when it was reversed.
Which hosts were never affected - useful, because it explains why their logs look different.
What could not be reversed, and what was done instead.
Whether any host lacked backup material, and how its previous state was determined.

What cannot be rolled back

Restoring a file does not reverse an action

Configuration files revert cleanly. These do not:

A data migration or on-disk format conversion. Old code on new data is a third broken state, not the original one.
A rotated credential. The old value may still be revoked at the other end; restoring the config file restores a reference to a credential that no longer works.
A call to an external system. A DNS record changed, a certificate issued, a ticket raised, a message published - none of these are undone by a file.
A deletion. A removed user, a dropped table, a deleted file. The configuration that said to remove it can be reverted; the thing is still gone.
A service restart. The previous process is not recoverable, and anything it held in memory is gone.

Classify every step of the change against this list before starting the rollback. If any host reached one of these steps, the honest position is that those hosts are forward-only, and that has to be stated rather than discovered halfway through.

Common patterns

Symptom	Likely cause	Resolution
Healthy hosts broken by the rollback	It ran against hosts that never had the change	Build the limit from measured checksums
Restored file, problem persists	The service never reloaded	Check `ExecMainStartTimestamp` against the file mtime
Service will not start after rollback	Restored content is invalid - wrong backup, bad render	Validate before activating; restore from another source
Change re-appears on the next converge	The repository was never reverted	`git revert` first, hosts second
No backup file exists	The task did not set `backup: true`	Restore from the repository; state that host drift is unaccounted for
Rollback succeeded, dependent system broke	Something was built on the change	Forward fix; this was an escalation, not a rollback
Two checksums remain at the end	The rollback stopped partway	Do not leave it split; finish or reverse deliberately
Rollback restored the file, credentials still fail	The credential was rotated as part of the change	Not reversible by file restore

Escalation

Escalate when:

The service does not support reload and a restart is needed.
The previous state cannot be located on some hosts.
Something has been built on top of the change.
Any host reached an irreversible step.
The change is a security control.

Why this matters

A rollback is a change made by people who are already under pressure, against a system they have just been surprised by. That combination is why it needs more structure than the original change, not less.

The two habits that matter are both about not trusting the record. Measure which hosts actually hold the change, because the change record describes what was intended and a stopped rollout is precisely the case where intent and reality diverge. And validate the restored configuration before activating it, because a degraded service is survivable and a service that cannot start is not.

The third is knowing, before you begin, which parts of the change have no way back. Every rollback plan that turns into a second incident does so at the moment someone discovers that restoring the file did not restore the state.