RunBook AcademyAnsible

← All checklists in Ansible

Before deploymentchange-management

Checklist: Promoting a change to production

20 items ·14 critical ·5 warn ·1 info

Run this when a change is proposed for production, before the change record is approved - hours or days ahead of the run, not minutes. It is the wrapper around the change; the pre-execution checklist is the gate at the keyboard. Both exist because they are answered by different people at different times.

Why so many items have no command

Nine of the twenty items here have no command, and that is the point. This checklist is mostly about things a machine cannot observe: whether an approver looked at the change, whether the on-call engineer knows, whether the abort criteria were agreed before anybody had a reason to want to continue.

Where a command exists it is doing one of two things: confirming a precondition against the real hosts, or producing a number for the record. Neither replaces the attestation - “backups confirmed” needs a person willing to put their name next to it, and the find command only tells you a file exists.

Access this needs

Six commands connect to the target hosts: ping, service_facts, setup, find, and the systemctl list-timers invocation. All of them read; none changes anything. service_facts and setup are fact-gathering modules and find only walks a directory. The command module invocation is the one to look at twice - systemctl list-timers is read-only, but the command module will run whatever you give it, so do not adapt that item casually.

The rest are local: the controller clock, the Git revision, the collection list, the configuration dump.

Where the evidence goes

Everything here belongs in the change record, with a timestamp per item. The freshness item is not decoration: a reviewer reading the record afterwards needs to know whether “targets healthy” was checked five minutes before the run or the previous afternoon, because those are different claims and they look identical without a time next to them.

Sign-off

Change owner: _____________ Date: ___________
Approver: _________________ Date: ___________
On-call briefed by: _______ Time: ___________

Critical14 items

A change record exists, names an approver who is not the author, and states what will change on which hosts. "Which hosts" is a count and a group name, not "the web tier".
The current time is inside the agreed change window, and outside any freeze. Print the controller time in UTC and compare it against the window in the record - not against the clock on the wall in your timezone.
```
date -u '+%Y-%m-%d %H:%M:%S UTC'
```
The commit being promoted is byte-identical to the one that ran in staging. Not the same branch, not a rebase of it - the same hash. A cherry-pick that "only changed the inventory path" is a different change with no staging evidence.
```
echo "promoting: $(git rev-parse HEAD)"
echo "staged:    <hash recorded from the staging run>"
```
The ansible-core version and every collection version are identical between the staging run and this one. A collection that moved a minor version between the two runs means the staging evidence describes different module code.
```
ansible --version | head -1; ansible-galaxy collection list
```
The production inventory has been re-read since the last change to it, and the target group contains the hosts the record names. Prints the count to compare.
```
ansible-inventory -i inventories/production --list | jq -r 'to_entries[] | select(.key != "_meta" and .value.hosts) | "\(.key)\t\(.value.hosts | length)"'
```
The vault identity on the command line is the production one. Running a production play with a staging vault id either fails on decryption or - worse, if `vault_id_match` is off - succeeds and configures production with staging credentials.
```
ansible-config dump | grep -E '^DEFAULT_VAULT_IDENTITY_LIST|^DEFAULT_VAULT_ID_MATCH'
```
Every target host answers right now. Prints one line per host that does not; no output is the pass condition. A host that is already unreachable before the change will be recorded as a change failure afterwards, and will confuse the incident review.
```
ansible -i inventories/production web -m ansible.builtin.ping --one-line | grep -v 'SUCCESS'
```
Every target is healthy before the change, and the evidence is from minutes ago rather than from this morning. Changing a host that was already degraded means you will not be able to tell which failure was yours.
```
ansible -i inventories/production web -m ansible.builtin.service_facts --one-line | grep -v 'SUCCESS'
```
Every precondition above records when it was checked, not just that it was. "Backups confirmed" is a different statement on a backup from an hour ago and one from a fortnight ago, and the change record cannot tell them apart unless the age is written down.
A restorable backup of whatever this change can destroy exists and is younger than the stated interval. Prints one line per host with no matching backup in the last day; no output is the pass condition.
```
ansible -i inventories/production web -m ansible.builtin.find -a 'paths=/var/backups patterns=app-*.tar.gz age=-1d' --one-line | grep -v '"matched": [1-9]'
```
The backup has been restored at least once, somewhere, and the restore was timed. An untested backup is a file, not a rollback.
Losing an entire batch simultaneously still leaves enough capacity to serve. Print the disk and memory facts for the group and compare against the batch size the run will use - a batch that is a third of the tier is a capacity decision, not a speed decision.
```
ansible -i inventories/production web -m ansible.builtin.setup -a 'filter=ansible_memfree_mb,ansible_mounts' --one-line | grep -v 'SUCCESS'
```
The on-call engineer knows this is happening, knows what the failure looks like, and knows where the rollback is. Somebody who finds out about a change from the alert is an incident responder without context.
The specific conditions that stop the run are written down before it starts: which health check failing, how many hosts, how long without progress. "We will stop if it looks wrong" is not a criterion, and under pressure it always resolves to continuing.

Warning5 items

No other change is in flight against the same hosts, and no scheduled job - patching, certificate renewal, a backup window - will start during the run. Two automations converging the same file at the same time produce a state neither of them intended.
```
ansible -i inventories/production web -m ansible.builtin.command -a 'systemctl list-timers --all --no-pager' --one-line
```
The rollback path has been walked through at least once against a non-production host, and the person running the change has done it. Reading a rollback procedure for the first time during the incident it was written for is where rollbacks go wrong.
A named person owns post-change validation and knows what they are validating. Not "the team" - a name, and a set of checks they have seen before.
Where the change is visible to users, the notice has gone out with a window, an impact statement and a contact. Where it is not visible, that judgement is recorded so nobody has to re-derive it during the review.
The run will be attributable: an identifiable account or pipeline job, and a log path that survives the terminal. This is what the post-change review reads.
```
ansible-config dump | grep '^DEFAULT_LOG_PATH'; id -un
```

Info1 item

The runbook for this change failing is linked from the change record. Finding it is a task for now, not for the moment the change goes wrong.