Skip to main content
RunBook Academy

← All checklists in Ansible

Before deploymentchange-management

Checklist: Pre-execution

16 items ·13 critical ·3 warn ·0 info

This is the checklist you run in the last minute before a production run, with your hand on the keyboard. Everything on it is either a one-line command or a question you already know the answer to. It is deliberately the shortest checklist in this course, because a pre-flight nobody has time for is a pre-flight nobody runs.

It assumes the other checklists have already been done. The playbook was reviewed once, the role was reviewed once, the inventory was verified once. This one asks a different question: is the thing about to run the thing that was reviewed, against the hosts you think, with the stops still in place?

Running it as one block

The commands are written so that no output means pass. That makes the whole thing pasteable - export the two variables, run it, and read whatever it prints.

export VAULT_PASS_FILE="$HOME/.ansible/vault-pass-prod"
export LIMIT="web"
export PLAY="playbooks/deploy.yml"
export INV="inventories/production"

ansible-playbook -i "$INV" "$PLAY" --syntax-check
ansible-playbook -i "$INV" "$PLAY" --limit "$LIMIT" --list-hosts
echo "$(ansible -i "$INV" "$LIMIT" --list-hosts | tail -n +2 | wc -l) hosts"

Anything it prints is something to read before you press return. Anything it does not print, you have confirmed.

What a failure means

Stop. Every critical item here is a reason not to run, and none of them takes more than a minute to resolve or to escalate. The cost of stopping at this point is the cost of a delayed change; the cost of continuing is whatever the item was protecting you from.

The three warn items - lint, tests, monitoring - are confirmations that a gate further upstream really did fire on this commit. A failure there is a process problem to raise afterwards, not necessarily a reason to abort, but it does mean you are running with less assurance than you thought.

Access this needs

The revision, environment, inventory, pattern, variable, batching and threshold checks are all local reads: a checkout, the controller virtualenv, and nothing on the network except the git fetch. The secrets check needs the vault password. The staging and canary runs connect to hosts in check mode, so they need credentials and they will open SSH connections - but they change nothing.

Where the evidence goes

The commit hash and the host count go into the change record before the run, not after. They are the two facts that make a post-incident review possible: what ran, and where. Everything else can be reconstructed from the log; those two cannot, because after the fact the branch has moved and the inventory has changed.

Sign-off

  • Operator: _________________ Time: ___________
  • Change reference: _______________________

Critical13 items

  1. [ -z "$(git status --porcelain)" ] || echo "FINDING: working tree is dirty - this is not the reviewed commit"
git fetch -q origin && [ "$(git rev-parse HEAD)" = "$(git rev-parse origin/main)" ] \
  || echo "FINDING: HEAD is $(git rev-parse --short HEAD), origin/main is $(git rev-parse --short origin/main)"
git rev-parse HEAD
  2. have=$(ansible --version | sed -n '1s/.*\[core \(.*\)\].*/\1/p')
want=$(sed -n 's/^ansible-core==//p' requirements.txt)
[ "$have" = "$want" ] || echo "FINDING: controller runs core $have, repo pins $want"
comm -23 <(grep -vE '^[[:space:]]*(#|$)' requirements.txt | tr 'A-Z' 'a-z' | LC_ALL=C sort) \
         <(pip freeze | tr 'A-Z' 'a-z' | LC_ALL=C sort)
  3. ansible-inventory -i inventories/production --graph
  4. ansible-playbook -i inventories/production playbooks/deploy.yml --limit web --list-hosts
n=$(ansible -i inventories/production web --list-hosts | tail -n +2 | wc -l)
echo "$n hosts - does this match the change record?"
  5. ansible-inventory -i inventories/production --host web01.example.com
  6. grep -rlZ --include='*.yml' '^\$ANSIBLE_VAULT' inventories/production/ 2>/dev/null \
  | xargs -0 -r -n1 sh -c '
      ansible-vault view --vault-password-file "$VAULT_PASS_FILE" "$1" >/dev/null 2>&1 \
        || echo "FINDING: $1 will not decrypt with the configured vault password"' sh
  7. ansible-playbook -i inventories/production playbooks/deploy.yml --syntax-check
  8. ansible-playbook -i inventories/staging playbooks/deploy.yml --check --diff
  9. ansible-playbook -i inventories/production playbooks/deploy.yml --limit web01.example.com --check --diff
  10. python3 - <<'PY'
import yaml
for p in yaml.safe_load(open('playbooks/deploy.yml')) or []:
    if isinstance(p, dict) and 'hosts' in p:
        print(f"play {p.get('name')!r}: hosts={p.get('hosts')} serial={p.get('serial', 'NONE - sweeps the whole group')}")
PY
  11. python3 - <<'PY'
import yaml
for p in yaml.safe_load(open('playbooks/deploy.yml')) or []:
    if not isinstance(p, dict) or 'hosts' not in p: continue
    if 'max_fail_percentage' not in p and not p.get('any_errors_fatal'):
        print(f"FINDING: play {p.get('name')!r} has no failure threshold - it will not stop on its own")
PY
  12. ansible-playbook -i inventories/production playbooks/deploy.yml --list-tasks | tail -10

Warning3 items

  1. ansible-lint --nocolor -q --profile production playbooks/deploy.yml; echo "exit=$?"
  2. git log -1 --format='%H %ci %s'