Skip to main content
RunBook Academy

← All checklists in Ansible

As neededchange-management

Checklist: Post-execution validation

19 items ·14 critical ·4 warn ·1 info

Run this immediately after a production run, while the terminal is still open and the log is still on screen. Several items read a saved run log; the commands assume you captured it, which means running the play as ansible-playbook ... 2>&1 | tee run.log or having a log_path configured on the controller.

Two behaviours verified by execution

Both of these were established by running them on ansible-core 2.21.3, because both cause runs to be recorded as successful when they were not:

A run whose only failure was under ignore_errors: true exits 0. The recap shows ignored=1; the exit code shows nothing. A pipeline that gates on the exit code alone accepts that run without comment.

Where a run has both a task failure and an unreachable host, the exit code is 4, not 2 - unreachable takes precedence. A pipeline written to treat 2 as the failure code will pass a run that both failed and never reached part of the fleet.

What a failure means

The purpose of this checklist is to stop a run being recorded as successful when it was not. Almost every item is critical, and that is honest here: each one describes a specific way the recap and the reality diverge.

The warn items are about the quality of the record rather than the state of the fleet, and the single info item is a prompt for the retrospective.

Access this needs

The log-reading items need only the saved output. The health checks connect to hosts and read: uri fetches, service_facts gathers, journalctl and systemctl status report. The convergence run and the check-mode run are ansible-playbook invocations against the target group - the convergence run is a real run, which is the point of it, and it changes nothing on a fleet that converged.

Where the evidence goes

The recap, the exit code, the convergence result and the health check outcome go into the change record. So does the list of hosts that failed, even if it is empty - “no hosts failed” recorded explicitly is a different statement from a record that says nothing, and only one of them survives a review.

Sign-off

  • Validated by: _____________ Time: ___________
  • Outcome (complete / partial / rolled back): ____________

Critical14 items

  1. ansible-playbook -i inventories/production playbooks/deploy.yml --limit web; echo "exit=$?"
  2. awk '/unreachable=[1-9]|failed=[1-9]|ignored=[1-9]|rescued=[1-9]/ {print "FINDING: " $0}' run.log
  3. echo "in recap:   $(grep -c ' : ok=' run.log)"
    echo "in pattern: $(ansible -i inventories/production web --list-hosts | tail -n +2 | wc -l)"
  4. awk '/ : ok=/ && !/changed=0/ {print $1}' run.log | wc -l
  5. ansible-playbook -i inventories/production playbooks/deploy.yml --limit web | awk '/changed=/ && !/changed=0/ {print "NOT CONVERGED: " $0}'
  6. ansible -i inventories/production localhost -c local -m ansible.builtin.uri -a 'url=https://web.example.com/health status_code=200 timeout=10'
  7. ansible -i inventories/production web -m ansible.builtin.service_facts --one-line
  8. ansible -i inventories/production web -m ansible.builtin.command -a 'journalctl --since -30min -p err --no-pager' --one-line
  9. awk '/ : ok=/ && (/failed=[1-9]/ || /unreachable=[1-9]/) {print $1}' run.log
  10. ansible -i inventories/production loadbalancers -m ansible.builtin.command -a 'systemctl status haproxy --no-pager' --one-line

Warning4 items

  1. awk '/ : ok=/ && !/skipped=0/ {print $0}' run.log
  2. ansible-playbook -i inventories/production playbooks/deploy.yml --limit web --check --diff
  3. ansible-config dump | grep '^DEFAULT_LOG_PATH'

Info1 item