Runbook: Audit a production execution

Someone asks: what did automation do to db01.example.com on the fourteenth, and who told it to?

Answering that is easy if the estate was set up to answer it and impossible if it was not, and the honest first move is finding out which of those you are in. Ansible ships with logging disabled. On a default installation there is no record of any run beyond what happened to be on someone’s terminal.

This runbook does two jobs. It assembles the best answer the available evidence supports, and it names the evidence that does not exist - so that the next audit is answerable rather than being the same conversation.

When to use this runbook

• A change appeared on a host and nobody knows where it came from.
• A compliance or incident question about what automation did, and when.
• Post-incident: establishing whether automation contributed.
• Verifying that a change record matches what actually happened.

Blast radius

None. Reading logs, Git history and file metadata. Any configuration change made to close a gap is an ordinary change afterwards.

Step 1: Establish what evidence exists

Before searching, find out what there is to search.

ansible-config dump --only-changed | grep -iE 'log_path|callback'
ansible-config dump | grep -iE 'DEFAULT_LOG_PATH|CALLBACKS_ENABLED'

Evidence sources, in descending order of usefulness:

Step 2: Search the controller log

LOGFILE=/var/log/ansible/ansible.log
HOST=db01.example.com

grep -n "$HOST" "$LOGFILE" | grep '2026-08-14' | head -50

# Every distinct run that touched it that day
grep '2026-08-14' "$LOGFILE" | grep -E 'PLAY \[' | sort -u

Verified format on 2.21.3 with ANSIBLE_LOG_PATH set:

2026-08-11 23:07:56,233 p=377026 u=ebrandi n=ansible INFO| PLAY [Rolling deployment]
2026-08-11 23:07:56,236 p=377026 u=ebrandi n=ansible INFO| TASK [Deploy the application]
2026-08-11 23:07:56,245 p=377026 u=ebrandi n=ansible INFO| changed: [db01.example.com]

Four useful fields per line: timestamp, p= process id, u= the OS user who invoked it, and the message.

u= is the single most valuable field for attribution, and it is also where the estate’s design shows. If every run is u=ansible, the log tells you a service account did it and nothing about which human. That is a finding about the estate, not about this incident.

grep "$HOST" "$LOGFILE" | grep '2026-08-14' | grep -E '^.*changed:' \
| sed 's/.*INFO| //' | sort | uniq -c

changed: lines are the ones that matter. ok: means the task ran and altered nothing - which is the majority of every converge, and filtering it out is what makes an audit tractable.

Step 3: Correlate each run to code and to a person

cd /srv/automation/repo

# What was on the production branch at that time?
git log --until='2026-08-14 23:59' --format='%H %ci %an %s' -5 main

# What did the role look like at that commit?
git show 4f2a9c1:roles/postgres/tasks/main.yml | head -40

# CI, if runs go through a pipeline
echo 'Pipeline history: job id, triggering user, commit, timestamp'

# Interactive invocations: who was on the controller
last -F | grep '2026-08-14'
sudo journalctl -t sudo --since '2026-08-14' --until '2026-08-15' --no-pager \
| grep -i 'ansible'

Step 4: Cross-check against the host’s own record

The controller says what it believes it did. The host records what happened. Compare.

HOST=db01.example.com

# Files under /etc modified in the window
ansible "$HOST" -b -m find \
-a 'paths=/etc recurse=true age=-3d age_stamp=mtime file_type=file' -o \
| python3 -m json.tool | grep '"path"' | head -40

# Package transactions
ansible "$HOST" -b -m command -a 'dnf history list --reverse' -o | tail -20

# Service restarts and sudo activity
ansible "$HOST" -b -m command \
-a 'journalctl --since "2026-08-14" --until "2026-08-15" --no-pager -u postgresql' -o
ansible "$HOST" -b -m command \
-a 'journalctl -t sudo --since "2026-08-14" --until "2026-08-15" --no-pager' -o

# Files written by template/copy with backup: true leave a dated copy
ansible "$HOST" -b -m find \
-a 'paths=/etc/postgresql patterns="*.conf.*" file_type=file' -o \
| python3 -m json.tool | grep -E '"path"|"mtime"'

Backup files are the most underrated audit source in an Ansible estate. Each one is a timestamped copy of the previous content, written by the task that changed it, sitting on the host. Where the role sets backup: true, you can reconstruct not just when a file changed but what it changed from - without any controller log at all.

Step 5: Find changes with no run behind them

# A file changed on the host at a time when no run touched it
ansible "$HOST" -b -m stat -a 'path=/etc/postgresql/postgresql.conf' -o \
| python3 -m json.tool | grep -E 'mtime|checksum|pw_name'

# Does the controller log show a run at that time?
grep "$HOST" /var/log/ansible/ansible.log | grep '2026-08-14 1[0-9]:'

Three possible explanations, and they need distinguishing:

1. Manual change. Someone edited the file. The sudo log and the journal may say who.
2. A run from a controller you have not searched. This is the common one, and it is why the pre-check asks which controllers existed in that window.
3. A run whose log was never written. Logging off, retention expired, or the run went to a terminal only.

Explanation 2 is worth pursuing hard. A second controller - a developer’s laptop, a decommissioned box still holding a valid key, a CI runner nobody registered - is both the answer to the audit and a finding in its own right.

Step 6: Assemble the timeline

Every entry attributed to its source:

2026-08-14 09:12  CI job #4417, commit 4f2a9c1, triggered by a.operator
                  ansible-playbook site.yml --limit db
                  Source: pipeline history
2026-08-14 09:14  db01: changed - Render postgresql.conf
                  Source: controller log /var/log/ansible/ansible.log
2026-08-14 09:14  db01: handler Restart postgresql ran
                  Source: controller log; corroborated by journal entry
                  for postgresql.service restart at 09:14:22
2026-08-14 14:41  db01: /etc/postgresql/pg_hba.conf mtime changed
                  NO corresponding automation run found.
                  sudo log shows b.operator escalated at 14:39.
                  Attribution: manual change, unconfirmed.
2026-08-14 22:00  Scheduled converge, u=ansible, commit 4f2a9c1
                  db01: changed=0
                  Source: controller log

The 14:41 entry is the useful one. An audit that reports only what it found is less valuable than one that reports what it found and what it could not attribute.

Step 7: State the limitations

This section is not a disclaimer. It is a finding.

Limitations of this audit:

- Controller log retention is 30 days. Anything before 2026-07-15
  cannot be established.
- Runs made interactively without CI cannot be attributed to a person
  beyond the OS user, because operators SSH directly into the
  automation account.
- Ad-hoc `ansible` commands are logged as tasks but not as an
  invocation, so the command line that produced them is not recoverable.
- controller02 was decommissioned on 2026-08-01; its logs were not
  archived. Any run from it in the window is unrecoverable.
- No callback plugin is configured, so per-task diffs are not available;
  only that a task reported changed.

Step 8: Close the gaps

The audit’s real deliverable. Each of these is an ordinary change.

[defaults]
# Every run written to a file, with timestamp, pid and invoking user.
# This is the single highest-value line in this runbook.
log_path = /var/log/ansible/ansible.log

[privilege_escalation]
become = False

That one setting produces the timestamped, attributed record that Step 2 searches. Everything else is an improvement on top of it.

For a structured record, ansible.posix.json is a stdout callback - it replaces the human-readable output rather than adding to it, so it is set per invocation rather than globally:

ANSIBLE_STDOUT_CALLBACK=ansible.posix.json \
ansible-playbook site.yml --limit db01.example.com --check \
> "run-$(date -u +%Y%m%dT%H%M%SZ).json"

Check which callbacks are actually available before writing either into a config file - ansible-core ships very few, and the rest come from collections you must pin in requirements.yml:

ansible-doc -t callback -l
ansible-config dump | grep -iE 'CALLBACKS_ENABLED|STDOUT_CALLBACK'

# /etc/logrotate.d/ansible
/var/log/ansible/ansible.log {
  daily
  rotate 400
  compress
  missingok
  notifempty
  create 0640 ansible ansible
}

Other gaps worth closing:

• Route runs through CI. Then attribution is the pipeline’s job and it is solved by default.
• Forbid direct SSH to the automation account. sudo -iu ansible from a named account preserves the chain to a person.
• Set backup: true on every task that writes a managed file. It gives you per-host, timestamped before-content on the host itself.
• Register every controller. An unregistered controller is a permanent hole in every future audit.
• Fix non-idempotent tasks. They make the changed signal meaningless.

Common patterns

Escalation

Escalate when:

• An execution is found that nobody can account for.
• The evidence does not exist and the question is a compliance or security question.
• Runs came from a controller that is not in the register.
• The log contains plaintext secrets.