"Who changed three hundred servers?" · Ansible

Somebody senior asks the question in the title. It sounds like one question and it is five, and the reason it is hard is that no single system holds more than two of the answers.

What changed — the managed node knows, in detail.
When — both ends know, to the millisecond, if their clocks agree.
Which hosts — only the controller knows, and only if the run recorded its own target set.
Which version of the automation — only the repository knows, and only if the run recorded its commit.
Who — nobody knows, unless you arranged for somebody to.

This lesson is about assembling those five from evidence that exists in three places, and about the last one, which is the only genuinely hard part.

The evidence chain

Three sources, joined on two keys.

Source	Answers	Does not answer
Controller run artefact and log	which hosts were targeted and reached, which limit, which commit, which OS account ran it, exact timestamps	what actually changed on disk
Managed-node journald and auditd	which files changed, which services restarted, which commands ran, under which account	why, or which run it belonged to
Package transaction history	which packages moved from which version to which, and when	who asked for it

The joins are timestamp and the automation account, per host. Both are fragile in ways worth knowing before you need them.

The Linux course teaches the managed-node half properly and this lesson does not repeat it: auditd architecture, audit record anatomy and attribution, ausearch and aureport, journalctl filters and authentication and sudo logs. If you are going to rely on that half, read those first — the controls described there are what make the join possible at all, and central logging design is what stops the evidence living only on a host that may be the one you just broke.

Working the chain, in order

Start at the host, because that is where the symptom is, and move outward.

1. What changed, and when. On the affected host, the service journal around the reported time, then the audit trail for the file. This is ordinary Linux forensics and the Linux course covers the syntax.

2. Which account did it. It will be your automation account. That is the expected answer and it is also the dead end — see the next section.

3. Take the timestamp to the controller. With the controller log enabled, the window is searchable:

Read-only / Safefind the run that was in flight at that moment— Searches the controller log for a time window. Reads a file; runs nothing.

WINDOW='2026-08-11 14:3'
grep -F "$WINDOW" /var/log/ansible/ansible.log | head -40

The p= field groups the lines belonging to one run, and u= names the operating-system account that started it.

4. Take the run to the artefact. The run_id gives you the commit, the inventory, the limit and whether it was --check.

5. Take the commit to the repository. git show names the author, git log gives the message, and the merge request or review names the approver. This is where a human finally appears — and notice it is the human who wrote the change, not necessarily the human who ran it.

6. Reconcile the host list. Intended targets from the limit, minus the hosts that reported results, equals the hosts that were targeted and never reached. As lesson 1 established, those hosts are absent from the recap rather than listed with zeros, so this subtraction is the only way to find them.

Six steps, three systems, two joins. None of the systems can be dropped.

The attribution problem

Here is why the managed node cannot answer “who”, ever.

Ansible connects to every host as one account. That is the whole design — the least privilege for the automation account lesson argues for exactly one, tightly scoped. So the host’s audit record says the change was made by ansible, on every host, for every change, by every operator, forever.

The audit trail on a managed node is therefore complete about what and useless about who. Three hundred hosts, three hundred identical answers, and the person is not in any of them.

The human name exists in exactly one place at the moment of the run: the controller. And it exists there only if three things are true.

Three ways the human name is lost before it is recorded

Shared interactive access to the controller. If several people log in as the same account to run playbooks, u= in the controller log is that account and the human is gone at the first step. This is the single most common cause, and it is why the shared versus personal controllers lesson treats controller access as an identity decision.

Scheduled runs with no requester. A nightly job has no human, correctly. The failure is when a change is smuggled into one — a variable edited in the repository takes effect on the next scheduled convergence, and the run record names the scheduler. The commit names the author, which is why step 5 is not optional.

Retention shorter than the question. The controller log rotated away after seven days; the question arrived after thirty. The evidence existed and was deleted on a schedule nobody chose deliberately.

Controls that put a name back

Four, in increasing order of strength. Each survives a different attack on the evidence.

1. Individual accounts on the controller, plus SUDO_USER. If people log in as themselves and escalate to the automation account, u= records the escalated account and SUDO_USER in the environment records the human. Stamp it into the artefact, as lesson 4 does. Weak against someone who logs in directly as the automation account; strong against ordinary use.

2. Runs go through a system that authenticates the requester. CI, or a controller platform with job records — the platform object model lesson describes runs as first-class objects with an owner. The identity is established before the run starts, by a system whose job is identity. Strong, and it has the side effect of making ad-hoc runs the exception, which is desirable for other reasons.

3. Per-operator SSH keys for the automation account. The automation account’s authorized_keys contains one key per operator, each with a distinct comment. The managed node’s authentication log then records which key authenticated — so the host itself holds a per-operator record, not just the controller.

This is the strongest control in the list, because it is the only one where the evidence lives on the target rather than on the machine that made the change. Controller evidence can be lost with the controller; the host’s auth log cannot. Its cost is real: key distribution and rotation across the fleet, which the key-rotation lesson in Part XLVII treats as its own operation.

4. A change identifier carried into the target. The ticket or change id is stamped into the run artefact and into a managed header on every file the run renders — the pattern from template review and managed headers. Then the file on disk names the change that produced it, and the chain can be walked backwards from the host with no controller evidence at all.

None of these is expensive. What they have in common is that they must be in place before the question is asked; every one of them is a decision about how runs are made, not a query you can run afterwards.

Under the hood

What does the audit trail on a managed node actually show when Ansible changes a file? Not what most people expect, and the difference matters when you write the audit rule.

copy and template do not open the destination and write to it. They transfer the content to a temporary path in the remote temporary directory, set ownership and mode there, and then perform an atomic move onto the destination — the mechanism the backups and atomic file replacement lesson describes, and the reason a reader of that file never sees a half-written version.

The consequence for auditing is that the record is a rename, not a series of writes. An audit rule watching the destination path for write access may see very little; a rule watching for attribute and rename operations sees the change. If you are building audit rules specifically to catch configuration changes made by automation, test them against a real Ansible run rather than against a hand-edit with an editor, because the two produce different syscall patterns.

There is a second consequence for the join. The temporary path contains a random component and lives under the remote temporary directory, so the audit record’s path field will not obviously correspond to the file you care about unless the rename is captured. Where the audit configuration is not yours to change, the more reliable managed-node evidence is often the service journal — a restarted unit is unambiguous and timestamped — and the package transaction history, which records exactly which versions moved and when.

What happens if the clocks disagree?

The join breaks, silently, and the reconstruction produces a confident wrong answer rather than no answer.

The timestamp is the primary join between controller and host. A host whose clock is ninety seconds fast places its change before the run that caused it, and a reconstruction that searches a two-minute window around the host’s timestamp finds the previous run — which was a different commit, a different limit and possibly a different operator.

That is worse than failing to find anything, because it names somebody.

Three practical defences:

Treat time synchronisation as an audit control, not a convenience. It is one of the few settings whose absence corrupts evidence rather than breaking a service, and it belongs in the baseline role that every host gets.
Search a generous window and expect several candidates. Then disambiguate with something other than time: the run’s host list, its commit, what it actually changed.
Record timestamps in UTC everywhere. A controller in one timezone and a fleet in another produces an hour-shaped error that looks exactly like clock skew and takes far longer to spot. The controller log records local time, so the offset is something you must know rather than assume.

The deeper point: every join in this chain is an inference. Correlating on time and account tells you which run probably made a change. The controls in the previous section — a change id in the file, a per-operator key in the auth log — replace inference with a direct record, and that is exactly why they are worth their cost.

The automation audit runbook

Given a hostname and a date, produce the change and its author.

On the host — the service journal and audit trail around the reported time, for the file or unit in question. Record the exact timestamp and the account.
Confirm the account is the automation account. If it is not, this is not an automation change and the rest of this runbook does not apply.
On the controller — grep the log for the window. Note p= and u=.
Find the run artefact for that run_id. Read: commit, inventory, limit, check mode, host results.
In the repository — git show the commit for the author and the message; the merge request for the approver.
Reconcile — intended targets minus reported hosts. Anything in the difference was targeted and not reached, and needs its own investigation.
Write down what you could not establish. A reconstruction with a named gap is evidence. One that quietly infers across a gap is not.

Preconditions, worth verifying before an incident rather than during one: controller logging enabled with a retention longer than your longest question, time synchronised on every host, individual identity at the point runs are started, and artefacts that record the limit.

Knowledge check

Why this matters

The uncomfortable fact underneath this lesson is that automation makes attribution structurally harder, not easier.

Before automation, three hundred changes meant three hundred logins, and each one carried a name. The audit trail was tedious and complete. Automation replaces that with one account making three hundred identical changes in ninety seconds, and the account is the same account every time. The change record got better in every dimension except the one an auditor asks about first.

That is not a flaw to apologise for; it is the trade that makes fleets manageable. But it does mean the human name has to be reintroduced deliberately, at the point where the run is started, by a decision made in advance. There is no query that recovers it afterwards.

So the honest summary is short. Git history, CI or controller job records, the run artefact and the managed node’s own logs each answer part of the question, and none answers it alone. Assembling them is a runbook, not a tool, and the preconditions — synchronised clocks, individual identity, retention longer than your longest question — have to be true before anyone asks.