What an ad-hoc change leaves behind · Ansible

Three weeks after the change, someone asks: who reconfigured the web tier on the 14th, what exactly did they change, and was it reviewed?

If the change was a playbook, the answer is a commit hash. If it was an ad-hoc command, the honest answer is that nobody knows, and the reconstruction that follows will take a day and end in a guess.

This lesson establishes the position the rest of the course takes: an ad-hoc inspection is fine forever. An ad-hoc change is a debt, and the interest is paid during the incident review.

What is actually recorded

By default, an ad-hoc run writes nothing anywhere. Enumerate the places you might look:

Where you might look	What is there
A log on the controller	Nothing, unless `log_path` is set. It is unset by default.
A log on the managed node	Nothing from Ansible. `sudo` logs the escalation if `-b` was used.
Version control	Nothing. There was no file.
A change ticket	Only if a human wrote one.
The terminal	Everything, until the terminal closes.
Shell history	The command line, on one workstation, for one user.

The last row is the one people lean on, and it is worth being precise about why it does not work.

Shell history is not an audit trail

It fails on every property an audit trail needs.

It is not durable. HISTSIZE is finite and default shells rotate. history -d removes an entry. A shell killed rather than exited may never flush.

It is not attributable. It records that a login session ran a command, not who was at the keyboard, and shared operator accounts are still common.

It is not timestamped, unless HISTTIMEFORMAT was set beforehand. Without it you have an ordering and no clock.

It does not record the outcome. You can see the command. You cannot see which hosts it reached, which succeeded, which were unreachable, or what the module actually did. A command that failed on 40 of 300 hosts looks identical in history to one that succeeded everywhere.

It is on the wrong machine. The evidence lives on the workstation of the person under discussion, who has write access to it.

The four questions a change record must answer

An incident review asks these. Test any record you have against all four.

Who ran it?
What did it do — not what was typed, but what changed on each host?
Where — which hosts, exactly, and which of them actually changed?
What was the previous state, so it can be restored?

A playbook in Git answers 1 and 2 from the commit, 3 from the recap and the inventory at that revision, and 4 from the diff plus whatever the change replaced.

Shell history answers a corrupted version of 2 and nothing else. Note particularly that question 4 is unanswerable by any ad-hoc mechanism. No module you ran recorded the prior value; lineinfile replaced a line and did not keep the old one unless you passed backup=yes, which nobody does at a prompt.

The two compensating controls

If you are going to run ad-hoc changes — and every estate does, sometimes — these are the mechanisms that make it partially defensible. Neither is on by default.

`log_path`: the controller writes a log

Read-only / Safelog every ad-hoc run— Sets the controller log path for one run. Every Ansible invocation on this controller appends to the file, with timestamp, PID and the local user.

$ ANSIBLE_LOG_PATH=./adhoc.log ansible localhost -m ansible.builtin.ping

Read-only / Safewhat it wrote— Reads back the log written by the previous command. Timestamp, PID, local user and the module result verbatim.

$ cat ./adhoc.log

2026-08-11 21:01:05,787 p=4037136 u=opsuser n=ansible WARNING| [WARNING]: No inventory was parsed, only implicit localhost is available

2026-08-11 21:01:05,989 p=4037136 u=opsuser n=ansible INFO| localhost | SUCCESS => {
  "changed": false,
  "ping": "pong"
}

Illustrative output

Set it permanently in the project ansible.cfg so every operator on the controller gets it without thinking:

[defaults]
log_path = /var/log/ansible/ansible.log

Read-only / Safeconfirm it is in effect— Shows the resolved value and its source. If this reports null, nothing on this controller is being logged.

$ ansible-config dump | grep DEFAULT_LOG_PATH

DEFAULT_LOG_PATH(/home/opsuser/estate/ansible.cfg) = /var/log/ansible/ansible.log

Illustrative output

What this buys: attribution and timing, for runs made on that controller by that config. It records u= — the local user on the controller — and the per-host results.

What it does not buy: anything about a run made from a laptop, from a different directory with a different ansible.cfg, or with ANSIBLE_LOG_PATH unset in the environment. It is a controller-local convention, not an enforcement mechanism, and an operator who bypasses it leaves no trace of having done so.

`--tree`: the results become files

Read-only / Safecapture per-host results— Writes one JSON file per host into the named directory, containing the module return value verbatim. Works for ad-hoc runs specifically.

$ ansible production -i inventory.ini -m ansible.builtin.setup -a "filter=ansible_kernel" --tree ./run-2026-08-11

This is the only ad-hoc mechanism that captures the outcome rather than the intent. One file per host that answered, containing exactly what the module returned — including changed, so you can tell which hosts actually moved.

It answers question 3 properly, which shell history cannot touch. It still says nothing about question 4.

Why there is no server-side record

Ansible is agentless. There is no daemon on the managed node keeping a journal of what Ansible did, because there is nothing on the managed node that knows it is being managed.

What actually happens is: SSH in, create a temporary directory under the connecting user’s home, copy a self-contained Python module into it, execute it, capture stdout, delete the directory, disconnect.

From the target’s perspective that is a user logging in and running a Python script. The evidence it leaves is the evidence any SSH session leaves — an sshd entry in the auth log, a sudo entry if -b was used, and whatever the module itself touched. Nothing says “Ansible”, and nothing says which module or which arguments.

This is a design consequence, not an oversight. The same property that means you do not have to install and maintain an agent on 2000 hosts means those hosts cannot tell you what was done to them. Auditability, in an agentless system, has to be built on the controller side — which is exactly why log_path and --tree are the levers, and why both are things you must remember to pull.

What the auditor is actually asking

Compliance regimes phrase it differently, but the question is the same one an incident review asks, and it has a shape:

Demonstrate that changes to production systems are authorised, recorded, and attributable to an individual.

Take that clause apart against an ad-hoc change:

Authorised — by whom? There was no review step. The authorisation and the execution were the same keystroke.
Recorded — where? See the table at the top of this lesson.
Attributable — to the shell history of a workstation, if it still exists, if the account was not shared, and if the timestamps were enabled.

None of the three survives. And the failure is not that the change was wrong. It may have been exactly right. The failure is that you cannot demonstrate anything about it, which under most regimes is treated identically to not having controlled it at all.

What happens if you have to reconstruct an ad-hoc change afterwards?

You piece it together from adjacent evidence, and you will not get all the way there.

On the managed nodes, the auth log shows the SSH session and the sudo invocation, which gives you a time window and the connecting account. The sudo log records the command that was escalated — which for Ansible is a Python interpreter running a temporary file that no longer exists. You get a timestamp, not a change.

File mtimes on the changed hosts narrow the window further, and if the file is under configuration management elsewhere, a checksum comparison tells you which hosts differ from expectation. This is often the most useful evidence and it only tells you that something changed.

On the controller, log_path if it was set, --tree output if it was used, and shell history for whichever account you can identify.

You will typically end up able to say “something changed these 40 hosts between 02:10 and 02:14, from this controller, via this account”. You will not be able to say what the previous value was, and that is the thing you needed in order to put it back.

Reconstructing this from inadequate evidence is a break/fix scenario later in the course, and the lesson of that scenario is this one: the evidence is inadequate because nothing was designed to record it.

What is actually recorded

The four questions a change record must answer

The two compensating controls

log_path: the controller writes a log

--tree: the results become files

What the auditor is actually asking

Knowledge check

`log_path`: the controller writes a log

`--tree`: the results become files