Why a wrong changed breaks the machinery

Up to here, an inaccurate changed has been described as a reporting problem. It is not. changed is wired into the execution of the play, and getting it wrong changes what the play does to the host.

The wiring is one link long:

a task reports changed → its notify fires → the named handler is queued → the handler runs at the next flush → the service is restarted.

Break that link in either direction and you get one of the two failures below. They are a matched pair: the same defect, opposite signs, and wildly different severities.

Proving the link exists

The clearest demonstration is a play where nothing reports changed. Here is one, with a notify on a task that uses debug:

$ ansible-playbook -i inventory.ini order.yml

TASK [Take the host out of the load balancer pool] *****************************
ok: [web01.example.com] => {
  "msg": "pre_task 1"
}

TASK [Apply the site-specific tuning] ******************************************
ok: [web01.example.com] => {
  "msg": "task 1"
}

PLAY RECAP *********************************************************************
web01.example.com          : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web02.example.com          : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Three tasks in that play carry notify:. No handler ran, and the output contains no RUNNING HANDLER line at all. The run is entirely green.

Add changed_when: true to the notifying tasks — changing nothing else — and the handlers appear:

$ ansible-playbook -i inventory.ini order2.yml

TASK [Take the host out of the load balancer pool] *****************************
changed: [web01.example.com]
RUNNING HANDLER [Announce a pre_tasks handler ran] *****************************
TASK [webserver : Install the web server package] ******************************
ok: [web01.example.com]
TASK [webserver : Deploy the vhost configuration] ******************************
changed: [web01.example.com]
TASK [Apply the site-specific tuning] ******************************************
changed: [web01.example.com]
RUNNING HANDLER [webserver : Reload the web server] ****************************
RUNNING HANDLER [Announce a tasks handler ran] *********************************

PLAY RECAP *********************************************************************
web01.example.com          : ok=8    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The changed result is the trigger. Nothing else in the play changed.

Failure one: the task that always reports changed

A command task with no creates and no changed_when. It reports changed on every run because it cannot do otherwise. It notifies a handler. The handler restarts nginx.

The play runs nightly from cron. nginx is therefore restarted every night, on every web server, forever.

What it costs. A restart drops in-flight connections. It resets connection pools and warm caches. On a fleet with no serial, every web server restarts within the same few seconds, which is a brief but real capacity gap. And on a service that takes thirty seconds to warm up, this is a nightly self-inflicted latency spike.

Why the obvious diagnosis is wrong. The first hypothesis is always “something is crashing nginx”. Someone will spend an afternoon in journalctl looking for a segfault, because a service that restarts every night at 02:03 looks like a service with a problem.

The evidence that redirects the investigation is a correlation, not a log line: service uptime resets at exactly the time the Ansible run happens, and the run’s recap reports changed on the same task every night, including on nights when nobody deployed anything.

The diagnosis in one question: which task in that play reports changed on a run where nothing should have changed? On an accurate play, the answer on a converged host is “none”, and the question answers itself.

Failure two: the task that never reports changed

The same defect with the sign flipped, and much worse.

A task deploys a configuration file. Somebody added changed_when: false to it — perhaps to quieten a noisy recap, perhaps by copying a neighbouring probe task. The task still writes the file. It just no longer says so.

Its notify never fires. The handler never runs. The service is never reloaded.

What it costs. The file on disk is new. The process in memory is running the old configuration. Every check you would normally make says the change succeeded:

• The playbook run is green: changed=0, failed=0, exit 0.
• The file on disk contains the new content.
• Git shows the change was deployed.
• The dashboard shows no failed runs.

And the service is doing the old thing. A firewall rule that was supposed to close a port is on disk and not in the ruleset. A TLS configuration that was supposed to disable a weak cipher is on disk and not in the listener. A rate limit that was supposed to protect a backend is on disk and not in effect.

Why this is the worst outcome in the course. Every other failure mode announces itself. A failed task prints fatal:. An unreachable host prints UNREACHABLE!. A wrong target shows up in --list-hosts. This one produces a clean run and a wrong host, and the gap can persist for months — until the service restarts for an unrelated reason and suddenly starts behaving differently, which is its own confusing incident.

Why the obvious diagnosis is wrong. When the discrepancy is finally noticed, the first hypothesis is “the change was never deployed”. People check Git, check the run history, check the file on disk — and all three say it was deployed. The natural next step is to conclude the change does not do what was intended and to start rewriting it.

The evidence that redirects the investigation is a comparison of on-disk configuration against what the running process actually loaded. Most services will tell you: nginx -T dumps the loaded configuration, sshd -T the effective one, systemctl show the unit as loaded. Comparing that against the file is the diagnostic, and it is the one nobody thinks of first.

Repairing both

The repair is the same in both directions: give the task an accurate result.

And one structural repair that helps in both directions: notify from the task that actually changes the resource. A pattern where task A writes the file and task B (a command that always reports changed) carries the notify produces a handler that fires on every run and tells you nothing about task A.

Knowledge check