The handler that never ran · Ansible

A notification is a promise to do something at the end of the section. A host that fails before reaching the end of the section never keeps it.

The resulting state is specific and it is worth being able to describe precisely: the new configuration is on disk, the running process is still holding the old configuration, and the recap reports that host as failed for an entirely different reason. Nobody reading the recap will connect failed=1 on a package task to a reverse proxy that is now running configuration from three weeks ago.

Watching it happen

Read-only / Safea failure between the notify and the flush— Debug and fail only, so nothing is modified. The first task notifies; the second fails; the flush point is never reached.

- name: A failure between the notify and the flush
hosts: local
gather_facts: false
tasks:
  - name: Deploy the configuration
    ansible.builtin.debug:
      msg: config written
    changed_when: true
    notify: Reload the service

  - name: A later task fails
    ansible.builtin.fail:
      msg: something went wrong

handlers:
  - name: Reload the service
    ansible.builtin.debug:
      msg: RELOADED

Read-only / Safeexecuted on ansible-core 2.21.3— No RUNNING HANDLER line. The recap says failed=1 and changed=1, and nothing in it mentions the handler.

$ ansible-playbook -i inv.ini fail.yml

TASK [Deploy the configuration] ************************************************
changed: [localhost]

TASK [A later task fails] ******************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "something went wrong"}

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

changed=1 and failed=1. The change happened; the consequence of the change did not. There is no counter anywhere for “handlers that were queued and discarded”, and no line of output that would let you tell this run apart from one where the failing task came first.

`force_handlers`

Ansible provides a switch for exactly this: run the notified handlers on a host even after that host has failed.

It exists in three forms, all equivalent:

Form	Scope
`--force-handlers`	one run
`force_handlers: true` (play keyword)	one play
`force_handlers = True` in `ansible.cfg` `[defaults]`, or `ANSIBLE_FORCE_HANDLERS=1`	every run, via `DEFAULT_FORCE_HANDLERS`

Read-only / Safethe same play with --force-handlers— Verified on ansible-core 2.21.3. Identical playbook. The host still fails; the handler runs anyway.

$ ansible-playbook -i inv.ini fail.yml --force-handlers

TASK [Deploy the configuration] ************************************************
changed: [localhost]

TASK [A later task fails] ******************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "something went wrong"}

RUNNING HANDLER [Reload the service] *******************************************
ok: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

The host is still reported failed, which is correct — something did go wrong. But the configuration it wrote is now live.

Is forcing the right thing?

Not always, and this is a genuine judgement call rather than a setting with a correct value.

Forcing is usually right when the handler applies configuration that is already on disk. The file was written; leaving the service out of step with it is worse than the failure that stopped the play. This covers the common case — a template task succeeded, a later package task failed on a repository timeout, and the reload should still happen.

Forcing is wrong when the handler is the dangerous half of the operation and the failure is evidence that the host is not in a fit state for it. A play that writes a new cluster configuration, fails validating quorum, and then restarts the cluster daemon anyway has taken a bad situation and made it an outage. Here the queued restart is the thing you want to not happen.

The distinguishing question: if this host is broken, does running the handler make it better or worse? A reload of already-written config is usually better. A restart of a service whose preconditions just failed to verify is usually worse.

The precise tool: flushing inside a `rescue`

force_handlers is a blunt instrument: every notified handler, on every failed host, regardless of what failed. A block/rescue gives you the same outcome scoped to a specific failure.

Service impact possibleapply what was written, then handle the failure— Verified structurally on 2.21.3 with debug tasks: the rescue-time flush ran the queued handler and the host finished with rescued=1 rather than failed=1.

- name: Deploy the reverse proxy configuration
hosts: webservers
become: true
tasks:
  - name: Configure and verify
    block:
      - name: Deploy the site configuration
        ansible.builtin.template:
          src: site.conf.j2
          dest: /etc/nginx/conf.d/site.conf
          mode: '0644'
          validate: nginx -t -c %s
        notify: Reload nginx

      - name: Confirm the upstream is reachable from this host
        ansible.builtin.wait_for:
          host: '192.0.2.40'
          port: 8080
          timeout: 10

    rescue:
      - name: Apply the configuration that was already written
        ansible.builtin.meta: flush_handlers

      - name: Record that this host needs attention
        ansible.builtin.debug:
          msg: 'Upstream check failed on {{ inventory_hostname }}; configuration applied, host needs review'

handlers:
  - name: Reload nginx
    ansible.builtin.systemd_service:
      name: nginx
      state: reloaded

A rescue converts a failure into a success

A host that enters a rescue and completes it is reported rescued=1 and failed=0, and ansible-playbook exits 0.

Verified on 2.21.3: the recap for a rescued host reads ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=1 ignored=0.

That is correct behaviour and it is a real hazard. A rescue whose only job is to flush handlers has quietly removed the failure signal that a scheduler, a CI job or an on-call human would have used. If you rescue, the rescue must record the problem somewhere that is read — a fail after the flush, a marker file, an alert, an entry in the run report.

The failure-handling part covers this in full. Here it is enough to notice that “make the handler run” and “keep the failure visible” are two separate obligations, and a rescue that meets only the first is a worse outcome than the failure it replaced.

Finding the hosts afterwards

You now have a fleet where some unknown subset of hosts has new configuration on disk and old configuration in memory. The recap does not tell you which. Two approaches, and the second is the one that scales.

Compare loaded configuration against on-disk configuration. Most services can be asked what they actually loaded, and the answer is authoritative in a way the file is not.

Read-only / Safeask the process, not the filesystem— Read-only across the fleet. changed_when: false keeps the probe out of the change signal.

- name: Find hosts whose running config differs from the file on disk
hosts: webservers
gather_facts: false
tasks:
  - name: Dump the configuration nginx actually loaded
    ansible.builtin.command: nginx -T
    changed_when: false
    register: loaded

  - name: Report hosts missing the expected directive
    ansible.builtin.debug:
      msg: '{{ inventory_hostname }} has not loaded the new listener'
    when: "'listen 8443' not in loaded.stdout"

Compare service start time against file modification time. More general, because it needs no per-service knowledge: if the configuration file is newer than the process that reads it, the process has not reloaded since the file changed.

Read-only / Safethe service-agnostic check— Read-only. Works for any unit whose configuration lives in a known path. The shell task is a read-only probe, marked changed_when: false so it cannot contaminate the change signal.

- name: Find services older than their configuration
hosts: webservers
gather_facts: false
tasks:
  - name: Stat the configuration file
    ansible.builtin.stat:
      path: /etc/nginx/conf.d/site.conf
    register: cfg

  - name: Read when the unit last entered the active state
    ansible.builtin.shell: >-
      date -d "$(systemctl show nginx --property=ActiveEnterTimestamp --value)" +%s
    changed_when: false
    register: started

  - name: Flag hosts whose configuration is newer than the running service
    ansible.builtin.debug:
      msg: '{{ inventory_hostname }}: config written after the last service start'
    when:
      - cfg.stat.exists
      - (cfg.stat.mtime | int) > (started.stdout | int)

Two caveats on that check. A reload does not always update ActiveEnterTimestamp — for a unit that reloads in place, the active state never changed — so this finds missed restarts reliably and missed reloads only for services that restart to pick up configuration. And a host whose file was written by something other than Ansible will also be flagged, which is arguably a feature.

Treat it as a triage sweep that narrows a fleet to a candidate list, then confirm each candidate with the service’s own answer.

Under the hood

A host that fails is removed from the play’s active host list. The flush at the end of the section iterates the handler list and runs each notified handler against the hosts still in play — the failed host is simply not among them, and its notification set is discarded with the rest of its play state.

force_handlers changes the set the flush iterates over: failed hosts are included for the purposes of handler execution while remaining failed for the purposes of the recap and the exit code. That is why the verified run above shows both RUNNING HANDLER and failed=1.

Unreachable hosts are handled by a different path and are not restored by force_handlers. This is consistent rather than arbitrary: an unreachable host has no results, so there is nothing to have been notified.

The interaction with serial is the one to think about on a fleet, and it is worth executing rather than assuming. On 2.21.3, a four-host play with serial: 2 where only the second batch fails produces two separate PLAY sections: batch one runs its RUNNING HANDLER [Reload] for both its hosts and finishes clean, then batch two changes its files, fails, and never reaches a flush.

PLAY [batch] ****************
TASK [Deploy config] ****    changed: [h1]  changed: [h2]
RUNNING HANDLER [Reload] *   ok: [h1]  ok: [h2]
PLAY [batch] ****************
TASK [Deploy config] ****    changed: [h3]  changed: [h4]
TASK [Fail ...] *********    fatal: [h3]  fatal: [h4]

Each batch is a complete play execution with its own flush points, so a failed batch takes only its own queued handlers with it. That means a failed rollout leaves a boundary: hosts before the failure point are consistent, hosts in the failing batch are the mixed-state ones, and hosts after it were never touched. Knowing which batch failed is most of the work of finding the affected hosts.

Knowledge check

Why this matters

Every other failure in a play announces itself. This one hides inside a failure about something else.

The recap says a package install failed. What it does not say is that the same host is now serving traffic with a TLS configuration that was replaced on disk twenty minutes ago and never loaded — so the weak cipher you removed is still being offered, the reviewed change is in Git, the file is correct, and the listener disagrees with all of it.

force_handlers is the blunt fix, a rescue-scoped flush is the precise one, and neither is a substitute for being able to answer the question afterwards: which hosts are running something other than what is on their disk? Build that check before you need it. You will need it during an incident, which is the worst time to be writing it.