Runbook: Troubleshoot a failed handler

A handler that does not fire produces the quietest bad state in configuration management: the file on disk is correct, the play reported success for the change, and the running process is using the old configuration. Nothing is broken. Nothing alerts. The host stays that way until something restarts the service - which might be next Tuesday’s patch window, which is when the change actually takes effect, weeks after the change record says it did.

This runbook finds those hosts and closes them out.

When to use this runbook

• A play failed after a task that notified a handler.
• A handler task itself failed.
• A configuration change was applied but the service is behaving as though it was not.
• After any aborted rolling change - the aborted batch is the most likely place for this.

Blast radius

Diagnosis is read-only. The remedy is service-affecting: reloading or restarting a service on every affected host. Treat the remedy as a rolling change with its own batching and verification, not as a clean-up.

The three failure modes

They look identical in the recap and have different causes.

Step 1: Classify

LOG=run-2026-08-11.log

# Did the handler section run at all?
grep -n 'RUNNING HANDLER' "$LOG"

# Did the handler itself fail?
grep -A5 'RUNNING HANDLER' "$LOG" | grep -E 'fatal|FAILED'

# Which hosts reported changed for the notifying task?
grep -A200 'TASK \[Render the site configuration\]' "$LOG" \
| grep -E '^changed:' | head -50

Step 2: Extract the affected host set

This is the artefact the rest of the runbook operates on. Build it from the log, not from memory.

LOG=run-2026-08-11.log

# Hosts where the notifying task reported changed
awk '/TASK \[Render the site configuration\]/,/^TASK |^RUNNING HANDLER/' "$LOG" \
| grep '^changed:' | sed 's/.*\[\(.*\)\].*/\1/' | sort -u > notified.txt

# Hosts where the handler actually ran
awk '/RUNNING HANDLER \[Reload nginx\]/,/^PLAY RECAP/' "$LOG" \
| grep -E '^(changed|ok):' | sed 's/.*\[\(.*\)\].*/\1/' | sort -u > fired.txt

comm -23 notified.txt fired.txt | tee stale-config-hosts.txt
wc -l stale-config-hosts.txt

Every host in stale-config-hosts.txt has new configuration on disk and an old configuration in the running process. That is the working set.

Step 3: Confirm the divergence on the host

Do not take the log’s word for it. Ask the host.

HOST=web01.example.com

# When was the file last written?
ansible "$HOST" -b -m stat -a 'path=/etc/nginx/conf.d/site.conf' -o \
| python3 -c 'import sys,json; s=sys.stdin.read(); print(json.loads(s[s.index("{"):])["stat"]["mtime"])'

# When did the process last load its configuration?
ansible "$HOST" -b -m command \
-a 'systemctl show nginx -p ActiveEnterTimestamp -p ExecMainStartTimestamp' -o

# What does the running service think its configuration is?
ansible "$HOST" -b -m command -a 'nginx -T' -o | head -40

A file mtime later than ExecMainStartTimestamp is the divergence, stated in numbers. For services that can report their live configuration

• nginx -T, sshd -T, postconf -n - that is the stronger check, because it reads what the process actually has rather than inferring from timestamps.

Step 4: Mode C - the handler was never notified

This is the most insidious of the three, because the play succeeded completely.

grep -rn -A6 -E 'ansible\.builtin\.(command|shell|raw|script)' \
roles/*/tasks/*.yml | grep -B4 'notify:'

The failure: a command or shell task that writes a config file, with changed_when: false set to stop it reporting a change on every run. It now never reports a change, so it never notifies, so the service is never reloaded. The play is beautifully idempotent and completely ineffective.

The fix is to make the task report the truth:

- name: Generate the configuration from the vendor tool
ansible.builtin.command:
  cmd: /usr/local/bin/gen-config --out /etc/app/app.conf
register: gen
changed_when: "'wrote' in gen.stdout"
notify: Reload app

Better still, use a module. template and copy report changed correctly because they compare content, and that is the whole reason to prefer them.

The other Mode C cause: notify naming a handler that does not exist by that exact string. Handler names are matched literally, and a rename on one side of the pair silently stops the notification.

grep -rhn 'notify:' roles/*/tasks/*.yml | sed 's/.*notify: *//' | sort -u
grep -rhn '^- name:' roles/*/handlers/*.yml | sed 's/^- name: *//' | sort -u

Compare those two lists. In recent ansible-core a notify that matches nothing is an error rather than a silent no-op, but a handler renamed to something that another handler already answers via listen can still resolve to the wrong thing.

Step 5: Apply the remedy as a rolling change

The remedy restarts or reloads a service on every host in stale-config-hosts.txt. That is a service-affecting change and it gets the same treatment as any other.

LIMIT=$(paste -sd, stale-config-hosts.txt)
ansible-playbook -i inventories/production reload.yml --limit "$LIMIT" --list-hosts

# Validate the config on disk BEFORE asking the service to load it
ansible "$LIMIT" -b -m command -a 'nginx -t' -o

- name: Bring the running service into line with its configuration
hosts: "{{ target_hosts }}"
become: true
serial: "25%"
max_fail_percentage: 0
tasks:
  - name: Configuration on disk is valid
    ansible.builtin.command: nginx -t
    changed_when: false

  - name: Drain from the load balancer
    ansible.builtin.uri:
      url: "https://lb.example.com/api/pool/web/{{ inventory_hostname }}/drain"
      method: POST
      status_code: [200, 204]
    delegate_to: localhost

  - name: Reload the service
    ansible.builtin.systemd_service:
      name: nginx
      state: reloaded

  - name: Service answers a real request
    ansible.builtin.uri:
      url: "http://{{ ansible_host }}:8080/healthz"
      status_code: 200
      return_content: true
    register: health
    retries: 6
    delay: 5
    until: health.status == 200 and 'ok' in health.content

  - name: Return to the load balancer pool
    ansible.builtin.uri:
      url: "https://lb.example.com/api/pool/web/{{ inventory_hostname }}/enable"
      method: POST
      status_code: [200, 204]
    delegate_to: localhost

Step 6: Fix the cause

The remedy addresses the symptom on the affected hosts. The cause is whichever of the three modes applied.

- name: Configuration and restart, before anything that could fail
ansible.builtin.template:
  src: site.conf.j2
  dest: /etc/nginx/conf.d/site.conf
  validate: 'nginx -t -c %s'
notify: Reload nginx

- name: Apply pending handlers now, not at the end of the play
ansible.builtin.meta: flush_handlers

- name: Tasks that might fail, after the service is already correct
ansible.builtin.include_tasks: post_checks.yml

meta: flush_handlers runs pending handlers immediately. Putting it straight after the configuration block means a failure in a later task cannot leave the service on stale configuration - which converts Mode B from a possibility into an impossibility for that play.

Step 7: Verify

# The process has loaded what is on disk
ansible "$LIMIT" -b -m command \
-a 'systemctl show nginx -p ExecMainStartTimestamp' -o

# The service reports the live configuration you expect
ansible "$LIMIT" -b -m command -a 'nginx -T' -o | grep -c 'listen 8080'

# A subsequent run of the real play is clean
ansible-playbook -i inventories/production site.yml --limit "$LIMIT" --diff \
| grep -E 'changed=[1-9]'

That last grep must produce nothing.

Then prove the mechanism is repaired, which is the check that distinguishes a fix from a workaround:

# Make a trivial, safe change on one host and watch for the handler
ansible-playbook -i inventories/production site.yml \
--limit web01.example.com -e nginx_frontend_worker_processes=2 --diff \
| grep -A3 'RUNNING HANDLER'

Step 8: Record the window

Write down the host set and the time range during which they were running stale configuration. Two reasons, and the second is the important one:

• Anyone investigating behaviour on those hosts in that window needs to know the running configuration was not the declared one.
• If the change that never took effect was a security change - a firewall rule, a TLS setting, an authentication change - then those hosts were unprotected for a measurable period. That is a security finding with a start time, not an operations note.

Common patterns

Escalation

Escalate when:

• The remedy requires a restart rather than a reload. That drops connections and it is the service owner’s call.
• Affected hosts run stateful services where a restart has data implications.
• The stale window covers a security change.
• The handler is failing for a reason that will recur on every host.