Lab: The handler that never fired

Objective

By the end of this lab you will have produced, on a real host, the state that this failure mode leaves behind: a new configuration file on disk, an old configuration in the running process, a red first run and a green second run. You will then have fixed it three different ways and be able to say which one belongs on which kind of handler.

Architecture

Three managed nodes so the failure can be induced on one of them and the divergence between hosts is visible.

controller
    │
    ├── node1   config new, service restarted    (converged)
    ├── node2   config new, service NOT restarted (diverged)  <- the defect
    └── node3   config new, service restarted    (converged)

Requirements

A controller with ansible-core 2.21.x.
Three managed nodes with systemd as PID 1. This lab is entirely about the difference between a file on disk and a running process. A container without an init system cannot show you that difference, so B-nested — real VMs — is required, not preferred.
SSH key access and become on each node.
This lab does not reconfigure SSH, the firewall or any network interface, so there is no out-of-band access requirement. It does stop and start a service; Cleanup returns it to a defined state.

Scenario

Your deployment play writes an application configuration and notifies a restart handler. Last Tuesday it ran against forty hosts. Thirty-eight restarted. Two did not, because a task later in the play failed on those two hosts — a transient DNS lookup during a health registration step.

The operator saw a red run, re-ran the play, and got a clean green result: changed=0, no failures. The two hosts served the old configuration for nine days.

Tasks

Task 1: Capture the starting state

WORKDIR="$HOME/ansible-handler-lab"
mkdir -p "$WORKDIR"
cd "$WORKDIR"

inventory.yml:

appservers:
  hosts:
    node1:
      ansible_host: 192.0.2.11
    node2:
      ansible_host: 192.0.2.12
    node3:
      ansible_host: 192.0.2.13
  vars:
    ansible_user: operator

Set up a service whose running configuration is observable. This is the whole apparatus the lab needs: a unit that reads a config file at start and writes the value it read somewhere you can inspect.

# setup.yml
- name: Set up the observable service
  hosts: appservers
  become: true
  gather_facts: false
  tasks:
    - name: Create the configuration directory
      ansible.builtin.file:
        path: /etc/labapp
        state: directory
        mode: '0755'

    - name: Install the service script
      ansible.builtin.copy:
        content: |
          #!/bin/bash
          # Reads its config once at start, exactly like a real daemon.
          . /etc/labapp/app.conf
          echo "$APP_VERSION" > /run/labapp.running-version
          exec sleep infinity
        dest: /usr/local/bin/labapp
        mode: '0755'

    - name: Install the unit
      ansible.builtin.copy:
        content: |
          [Unit]
          Description=Lab application

          [Service]
          ExecStart=/usr/local/bin/labapp
          Restart=no

          [Install]
          WantedBy=multi-user.target
        dest: /etc/systemd/system/labapp.service
        mode: '0644'

    - name: Write the initial configuration
      ansible.builtin.copy:
        content: "APP_VERSION=1.0.0\n"
        dest: /etc/labapp/app.conf
        mode: '0644'

    - name: Start the service
      ansible.builtin.systemd_service:
        name: labapp
        state: started
        enabled: true
        daemon_reload: true

Service impact possiblecontroller— Installs a unit and starts a service on three nodes

$ ansible-playbook -i inventory.yml setup.yml

Confirm all three nodes agree:

Read-only / Safecontroller— Reads the version the running process loaded

$ ansible -i inventory.yml appservers -b -m command -a 'cat /run/labapp.running-version'

node1 | CHANGED | rc=0 >>
1.0.0
node2 | CHANGED | rc=0 >>
1.0.0
node3 | CHANGED | rc=0 >>
1.0.0

Illustrative output

Task 2: The deployment play, as written

# deploy.yml
- name: Deploy the application configuration
  hosts: appservers
  become: true
  gather_facts: false
  vars:
    app_version: '2.0.0'

  handlers:
    - name: Restart labapp
      ansible.builtin.systemd_service:
        name: labapp
        state: restarted

  tasks:
    - name: Write the application configuration
      ansible.builtin.copy:
        content: "APP_VERSION={{ app_version }}\n"
        dest: /etc/labapp/app.conf
        mode: '0644'
      notify: Restart labapp

    - name: Register with service discovery
      ansible.builtin.command: /usr/local/bin/register-service
      changed_when: false

    - name: Report the deployment
      ansible.builtin.debug:
        msg: "{{ inventory_hostname }} deployed {{ app_version }}"

/usr/local/bin/register-service does not exist yet — that is the induced failure, and it is going to fail on every host. Install it on node1 and node3 only, so the failure is confined to node2:

# partial-setup.yml
- name: Install the registration helper on two of three nodes
  hosts: node1,node3
  become: true
  gather_facts: false
  tasks:
    - name: Install register-service
      ansible.builtin.copy:
        content: |
          #!/bin/bash
          exit 0
        dest: /usr/local/bin/register-service
        mode: '0755'

ansible-playbook -i inventory.yml partial-setup.yml

Task 3: Reproduce the divergence

Service impact possiblecontroller— Writes config on three hosts; restarts on two. This is the reproduction.

$ ansible-playbook -i inventory.yml deploy.yml

TASK [Write the application configuration] *************************************
changed: [node1]
changed: [node2]
changed: [node3]

TASK [Register with service discovery] *****************************************
ok: [node1]
fatal: [node2]: FAILED! => {"msg": "[Errno 2] No such file or directory: b'/usr/local/bin/register-service'"}
ok: [node3]

RUNNING HANDLER [Restart labapp] ***********************************************
changed: [node1]
changed: [node3]

PLAY RECAP *********************************************************************
node1                      : ok=4    changed=2    unreachable=0    failed=0
node2                      : ok=1    changed=1    unreachable=0    failed=1
node3                      : ok=4    changed=2    unreachable=0    failed=0

Illustrative output

Note what the recap does and does not tell you. It says node2 failed. It does not say that node2’s configuration was written and its service was not restarted. That state — the actual damage — appears nowhere.

Confirm it from the host. Two slurp reads and a comparison — a command would report changed for a pure read:

ansible -i inventory.yml appservers -b -m slurp -a 'src=/etc/labapp/app.conf'
ansible -i inventory.yml appservers -b -m slurp -a 'src=/run/labapp.running-version'

Decoded, the three hosts report:

host    on disk   in the running process
node1   2.0.0     2.0.0
node2   2.0.0     1.0.0     <- diverged
node3   2.0.0     2.0.0

Task 6 turns this comparison into a playbook you can run on a whole fleet.

node2: disk=2.0.0, running=1.0.0. That is the failure this lab is about.

Task 4: The re-run that makes it invisible

Fix the underlying error the way the operator did — install the missing helper — and re-run:

ansible-playbook -i inventory.yml partial-setup.yml \
  --extra-vars 'target=all' 2>/dev/null || true

ansible -i inventory.yml node2 -b -m copy \
  -a "content='#!/bin/bash\nexit 0\n' dest=/usr/local/bin/register-service mode=0755"

ansible-playbook -i inventory.yml deploy.yml | tail -6

Task 5: Three fixes, and when each is right

Reset the environment before each experiment:

ansible -i inventory.yml appservers -b -m copy \
  -a "content='APP_VERSION=1.0.0\n' dest=/etc/labapp/app.conf mode=0644"
ansible -i inventory.yml appservers -b -m systemd_service \
  -a 'name=labapp state=restarted'
ansible -i inventory.yml node2 -b -m file \
  -a 'path=/usr/local/bin/register-service state=absent'

Fix 1 — --force-handlers. Runs notified handlers even on hosts where a later task failed:

ansible-playbook -i inventory.yml deploy.yml --force-handlers

All three restart, including node2. node2 still reports failed, which is correct — the registration genuinely did not happen — but its service now matches its config.

Fix 2 — meta: flush_handlers immediately after the change. Runs the handler before the task that can fail:

  tasks:
    - name: Write the application configuration
      ansible.builtin.copy:
        content: "APP_VERSION={{ app_version }}\n"
        dest: /etc/labapp/app.conf
        mode: '0644'
      notify: Restart labapp

    - name: Apply pending restarts before anything else can fail
      ansible.builtin.meta: flush_handlers

    - name: Register with service discovery
      ansible.builtin.command: /usr/local/bin/register-service
      changed_when: false

Now the restart happens at a defined point rather than at the end of the play. node2 restarts, then fails registration. Disk and process agree; the registration is genuinely outstanding and the recap says so.

This is the better fix here, because it makes the ordering explicit in the playbook rather than depending on a command-line flag somebody has to remember.

Fix 3 — block/rescue that leaves the service consistent. For the case where the failure means the change should be reverted:

  tasks:
    - name: Deploy with rollback
      block:
        - name: Back up the current configuration
          ansible.builtin.copy:
            src: /etc/labapp/app.conf
            dest: /etc/labapp/app.conf.prev
            remote_src: true
            mode: preserve

        - name: Write the application configuration
          ansible.builtin.copy:
            content: "APP_VERSION={{ app_version }}\n"
            dest: /etc/labapp/app.conf
            mode: '0644'
          notify: Restart labapp

        - name: Apply pending restarts
          ansible.builtin.meta: flush_handlers

        - name: Register with service discovery
          ansible.builtin.command: /usr/local/bin/register-service
          changed_when: false

      rescue:
        - name: Restore the previous configuration
          ansible.builtin.copy:
            src: /etc/labapp/app.conf.prev
            dest: /etc/labapp/app.conf
            remote_src: true
            mode: preserve

        - name: Restart onto the restored configuration
          ansible.builtin.systemd_service:
            name: labapp
            state: restarted

        - name: Fail loudly, having left the host consistent
          ansible.builtin.fail:
            msg: >-
              Deployment of {{ app_version }} failed on
              {{ inventory_hostname }}; rolled back to the previous
              configuration and restarted.

      always:
        - name: Report what is actually running
          ansible.builtin.command: cat /run/labapp.running-version
          register: running
          changed_when: false

        - name: State the end state for the record
          ansible.builtin.debug:
            msg: "{{ inventory_hostname }} running {{ running.stdout }}"

Task 6: Build the detector

None of the three fixes helps with the hosts that already diverged. Write the check that finds them:

# detect-divergence.yml
- name: Detect config-on-disk versus config-in-process divergence
  hosts: appservers
  become: true
  gather_facts: false
  tasks:
    - name: Read the configured version
      ansible.builtin.slurp:
        src: /etc/labapp/app.conf
      register: conf

    - name: Read the running version
      ansible.builtin.slurp:
        src: /run/labapp.running-version
      register: running

    - name: Compare
      ansible.builtin.set_fact:
        configured: "{{ (conf.content | b64decode).split('=')[1] | trim }}"
        in_process: "{{ (running.content | b64decode) | trim }}"

    - name: Report divergence
      ansible.builtin.debug:
        msg: >-
          DIVERGED {{ inventory_hostname }}:
          disk={{ configured }} process={{ in_process }}
      when: configured != in_process

    - name: Fail the check when any host has diverged
      ansible.builtin.assert:
        that: configured == in_process
        fail_msg: >-
          {{ inventory_hostname }} is running {{ in_process }} but
          configured for {{ configured }} - restart required.
        success_msg: "{{ inventory_hostname }} consistent at {{ configured }}"

Run it after each experiment. This is the check that would have found the two hosts on day one instead of day nine.

Validation

After Task 3, node2 reports disk=2.0.0 and running=1.0.0, while node1 and node3 report both as 2.0.0.
The re-run in Task 4 reports changed=0 failed=0 on all three hosts, and node2 still reports running=1.0.0.
With --force-handlers, node2 restarts and reports running=2.0.0 while still reporting failed=1.
With meta: flush_handlers after the config task, node2 restarts before the failing task and reaches the same end state without the flag.
With the block/rescue version, node2 ends with running=1.0.0 and disk=1.0.0 — rolled back and consistent — and the play reports a failure with a message naming the host.
detect-divergence.yml fails on node2 after Task 3 and passes after any of the three fixes.

Expected Outcome

ansible-handler-lab/
├── deploy.yml            <- three variants, or one with the fix applied
├── detect-divergence.yml
├── inventory.yml
├── partial-setup.yml
├── setup.yml
└── strategies.md

On the nodes: a labapp service whose running version matches its configured version on all three hosts. You have observed the divergence, the green re-run that hides it, and three different corrections, and you can name the situation in which each is the wrong choice.

Troubleshooting

RUNNING HANDLER does not appear at all. No task reported changed, so nothing was notified. On a second run this is expected and is the whole point of Task 4.

The handler fires on the failed host without --force-handlers. The task that failed came after an implicit or explicit flush_handlers, or the play uses serial and the failure was in a later batch. Handlers flush at the end of each batch, so under serial a failure in batch 3 does not prevent batch 1’s handlers from having run.

/run/labapp.running-version does not exist. /run is a tmpfs and is cleared on reboot; the file is written by the service at start. If the service is not running, there is nothing to read. systemctl status labapp on the node.

The copy task reports changed on every run. The content string ends without a newline, or with a different one than the file has. Use --check --diff to see the one-character difference.

node1 or node3 also fails the registration task. The helper was not installed on that host. partial-setup.yml targets node1,node3; a typo in either name is ignored with a Could not match supplied host pattern warning, so check for that warning rather than assuming the play ran everywhere you meant.

Cleanup

This lab installed a systemd unit, a script and a config file on three nodes and started a service. All of it must be removed, and the service must not be left running.

# cleanup.yml
- name: Remove the lab application
  hosts: appservers
  become: true
  gather_facts: false
  tasks:
    - name: Stop and disable the service
      ansible.builtin.systemd_service:
        name: labapp
        state: stopped
        enabled: false
      failed_when: false

    - name: Remove the unit
      ansible.builtin.file:
        path: /etc/systemd/system/labapp.service
        state: absent

    - name: Reload systemd so the removed unit is forgotten
      ansible.builtin.systemd_service:
        daemon_reload: true

    - name: Remove the helper scripts
      ansible.builtin.file:
        path: "{{ item }}"
        state: absent
      loop:
        - /usr/local/bin/labapp
        - /usr/local/bin/register-service

    - name: Remove the configuration directory
      ansible.builtin.file:
        path: /etc/labapp
        state: absent

    - name: Remove the runtime marker
      ansible.builtin.file:
        path: /run/labapp.running-version
        state: absent

Service impact possiblecontroller— Stops the lab service and removes every file the lab created

$ ansible-playbook -i inventory.yml cleanup.yml

Verify the removal actually happened, rather than assuming it:

ansible -i inventory.yml appservers -b -m shell \
  -a 'systemctl list-unit-files labapp.service 2>/dev/null | wc -l; ls /etc/labapp 2>&1' \
  || true

Expected: no labapp.service unit file, and No such file or directory for /etc/labapp. The daemon_reload step matters — without it systemd keeps the removed unit in memory and systemctl start labapp still works, which is a confusing thing to leave behind.

Then remove the working directory:

rm -rf deletes the whole lab directory, including anything you added

The next block removes ~/ansible-handler-lab and everything under it, recursively and without confirmation. If you saved notes, an inventory you want to keep, or a playbook you extended, copy it out first — the step above does that for the lab’s own deliverables and knows nothing about yours.

The path is written in full and anchored to $HOME deliberately. A relative rm -rf run from the wrong directory is how people delete work they meant to keep, and the shell gives no second chance.

Confirm what you are about to delete before you delete it:

ls -la "$HOME/ansible-handler-lab"

mkdir -p "$HOME/ansible-lab-deliverables"
cp -a "$HOME/ansible-handler-lab/detect-divergence.yml" \
      "$HOME/ansible-handler-lab/strategies.md" \
      "$HOME/ansible-lab-deliverables/"

rm -rf "$HOME/ansible-handler-lab"

Under the hood

notify does not run anything. It appends the handler’s name to a per-host queue, and only when the notifying task returned changed: true. The queue is drained at three moments: the end of the play, an explicit meta: flush_handlers, and the end of each serial batch.

Draining is per host. A host that has failed is removed from the play’s active host list, and a host not in that list is not offered the handler queue — which is the entire mechanism behind the failure in this lab. --force-handlers (and force_handlers: true on the play) changes the drain step to include hosts that failed after being notified.

Two consequences follow that are worth holding on to.

First, changed: true is the trigger, so any task whose change detection is wrong breaks handlers in one of two directions: a task that always reports changed restarts the service on every run, and a task that never reports changed never restarts it at all. The idempotency work in the previous lab is therefore a prerequisite for handlers being trustworthy, not a separate concern.

Second, the queue is deduplicated by handler name. Ten tasks notifying Restart nginx produce one restart, which is the behaviour everyone wants and relies on. It is also why two roles defining a handler with the same name collapse into one — same key, same queue.

What You Learned

You produced the divergence and read it from the host, not from the recap. disk=2.0.0 running=1.0.0 is a state Ansible’s output never mentions.
A clean second run does not mean a converged host. The config task was unchanged, so the handler was never notified, so the process kept serving the old configuration under a green recap.
--force-handlers is right when the handler makes the host match the change already on disk, and wrong when the failure was the check that was supposed to gate the handler.
meta: flush_handlers puts the decision in the playbook rather than in a flag somebody has to remember to pass.
A rescue restart is a plain task, never a notify, because a handler notified in recovery queues for the end of the play — the exact behaviour being recovered from.
The detector is the durable artefact. Fixes prevent new divergence; only a check finds the hosts that already diverged.

Objectives

Prerequisites