Handlers flush per batch · Ansible

In an unbatched play, handlers run once, at the end, after every host has finished every task. That is the behaviour the handlers part established, and on a fleet it is a problem: a template task that changes configuration on 400 hosts queues 400 notifications, and the flush at the end of the play restarts 400 services at the same moment.

The handler is doing exactly what it was told. The play just never gave it a reason to stagger.

serial is that reason, and the mechanism is worth stating precisely because everything in the rolling-deployment part is built on it.

The observed behaviour

Read-only / Safesix hosts, serial: 2, executed on ansible-core 2.21.3— A debug task with changed_when: true notifies a debug handler. No host state is modified. Three RUNNING HANDLER banners for one handler in one play.

$ ansible-playbook -i inv6.ini handlers.yml

PLAY [Handlers per batch] ******************************************************

TASK [Deploy config] ***********************************************************
changed: [h01]
changed: [h02]

RUNNING HANDLER [Restart service] **********************************************
ok: [h01] => { "msg": "RESTARTED h01" }
ok: [h02] => { "msg": "RESTARTED h02" }

PLAY [Handlers per batch] ******************************************************

TASK [Deploy config] ***********************************************************
changed: [h03]
changed: [h04]

RUNNING HANDLER [Restart service] **********************************************
ok: [h03] => { "msg": "RESTARTED h03" }
ok: [h04] => { "msg": "RESTARTED h04" }

PLAY [Handlers per batch] ******************************************************

TASK [Deploy config] ***********************************************************
changed: [h05]
changed: [h06]

RUNNING HANDLER [Restart service] **********************************************
ok: [h05] => { "msg": "RESTARTED h05" }
ok: [h06] => { "msg": "RESTARTED h06" }

Three batches, three flushes, two hosts restarted per flush. Never more than two instances of the service down at once.

This follows directly from the previous lesson: a batched play is the same play executed once per batch, and the implicit handler flush at the end of a play therefore happens at the end of each batch. There is no separate rule for handlers under serial. There is one rule — the play restarts — and this is a consequence of it.

Why this is the whole mechanism

Take the per-batch flush away and a rolling restart is not expressible.

Consider what a rolling deploy needs: for each batch, change the configuration, restart the service, confirm it came back, then move on. The “restart the service” step must happen inside the batch, between the change and the health check. If handlers only flushed at the end of the play, the sequence would be:

Batch 1 writes config. Notifies.
Batch 1 health-checks — against a service still running the old config.
Batch 2 writes config. Notifies.
Batch 2 health-checks — same problem.
…
End of play: every service restarts simultaneously.

Which is the fleet-wide simultaneous restart you used serial to avoid, with the health checks rendered meaningless because they all ran before any restart happened.

Per-batch flushing is what makes the health check land after the restart of the hosts it is checking, and before the next batch is touched.

The handler that fires once per batch

The same property that makes rolling restarts work makes a certain class of handler wrong under serial, and it is wrong quietly.

A handler is per-host. Under serial, it runs once per batch for the hosts in that batch. If what the handler touches is not per-host — if it is a shared dependency, a central cache, a load balancer, an external API — then it fires once per batch against the same shared thing.

Service impact possiblea handler that is wrong under serial— Delegated to a single shared host. Under serial: [1, 5, 25%] over 200 hosts this reloads the proxy on every batch, and the reload count scales with the number of batches rather than with the number of changes.

- name: Deploy backend configuration
hosts: appservers
become: true
serial:
  - 1
  - 5
  - 25%
tasks:
  - name: Write the backend definition
    ansible.builtin.template:
      src: backend.conf.j2
      dest: /etc/app/backend.conf
      mode: '0644'
    notify: Reload the shared proxy

handlers:
  # WRONG under serial: fires once per batch, against the same proxy.
  - name: Reload the shared proxy
    ansible.builtin.systemd_service:
      name: haproxy
      state: reloaded
    delegate_to: proxy01.example.com

Without serial this handler is fine — it fires once, at the end of the play, however many hosts notified it. Add serial and it fires once per batch. A 200-host fleet with serial: [1, 5, 25%] runs six batches, so the shared proxy is reloaded six times during a single deploy.

Six reloads of HAProxy is survivable. Six restarts of a shared message broker, six cache flushes, or six calls to an API with a rate limit are not, and the failure surfaces as something unrelated: the fourth batch fails its health check because the proxy was mid-reload when it was probed.

The fix is to move the shared work out of the batched play. Anything that touches a single shared resource once belongs in a separate play with its own hosts: — usually hosts: localhost or the resource itself — that runs after the rolling play completes.

Service impact possibleshared work in its own play— The rolling play touches only per-host state. The proxy reload happens once, after the rollout, in a play that is not batched and targets the proxy directly.

- name: Roll the backend configuration across the fleet
hosts: appservers
become: true
serial:
  - 1
  - 5
  - 25%
tasks:
  - name: Write the backend definition
    ansible.builtin.template:
      src: backend.conf.j2
      dest: /etc/app/backend.conf
      mode: '0644'
    notify: Restart the local app

handlers:
  - name: Restart the local app
    ansible.builtin.systemd_service:
      name: app
      state: restarted

- name: Reload the shared proxy once, after the rollout
hosts: proxy01.example.com
become: true
tasks:
  - name: Reload haproxy
    ansible.builtin.systemd_service:
      name: haproxy
      state: reloaded

Where the failure boundary falls

Per-batch flushing has a second consequence, and it is the useful one during an incident.

Each batch has its own flush points. A batch that fails takes only its own queued handlers down with it. Batches that already completed have already flushed.

That partitions the fleet into three states after a failed rollout:

Group	State
Hosts in completed batches	configuration written and applied — consistent
Hosts in the failing batch	configuration possibly written, handler discarded — mixed
Hosts in later batches	never touched — consistent, on the old version

The middle row is the only one you have to investigate, and it is bounded by the batch size. On a 200-host fleet with serial: 25%, a failed rollout leaves at most 50 hosts in an uncertain state, not 200.

This is a real argument for smaller batches that has nothing to do with capacity: the batch size is also the size of the search you will have to run afterwards.

Under the hood

There is no handler-specific code path for serial. The play iterator splits the host list into batches and runs the play once per batch; the handler flush is an ordinary end-of-play step that therefore executes once per batch.

The notification state lives in the play context for the current execution. When the iterator moves to the next batch, that context is replaced — which is why a notification raised in batch one cannot be satisfied by a flush in batch two, and why a handler that was notified by a host that subsequently failed is simply discarded along with the rest of that host’s play state.

One consequence that is easy to miss: handler deduplication is also per batch. A handler notified by all six hosts in an unbatched play runs once per host at a single flush point. Under serial: 2, it runs at three flush points, against two hosts each. The total number of handler executions across the fleet is the same — one per notifying host — but the number of flush events is the number of batches, and for a handler that is delegated to a single shared target, flush events are what matter, because every flush event is one action against that target.

That distinction is the whole content of the shared-dependency trap above: per-host handlers scale with hosts, delegated handlers scale with batches.

What happens if I use meta: flush_handlers inside a batched play?

It adds a flush point inside each batch, and it runs once per batch like everything else in the play body.

This is the correct tool when the ordering inside the batch matters — which in a rolling deploy it always does. The canonical sequence is: write the configuration, flush so the restart happens now, then health-check the restarted service, all before the batch ends.

Service impact possibleforcing the restart to land before the health check— Without the explicit flush, the handler would run after the health check, and the health check would be probing the old process.

    - name: Write the application configuration
    ansible.builtin.template:
      src: app.conf.j2
      dest: /etc/app/app.conf
      mode: '0644'
    notify: Restart app

  - name: Apply the restart now, not at the end of the batch
    ansible.builtin.meta: flush_handlers

  - name: Confirm the service answers after the restart
    ansible.builtin.uri:
      url: 'http://{{ inventory_hostname }}:8080/healthz'
      status_code: 200
    register: health
    retries: 12
    delay: 5
    until: health.status == 200

The trap to avoid is assuming the implicit end-of-batch flush is early enough. It is early enough to keep the next batch safe, and not early enough to let this batch verify what it just did.

Knowledge check

Why this matters

A rolling deployment is not a feature of Ansible. It is a shape you assemble out of serial, handlers and a health check, and per-batch handler flushing is the joint that holds it together.

Understanding it as a consequence of “the play restarts per batch” rather than as a special rule about handlers is what lets you predict the rest: why run_once runs once per batch, why a set_fact from batch one is invisible in batch two, and why a delegated handler fires as many times as you have batches.

The shared-dependency trap is worth carrying forward specifically. Somebody will one day change serial: 5 to serial: 2 to make a rollout gentler, and in doing so will double the number of times a delegated handler hammers something that was only ever meant to be touched once. Nothing in the diff will mention the handler.