Restart, reload and handler timing · Ansible

Writing the file changes nothing. The running process is still holding whatever it read at startup, and it will keep holding it until something makes it read again.

Two things can make it read again, and they are not interchangeable.

Reload versus restart

Reload asks the running process to re-read its configuration. The process ID does not change, open connections survive, in-memory state survives, and there is no gap in service.

Restart stops the process and starts a new one. Connections are dropped, in-memory state is lost, and there is a window — usually short, occasionally not — where the service is not answering.

In a rolling deployment the host is drained, so the disruption of a restart is contained. That is exactly why the pattern drains first. But “contained” is not “free”: a restart costs the startup time of the service on every host in the fleet, and where a reload would have done, that time is pure waste.

What a reload cannot do

The distinction is not stylistic — a reload genuinely cannot apply certain changes, and applying it anyway leaves the process running the old behaviour while every task reports success.

Change	Reload enough?
A value in the application configuration file	usually yes
A new virtual host or backend definition	usually yes
TLS certificate replacement	usually yes, if the server supports it
A new binary or package version	no — restart
A change to the unit file itself	no — `daemon_reload` then restart
An environment variable the process reads at startup	no — restart
A listening port or socket change	usually no — restart
A library the process links against	no — restart

The row that matters in a deployment is the first bold one. You are deploying new code. A reload does not replace a running binary, so a rolling deployment of a new application version is a restart, always. Reload is for the configuration-only changes that a convergence run makes.

`reloaded` starts a stopped service

Worth knowing because it changes the failure mode. The ansible.builtin.systemd_service documentation for state says:

started/stopped are idempotent actions that will not run commands unless necessary. restarted will always bounce the unit. reloaded will always reload and if the service is not running at the moment of the reload, it is started.

So state: reloaded on a service that has crashed will start it. That is usually convenient and occasionally hides a problem: a service that has been down since the last deploy comes back during this one, and nothing in the output distinguishes “reloaded a healthy service” from “quietly resurrected a service that had been dead for a week”.

Note also that reloaded and restarted always report changed — they are not idempotent actions in the way started is. That is correct behaviour for a handler, which only fires when something actually changed, and it is a reason not to put them in an ordinary task.

Why handlers, not a restart task

The temptation in a rolling play is to write the restart as a plain task, because the ordering is then obvious on the page.

Service impact possiblethe tempting version— Restarts the service on every run, whether or not anything changed. On a convergence run over 200 hosts this is 200 unnecessary restarts.

    # Avoid this in a rolling play.
  - name: Restart the application
    ansible.builtin.systemd_service:
      name: app
      state: restarted

It runs unconditionally. A rolling play executed against a fleet that is already at the target version — a re-run, a convergence pass, a CI verification — restarts every service in the fleet for no reason. state: restarted “will always bounce the unit”, so there is no idempotence to fall back on.

The handler form fires only when something changed:

Service impact possiblechange-gated, and correctly ordered— The restart happens only if the unarchive actually changed something. The explicit flush moves it ahead of the health check.

    - name: Install the release
    ansible.builtin.unarchive:
      src: 'app-{{ release_version }}.tar.gz'
      dest: /opt/app
      owner: app
      group: app
    notify: Restart app

  - name: Apply the restart now
    ansible.builtin.meta: flush_handlers

  - name: Assert the service is serving the new version
    ansible.builtin.uri:
      url: 'http://{{ ansible_host | default(inventory_hostname) }}:8080/healthz'
      return_content: true
      status_code: 200
    register: health
    retries: 12
    delay: 5
    until:
      - health.status | default(0) == 200
      - health.json.version | default('') == release_version
    changed_when: false

handlers:
  - name: Restart app
    ansible.builtin.systemd_service:
      name: app
      state: restarted

You get change-gating and correct ordering from the same construct, which is the argument for handlers in a nutshell.

The flush has to be explicit

Part XXXI established that handlers flush at the end of every batch. That is what makes a rolling restart possible at all — and inside the batch, it is too late.

Without the explicit meta: flush_handlers, the order is:

Install the release. Handler notified, not run.
Health check — against a process still running the old code.
End of batch. Handler runs. Service restarts.
Next batch begins.

Step 2 is the problem. The health check passes, because the old version is running perfectly well. It asserts nothing about the thing you just deployed, and the rollout proceeds to the next batch having verified the previous release.

Worse, if you took the advice from the health-check lesson and assert on the version string, the check now fails on every host — the process reports the old version because it has not restarted yet. Either way the gate is wrong.

Restart, then wait for the thing to actually be back

systemd_service returns when systemd reports the unit started. That is not the same as the application being ready to serve — systemd considers a Type=simple unit started the moment it has forked the process, which for a JVM or a large Python application may be tens of seconds before it can answer a request.

This is why the health check needs retries and delay rather than running once. It is also why a wait_for on the port is a reasonable step between the restart and the correctness check for slow-starting services:

Read-only / Safegiving a slow starter time to bind before asserting on it— Read-only. Cheap liveness gate first, then the expensive correctness assertion. Failing fast on a service that never binds gives a clearer error than a version assertion timing out.

    - name: Wait for the application to bind its port
    ansible.builtin.wait_for:
      host: '{{ ansible_host | default(inventory_hostname) }}'
      port: 8080
      state: started
      timeout: 60

  - name: Assert it is serving the new version
    ansible.builtin.uri:
      url: 'http://{{ ansible_host | default(inventory_hostname) }}:8080/healthz'
      return_content: true
    register: health
    retries: 12
    delay: 5
    until: health.json.version | default('') == release_version
    changed_when: false

The diagnostic value is the reason to do both. A failure at the wait_for says “the service never started”. A failure at the uri says “the service started and is wrong”. Those lead to different investigations, and a single combined check would report them identically.

Under the hood

A handler is a task with a name that other tasks can reference. The notification is a per-host record that this handler should run at the next flush point, and it is deduplicated: ten tasks notifying the same handler produce one execution per host, not ten.

Flush points are where the interesting behaviour lives. There are three:

After the pre_tasks section.
After the tasks section — the implicit end-of-play flush, which under serial is the end of each batch.
Wherever you write meta: flush_handlers.

At a flush point, Ansible iterates the handlers in the order they are defined in the play, not in the order they were notified, and runs each one that has a notification pending on at least one active host. That definition ordering is why two handlers whose relative order matters must be written in that order, and why a flush_handlers in the middle of a batch runs every pending handler rather than a subset.

Under serial, the notification record belongs to the current batch execution. A handler notified in batch one cannot be satisfied by a flush in batch two — verified in Part XXXI, where each batch produced its own RUNNING HANDLER banner. It also means a host that fails before reaching a flush point discards its notifications entirely, which is the new-config-on-disk-old-config-in-memory state the handlers part covers, and which in a rolling play is bounded to the size of one batch.

What happens if the unit file itself changed?

systemd is still running the old unit definition, and a restart will apply the old one.

systemd caches unit files in memory. Editing /etc/systemd/system/app.service changes the file on disk and nothing else; systemctl restart app then restarts the service using the definition systemd already has. New environment variables, changed ExecStart lines, modified resource limits — none of them take effect.

The systemd_service module has daemon_reload for exactly this:

Service impact possiblereloading the unit definition, then the service— daemon_reload makes systemd re-read unit files; the restart then uses the new definition. Both belong in the handler, in this order.

  handlers:
  - name: Reload systemd and restart the application
    ansible.builtin.systemd_service:
      name: app
      state: restarted
      daemon_reload: true

The failure mode without it is quietly wrong in the way this lesson keeps returning to: every task green, the service restarted, and the change not applied. A health check asserting the deployed version will not catch it either, if the version comes from the application rather than from the unit file — so a change to a unit file deserves a health check that asserts something the unit file controls.

Knowledge check

Why this matters

Restart versus reload looks like a micro-optimisation and is really a correctness question.

Choosing reload when the change needs a restart produces the failure this whole part is built to catch: every task green, the service answering, and the change not applied. It is invisible to the run log, invisible to systemd, and invisible to any health check that only asks whether the service responds. The fleet ends up in a state where the configuration on disk and the behaviour in production disagree, and the disagreement was introduced by a deployment that reported complete success.

The timing question has the same shape. A health check that runs before the flush is not a weaker check — it is a check of the previous release, which is worse than no check because it produces a green result about something you were not asking about.

Both come down to the same discipline the part opened with: the action and the verification are separate things, and the verification only means something if it happens after the action it is verifying. Getting the order wrong is not a subtle bug. It is a gate that reports on the past.