Service state, enablement and restart discipline · Ansible

A service task asks two questions that people routinely conflate: is it running now, and will it come back after a reboot. Ansible keeps them as separate parameters because they are separate decisions, and every combination of the two is a legitimate state that some host somewhere is deliberately in.

`state` and `enabled` are independent

`state`	`enabled`	What it means
`started`	`true`	running now, running after a reboot — the normal case
`started`	`false`	running now, gone after a reboot — a service being decommissioned, or one started by hand
`stopped`	`true`	not running now, back after a reboot — the state a failed maintenance leaves behind
`stopped`	`false`	not running, not coming back — properly disabled

The two middle rows are where the damage lives. started with enabled: false is the shape of a service that survives every check until the host reboots — which, in this course, is the night of the patch window.

Service impact possibleboth decisions, stated— Starts the service if it is not running and enables it if it is not enabled. Idempotent: on a host already in this state the module inspects, finds no difference, and reports ok.

- name: Ensure the application service is running and will survive a reboot
ansible.builtin.systemd_service:
  name: myapp
  state: started
  enabled: true

Omitting enabled does not mean false. It defaults to null, which means leave it as it is. That is the correct default for a module that must not surprise you, and it is also why “the service was running but did not come back after the reboot” is such a common finding: the role only ever expressed state: started, and the host was never enabled by anything.

`systemd_service` against the generic `service`

ansible.builtin.service is the portable abstraction, and it is a weaker tool in a way that matters.

The parameter lists diverge exactly where the interesting operations are. systemd_service has daemon_reload, daemon_reexec, masked, scope and no_block. service has none of them; it carries runlevel, pattern, sleep and arguments instead — parameters shaped by SysV init.

The check-mode difference is sharper still. Read from ansible-doc on ansible-core 2.21.3:

Module	`check_mode`	`diff_mode`
`ansible.builtin.systemd_service`	`full`	`none`
`ansible.builtin.service`	`N/A`	`N/A`

N/A on the generic module means the guarantee depends on whichever implementation it dispatched to — the same leak the package module has in the patching part, arriving from the same direction. A --check run of a deployment role built on service carries a guarantee that varies by host and does not say so.

`daemon_reload`, and doing it only when needed

systemd caches unit files. Writing a new .service file changes nothing until the manager re-reads it, and until then systemctl restart runs the old unit definition — the old ExecStart, the old environment, the old resource limits.

The symptom is memorable: you fix a unit file, restart the service, watch it come up with the behaviour you just corrected, and conclude the file is wrong. It is not; it has not been read.

Configuration changereload the manager only when the unit file changed— Writes the unit file and reloads the systemd manager only if the file actually changed. daemon_reload rereads unit definitions; it does not restart anything.

- name: Install the application unit file
ansible.builtin.template:
  src: myapp.service.j2
  dest: /etc/systemd/system/myapp.service
  owner: root
  group: root
  mode: '0644'
notify:
  - Reload systemd
  - Restart myapp

# handlers/main.yml
- name: Reload systemd
ansible.builtin.systemd_service:
  daemon_reload: true

- name: Restart myapp
ansible.builtin.systemd_service:
  name: myapp
  state: restarted

Two things make that correct.

daemon_reload is notified, not unconditional. A daemon_reload: true task in the main task list runs on every play, on every host, reporting changed each time — which turns every run yellow and makes the run output stop meaning anything.

Handler order comes from the handler file, not the notify list. Handlers run in the order they are defined, not the order they are notified. Reload systemd is defined before Restart myapp, so it runs first. Reversing the definitions would restart the service against the stale unit definition and then reload the manager, which is the same bug with extra steps.

daemon_reload and daemon_reexec are different operations. daemon_reload re-reads unit files; daemon_reexec re-executes the systemd manager binary itself, which is what you do after upgrading systemd, and is not part of an application deployment.

`masked` is a stronger statement than disabled

masked: true symlinks the unit to /dev/null, and a masked unit cannot be started at all — not by systemctl start, not by a dependency, not by socket activation.

That is the right tool for a genuinely retired service, and a foot-injury waiting to happen if it is left behind:

Service impact possibleretiring a service properly— Stops the service, prevents it starting at boot, and masks it so nothing can start it including a dependency of another unit.

- name: Retire the legacy sync daemon
ansible.builtin.systemd_service:
  name: legacy-sync
  state: stopped
  enabled: false
  masked: true

Restart only through a handler

The rule is simple and the reason is worth stating: a state: restarted task in the main task list restarts the service on every run, whether or not anything changed.

On a single host that is a few seconds of downtime nobody notices. On a rolling deployment across 200 hosts it is 200 unnecessary service interruptions, and it destroys the property the whole part depends on — that a run against an already-converged fleet is a no-op.

Service impact possiblethe difference, side by side— The first restarts unconditionally on every run. The second restarts only when a preceding task reported changed. Both are service-impacting; only the second is proportionate.

# Restarts on every run, converged or not.
- name: Restart the application
ansible.builtin.systemd_service:
  name: myapp
  state: restarted

# Restarts only when something notified it.
- name: Write the application configuration
ansible.builtin.template:
  src: myapp.conf.j2
  dest: /etc/myapp/myapp.conf
  mode: '0640'
notify: Restart myapp

state: started in the main task list is the complement and is correct: it is idempotent, it asserts the service is running, and it does nothing when it already is. started asserts; restarted commands.

`flush_handlers` before a validation gate

Handlers run at the end of the play by default. A validation task placed after the configuration change therefore runs before the restart, and validates the old process.

The result is the worst kind of green: the gate passed, the service was never restarted at the time it passed, and the new configuration went live afterwards with nothing checking it.

Service impact possibleforcing the restart to happen before the check— meta: flush_handlers dispatches all pending handlers at that point in the task list, so the health gate that follows tests the restarted service rather than the old process.

- name: Write the application configuration
ansible.builtin.template:
  src: myapp.conf.j2
  dest: /etc/myapp/myapp.conf
  mode: '0640'
  validate: '/usr/bin/myapp --check-config %s'
notify: Restart myapp

- name: Apply any pending restart now, before we check anything
ansible.builtin.meta: flush_handlers

- name: Prove the service is serving the configuration we just wrote
ansible.builtin.uri:
  url: 'http://{{ ansible_host | default(inventory_hostname) }}:8080/healthz'
  return_content: true
register: health
retries: 20
delay: 3
until: health.json.config_generation | default(0) | int >= expected_generation | int
changed_when: false

meta: flush_handlers is one of the choices the meta module accepts, alongside end_play, end_batch, end_host, clear_facts, reset_connection and the rest. It dispatches every pending handler at that point in the task list.

Under serial, handlers already flush at the end of each batch rather than at the end of the run — so the failure this fixes is narrower than it first appears, but it is exactly the failure a validation gate placed after the change runs into.

force_handlers is a different keyword answering a different question

meta: flush_handlers decides when handlers run. force_handlers decides whether they run when a host has failed.

The keyword documentation states it precisely: it “will force notified handler execution for hosts even if they failed during the play”, and “will not trigger if the play itself fails”.

The failure it addresses is a host that wrote new configuration to disk, notified a restart, then failed a later task — leaving the host with new configuration on disk and the old configuration in memory, which is a time bomb that detonates at the next unrelated restart.

force_handlers: true restarts it. Whether that is the right answer is genuinely a judgement — restarting a service on a host that just failed a task may be the last thing you want — and the handler part of this course works through both directions.

Under the hood

The handler mechanism is a deduplicated queue, and three of its properties explain most surprising handler behaviour.

Notification is per host. Each host maintains its own set of notified handlers. A handler notified on web01 and not on web02 runs on web01 only. This is why a rolling deployment restarts exactly the hosts whose configuration changed — a genuinely useful property that falls out of the design rather than being specified.

Notification is deduplicated. Three tasks notifying Restart myapp produce one restart, not three. This is what allows a role to notify the same handler from every task that warrants it without counting.

Execution order is definition order. The handler list is walked in the order handlers are defined, and notified ones fire. The order of the notify: entries in a task is irrelevant. This is the single most common handler surprise, and it is why Reload systemd must be defined above Restart myapp rather than merely listed first.

meta: flush_handlers drains the queue at that point and leaves it empty, so a handler notified again afterwards fires again at the end of the play. That is what makes the flush-then-validate-then-change-again pattern work rather than silently doing nothing the second time.

One asymmetry worth knowing: ansible.builtin.meta declares check_mode: partial. Under --check, flush_handlers dispatches the handlers in check mode — so the restart is predicted rather than performed, and the validation task after it, being uri with check_mode: none, is skipped entirely. A check run of this pattern verifies the shape of the play and nothing about the service.

What happens if the service fails to start after the deployment?

The handler fails, and what that means depends on where the failure lands.

systemd_service with state: restarted fails the task when systemd reports the unit did not reach an active state. Because it is a handler, the failure occurs during the handler flush — at the end of the play, or at meta: flush_handlers, or at the end of the batch under serial.

The consequence people find surprising: the tasks after the failing handler still ran, because they ran before the flush. A run can show green tasks throughout and a failed handler at the bottom, which reads as “almost everything worked” and actually means the service is down.

The diagnostic sequence is short and worth having memorised, because the handler failure message alone rarely says why:

systemctl status myapp --no-pager --lines=50
journalctl -u myapp --since '-10 min' --no-pager
systemd-analyze verify /etc/systemd/system/myapp.service

The third one catches unit-file errors that daemon-reload accepts silently.

The commonest causes, in the order they occur in practice: a configuration file that is syntactically valid and semantically wrong — which is what the validate parameter in the next lesson does and does not protect against; a missing dependency the new version needs; a port already bound by the old process because the restart raced its own shutdown; and a unit file that was changed without daemon_reload, so systemd is still running the previous definition.

Knowledge check

Why this matters

Service management is the part of a deployment that feels too simple to get wrong, and it supplies a steady share of production incidents anyway.

The pattern in all of them is the same: a state that was never declared, inherited from somewhere, correct by accident until something disturbed it. The service that was running and not enabled, discovered on the night of a patch reboot. The unit file that was updated without a daemon reload, whose fix appeared not to work. The masked unit from a retirement two years ago that makes a fresh install fail with an error nobody can trace.

None of these are hard problems. Each is one parameter, stated deliberately, in a role that gets rerun often enough to keep asserting it — which is the whole argument for configuration management, demonstrated on the smallest possible surface.

And the discipline that costs least and saves most: restart through a handler, never in the task list. It is what makes a converged fleet a no-op instead of two hundred unnecessary service interruptions.

state and enabled are independent

systemd_service against the generic service

daemon_reload, and doing it only when needed

masked is a stronger statement than disabled