Proving a host came back, not just answered · Ansible

A host that answers SSH after a reboot has proven exactly one thing: sshd started.

That is a low bar and it is the bar most patch plays stop at. The host whose data volume failed to mount answers SSH. The host whose application did not start answers SSH. The host that came up in degraded state with a failed unit answers SSH. The host that rebooted into the old kernel because the bootloader default was never updated answers SSH, cheerfully, for months.

Validation is the step that turns “answered” into “working”, and it is the only thing standing between a rolling patch run and a rolling outage.

Three questions, in order

Question	Answered by	What it proves
Is it reachable?	`wait_for_connection`, or the `reboot` module’s `test_command`	the transport works
Is it ready?	`systemctl is-system-running`, mounts, `service_facts`	the operating system finished booting correctly
Is it healthy?	an application probe, cluster membership	the host is doing its job

They are strictly ordered, and each one failing means something different. Skipping to the third produces confusing failures: an application probe against a host that is still booting fails for reasons that have nothing to do with the application.

Readiness: `systemctl is-system-running`

The command reports one of eight states and exits zero for exactly one of them.

State	Meaning	Exit code
`initializing`	early bootup, before `basic.target`	non-zero
`starting`	late bootup, job queue not yet idle	non-zero
`running`	fully operational	`0`
`degraded`	operational, but one or more units failed	non-zero
`maintenance`	rescue or emergency target is active	non-zero
`stopping`	the manager is shutting down	non-zero
`offline`	the manager is not running	non-zero
`unknown`	state could not be determined	non-zero

degraded is the state this whole lesson exists for. The system is operational. SSH works. Everything looks fine. One or more units failed to start — and on a host you just patched, the unit that failed is disproportionately likely to be the one whose package you just replaced.

The --wait option is important for post-reboot use: it blocks until the boot process completes rather than reporting initializing or starting. Without it, running the check too early returns a non-zero state that means “not finished yet”, which is indistinguishable in the exit code from “broken”.

Read-only / Safethe readiness gate— Read-only. is-system-running queries the systemd manager and changes nothing. --wait blocks until boot completes, so the result means the system finished rather than that it was asked too early.

- name: Wait for systemd to finish booting and report its state
ansible.builtin.command:
  argv: [systemctl, is-system-running, '--wait']
register: system_state
changed_when: false
failed_when: false

- name: Refuse to continue with a system that is not fully running
ansible.builtin.assert:
  that: system_state.stdout | trim == 'running'
  fail_msg: >-
    {{ inventory_hostname }} came back in state
    '{{ system_state.stdout | trim }}' rather than 'running'.
    Investigate with: systemctl --failed
  success_msg: '{{ inventory_hostname }} reports running'

Asserting on the stdout string rather than the exit code is deliberate. Both give you pass or fail; only the string tells you which of the seven failure states you are in, and degraded and maintenance want very different responses.

Some fleets live in degraded, and pretending otherwise makes the gate useless

On a real fleet there is often a unit that has been failing since 2023 that nobody will fix — a vendor agent, a hardware monitor on a machine that lacks the hardware.

If degraded is the normal state of forty hosts, an assertion on running fails those forty every window, and within two months somebody adds ignore_errors: true and the gate is gone.

The honest resolutions, in order of preference: fix or mask the unit, so running is achievable; or compare against a recorded baseline of which units were failing before the patch, and fail only on units that are newly failed. The second is more work and is the correct answer for a fleet you have inherited.

What is not acceptable is weakening the assertion to accept degraded everywhere. That deletes the check on every host in order to accommodate forty.

Readiness: services, from facts rather than shell

ansible.builtin.service_facts populates ansible_facts.services. The structure was confirmed by running it under ansible-core 2.21.3: a dictionary keyed by service name, each value carrying name, source, state and — for systemd units — status.

Read-only / Safewhat service_facts actually returns— Read-only. Executed against a local connection on 2.21.3. The module gathers facts and changes nothing; 339 services were reported on this host.

$ ansible -i localhost, -c local localhost -m service_facts

"ModemManager.service": {
"name": "ModemManager.service",
"source": "systemd",
"state": "running",
"status": "enabled"
}
"NetworkManager.service": {
"name": "NetworkManager.service",
"source": "systemd",
"state": "stopped",
"status": "not-found"
}
"apparmor": {
"name": "apparmor",
"source": "sysv",
"state": "running"
}

Two details in that output are worth carrying.

status: not-found with state: stopped. A unit name can appear in the dictionary without the unit existing — systemd knows the name because something references it. So a check written as “the key is present and state is not running” will flag units that were never installed. Check status too, or check for state == 'running' directly.

source: sysv entries have no status key. A filter that assumes status exists on every entry raises an undefined-attribute error on the first host with a SysV init script. Use | default('').

Read-only / Safeasserting the services this host owns are running— Read-only. Reads service state from gathered facts. The list of services comes from inventory, so each host asserts on what it is actually supposed to run.

- name: Collect service state
ansible.builtin.service_facts:

- name: Assert every service this host owns is running
ansible.builtin.assert:
  that:
    - ansible_facts.services[item] is defined
    - ansible_facts.services[item].state | default('') == 'running'
  fail_msg: >-
    {{ inventory_hostname }}: {{ item }} is
    '{{ ansible_facts.services[item].state | default("absent") }}'
    after the reboot, not running.
loop: '{{ host_required_services }}'
loop_control:
  label: '{{ item }}'

host_required_services belongs in group_vars, next to the group that defines what the host is for. A validation step that hard-codes nginx is a validation step for one tier that gets copied and half-edited for the next.

Readiness: mounts

A host that boots without one of its filesystems is the classic silent post-reboot failure. The application starts, writes to what it thinks is the data directory, and is actually writing to the root filesystem underneath an empty mountpoint. Nothing errors. Disk usage on / climbs for a week.

Read-only / Safeasserting the filesystems came back— Read-only. Reads ansible_facts.mounts, gathered by setup. Confirms the mountpoint is present rather than that the directory exists.

- name: Refresh facts after the reboot
ansible.builtin.setup:
  gather_subset:
    - '!all'
    - '!min'
    - hardware

- name: Assert every expected filesystem is mounted
ansible.builtin.assert:
  that: >-
    ansible_facts.mounts | selectattr('mount', 'equalto', item)
    | list | length == 1
  fail_msg: >-
    {{ inventory_hostname }}: {{ item }} is not mounted after the
    reboot. Anything writing there is writing to the parent filesystem.
loop: '{{ host_required_mounts }}'

Health: the application probe

Readiness says the operating system is fine. Health says the service is serving.

Read-only / Safea probe that asserts on content, not on 200— Read-only. A GET against the application's own health endpoint, retried until it reports ready. return_content and the until conditions make it a statement about what the service said, not about whether a socket accepted a connection.

- name: Wait for the application to report itself healthy
ansible.builtin.uri:
  url: 'http://{{ ansible_host | default(inventory_hostname) }}:8080/healthz'
  return_content: true
  status_code: [200]
  timeout: 10
register: health
retries: 30
delay: 5
until:
  - health.status | default(0) == 200
  - health.json.database | default('') == 'connected'
  - health.json.version | default('') == expected_version
changed_when: false

Every default() in those conditions is chosen so an absent field fails. health.json.database | default('') compared against 'connected' is false when the field is missing, which is the correct reading: a health endpoint that stopped reporting a field is not a health endpoint that is reporting good news.

The version check is the one people leave out and the one that catches the worst outcome. A host that rebooted into the old kernel, or whose package upgrade silently rolled back, will pass every other check in this lesson. Asserting that the running version is the version you deployed is the only check that catches it.

Health: cluster membership

For anything clustered — a database replica set, a Ceph OSD host, an etcd member, a message broker — “the process is running” is a particularly weak claim. A node can run, accept connections and be outside the cluster.

Read-only / Safeasking the cluster, not the node— Read-only. Queries the cluster's own view of membership from a different node, so a node that thinks it is fine but is not in the cluster fails the check.

- name: Ask the cluster whether this node has rejoined
ansible.builtin.uri:
  url: 'http://{{ cluster_api_host }}/v1/members'
  return_content: true
delegate_to: '{{ groups["cluster"] | difference([inventory_hostname]) | first }}'
register: members
retries: 20
delay: 15
until: >-
  members.json.members | default([])
  | selectattr('name', 'equalto', inventory_hostname)
  | selectattr('healthy', 'equalto', true)
  | list | length == 1
changed_when: false

The delegate_to is what makes this meaningful: the question is put to a node other than the one being validated. A node asked about itself will report itself present.

Placing validation so its failure matters

Everything above is worthless if a failure gets recorded and the run carries on.

Service impact possiblethe batch shape, with validation as the gate— Patches, reboots and validates one batch. The play-level max_fail_percentage means a validation failure in any batch prevents the next batch from starting.

- name: Patch and reboot with validation gating the next batch
hosts: '{{ patch_target }}'
become: true
serial: '{{ patch_serial }}'
max_fail_percentage: 0
tasks:
  - name: Apply updates
    ansible.builtin.include_tasks: patch.yml

  - name: Reboot if required
    ansible.builtin.reboot:
      reboot_timeout: 900
      post_reboot_delay: 30
    when: reboot_required | default(false)

  - name: Refresh facts, then validate
    ansible.builtin.include_tasks: validate.yml

Three properties of that shape are worth stating explicitly.

Validation is a task in the same play, not a separate playbook run afterwards. A separate run cannot stop a batch that has already finished.

It comes after the reboot in the task list, so the batch is not considered complete until it passes.

ignore_errors appears nowhere. A validation task with ignore_errors: true is a log line, not a gate, and the difference is invisible in the playbook until the night it matters.

Read-only / Safea validation failure stopping the run, executed on 2.21.3— Ten local-connection hosts, serial: [1, 3, 6], with an assert standing in for the validation step and failing on the canary. Only the assert module ran; no host state was touched.

$ ansible-playbook -i inv10.ini batchstop_canary.yml

TASK [Post-reboot validation gate (h01 fails it)] ******************************
fatal: [h01]: FAILED! => {
  "assertion": "inventory_hostname != 'h01'",
  "changed": false,
  "evaluated_to": false,
  "msg": "h01 came back but its service did not start"
}

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
h01                        : ok=1    changed=0    unreachable=0    failed=1

exit=2

One host in the recap, out of ten in the inventory. The other nine were never touched. That is what a working gate looks like: the evidence of its success is a recap that is shorter than the fleet.

Under the hood

Why validation must live in the same play as the reboot, mechanically.

serial batching happens in the play iterator. The iterator hands the strategy a batch, the strategy runs the play’s entire task list against it, and only when the task list is exhausted does the iterator hand over the next batch. Failure policy — max_fail_percentage, any_errors_fatal — is evaluated by the strategy after each task, against the current batch.

That machinery has exactly one lever: a task that fails. There is no mechanism by which anything outside the play can prevent the iterator from issuing the next batch.

So a validation playbook run after the patch playbook cannot gate anything. By the time it runs, every batch has completed. It can tell you the fleet is broken; it cannot stop the fleet from being broken. The same is true of a post_tasks section — which does run per batch, but after the handler flush, and is the wrong place for a gate that should prevent the reboot’s consequences from propagating.

The corollary applies to async as well. A validation task with async and poll: 0 returns immediately and the strategy sees a successful task launch, not a validation result. Fire-and-forget validation is not validation; it is a job you have started and will never look at.

What happens if validation passes but the host is broken anyway?

It happens, and the useful response is to make the check stronger the next time rather than to conclude that checks do not work.

The commonest instances, and what each one teaches:

The health endpoint returns 200 unconditionally. Many application health endpoints are a static handler that reports the process is alive. It proves the HTTP listener exists. Assert on content — a database connection state, a version, a queue depth — and the endpoint has to know something to pass.

The service is running but has no work. A worker that starts, fails to connect to its broker, and retries silently is active (running) in systemd and idle in reality. The check that catches it is a metric that moves, not a state that is reported.

A slow failure. A memory leak in the new kernel’s driver, a filesystem that mounts and then goes read-only under load. No check run sixty seconds after boot will see these. The soak interval from Part XXXII is what covers them, and it is why a patch run wants a soak just as much as a deployment does.

Everything is fine and the fleet is wrong. Each host passes individually and the fleet has lost quorum because the batch was too large. Per-host validation cannot see this. It needs a fleet-level check in an unbatched play — and that is a genuinely different check from everything in this lesson.

The pattern across all four: validation catches the failures you anticipated. Every incident is an opportunity to add the one you did not, and a validation suite that has not grown in two years is a validation suite that is testing the failures of two years ago.

Knowledge check

Why this matters

The conditional-reboot argument from the previous lesson, and the serial argument from the first, both cash out here.

Batching limits how many hosts a bad change reaches per batch. It does nothing about how many batches run. What stops the second batch is a task in the first that failed, and the only task in a patch play positioned to notice that a host came back wrong is the validation step.

Without it, serial: [1, 5, 30] is a slower way to break three hundred hosts. The canary gets patched, rebooted, and comes back with its data volume unmounted; nothing checks; the batch completes; five more; then thirty; then thirty. The recap is green throughout, because every task did what it was told.

That is the incident this part is built around, and it is not a failure of serial. It is serial working exactly as designed, on a play that never gave it anything to stop for.

Validation is what a batch boundary is for.