Health checks that actually assert something · Ansible

Everything else in a rolling deployment is machinery for making this one step possible. serial exists so that a health check has a small batch to protect. The failure policy exists so that a failed health check stops something. The drain exists so that the host being checked is not serving users while you check it.

If the health check does not actually assert anything, all of that machinery is running in support of nothing.

Three levels of assertion

Health checks fall into three tiers, and the distance between them is where deployments go wrong.

Level	Question	Typical check	What it misses
Liveness	is the port open?	`wait_for` on the port	a process that accepts connections and returns 500 to all of them
Readiness	is the process serving?	`uri` expecting `200`	a service serving the old version, or serving from cache
Correctness	is it serving what I deployed?	`uri` asserting version and dependencies	very little — this is the one worth writing

The first two are the ones people write, because they are short. The third is the one that catches the failure this part exists to prevent.

Liveness: the port is open

Read-only / Safethe weakest useful check— Read-only. Confirms something is listening. Fails if the service did not start at all, which is a real class of failure and not the interesting one.

    - name: Wait for the application port to accept connections
    ansible.builtin.wait_for:
      host: '{{ ansible_host | default(inventory_hostname) }}'
      port: 8080
      state: started
      timeout: 60

This catches a service that failed to start. It does not catch a service that started and is broken, which — because the unit file, the systemd dependencies and the port binding all still work — is the more common outcome of a bad configuration change.

A process can bind a port before it is ready to serve, so this check can even pass before the service is usable.

Readiness: it answers

Read-only / Safethe check most people stop at— Read-only: a GET request, marked changed_when: false. Retries because a service that has just restarted needs a moment. Better than a port check and still not sufficient.

    - name: Wait for the health endpoint to return 200
    ansible.builtin.uri:
      url: 'http://{{ inventory_hostname }}:8080/healthz'
      status_code: 200
    register: health
    retries: 12
    delay: 5
    until: health.status == 200
    changed_when: false

The until / retries / delay combination is what makes this a gate rather than a coin flip. Without it, the single request lands whenever the task happens to run — often two seconds after a restart, when nothing is ready — and the task fails on a service that would have been fine four seconds later.

With retries: 12 and delay: 5 it will keep asking for a minute before giving up. A service that never comes back fails the task; a service that takes twenty seconds passes at the fourth attempt.

The named failure: 200 from a page the proxy served from cache

A health check asserting status_code: 200 against / is one of the most common ways to build a gate that cannot fail.

The request goes to the reverse proxy in front of the application. The proxy has a cached copy of / from four minutes ago. It returns it, with a 200, without ever consulting the application — which is down.

Every host passes. Every batch proceeds. The site is broken throughout, and the deployment log is spotless.

Three rules that prevent it:

Check a dedicated health endpoint, not the front page. /healthz or /-/ready, something the application implements specifically to answer this question and that is configured never to be cached.
Check the application directly, at its own port on the host being deployed, not through the proxy. You are asking whether this host is healthy; routing the question through a load balancer means it may be answered by a different host entirely.
Assert on the body, not only the status code. A cache cannot fabricate a version string that did not exist before this deploy.

Correctness: it is serving what you deployed

Read-only / Safethe check worth writing— Read-only. Asserts the running service reports the version just deployed and that its dependencies are healthy. A cached response cannot satisfy this, and neither can a service still running the previous release.

    - name: Assert the service is running the release we just deployed
    ansible.builtin.uri:
      url: 'http://{{ ansible_host | default(inventory_hostname) }}:8080/healthz'
      return_content: true
      status_code: 200
      timeout: 10
    register: health
    retries: 12
    delay: 5
    until:
      - health.status | default(0) == 200
      - health.json.version | default('') == release_version
      - health.json.database | default('') == 'ok'
    changed_when: false
    failed_when:
      - health.status | default(0) != 200
        or health.json.version | default('') != release_version

Every default() in that expression defaults to a value that fails the condition. This is the same rule as the drain check, and it matters more here: if the endpoint returns an error page instead of JSON, health.json.version does not exist, and default('') makes the comparison false rather than raising an undefined-variable error that might be caught somewhere unhelpful.

Health checks that are not HTTP

Not every service answers HTTP, and the same tiering applies.

Read-only / Safeasserting on a log line— Read-only: wait_for reads the file and matches a pattern. Useful for services whose readiness is only expressed in their log.

    - name: Wait for the service to log that it finished starting
    ansible.builtin.wait_for:
      path: /var/log/app/app.log
      search_regex: 'listening on :8080 version={{ release_version | regex_escape }}'
      timeout: 90

regex_escape matters — a version string containing a . is otherwise a regex wildcard, and 1.2.3 would match 1x2x3. It is a small thing that turns an exact assertion into a fuzzy one.

Matching a log file has a specific hazard

The upstream documentation for search_regex carries a warning worth repeating: when inspecting a system log file for a static string, remember that Ansible logs its own actions there.

If your check searches /var/log/syslog for a string, and your playbook mentions that string in a task name, the task can match its own footprint. Prefer the application own log file, and prefer a pattern that includes something only the new release could have written — the version number being the obvious candidate.

There is also a staleness problem wait_for cannot solve: a log line from the previous start is still in the file. Matching it proves the service started at some point, not that it started just now. Where that matters, check the process start time or rotate the log as part of the deploy.

For a service with a command-line health tool:

Read-only / Safeasserting via the service own tooling— Read-only: the health subcommand reports state without changing it, and changed_when: false keeps it out of the change signal.

    - name: Wait for the cluster member to report healthy
    ansible.builtin.command: /usr/local/bin/appctl health --json
    register: appctl
    changed_when: false
    retries: 20
    delay: 3
    until:
      - appctl.rc == 0
      - (appctl.stdout | from_json).state | default('') == 'healthy'

What `--check` does not tell you

This is the part that catches careful people, and it is verified from the module documentation itself.

Both of the modules that do the verifying in this pattern report check_mode: support: none:

Read-only / Safemodule check-mode support on ansible-core 2.21.3— Read-only introspection of the module documentation. support: none means the action is skipped in check mode, not simulated.

$ ansible-doc -t module ansible.builtin.uri | sed -n '/ATTRIBUTES/,/NOTES/p'

ATTRIBUTES:

      check_mode:
      description: Can run in check_mode and return changed status prediction
        without modifying target, if not supported the action will be skipped.
      support: none

      diff_mode:
      ...
      support: none

--- ansible.builtin.wait_for reports the same ---

      check_mode:
      support: none

Read the description carefully: “if not supported the action will be skipped”. Not simulated, not approximated — skipped.

So a --check run of a rolling playbook does this:

Step	Under `--check`
Drain (`command`/`uri`)	skipped
Verify drained (`wait_for`)	skipped
Deploy (`template`, `copy`)	simulated, reports would-change
Restart (handler)	simulated
Health check (`uri`)	skipped
Return to service	skipped

The deployment steps are simulated. Every verification step is skipped.

A clean --check run proves nothing about your health gate

This inverts the usual reassurance. --check is normally the thing you run to gain confidence before a real change, and for a rolling deployment it gives you confidence about exactly the half you were not worried about.

It will tell you the template renders and which files would change. It will not tell you whether the health check URL is right, whether the port is correct, whether the JSON path exists, or whether the whole gate is a typo that silently passes.

A health check has to be tested by making it fail. Point it at a port nothing is listening on and confirm the play stops. Point it at the wrong version string and confirm the play stops. A gate that has never been observed failing is a gate nobody has tested, and this pattern gives it no other opportunity to prove itself.

Under the hood

uri executes on the managed node by default, which is why http://{{ inventory_hostname }}:8080/healthz works: the request is made from the host to itself, or to whatever that name resolves to from there.

That detail has a consequence worth knowing. If you delegate_to: localhost a health check, it runs from the controller — and the controller may reach the service through a load balancer, a NAT, or a different DNS view. The check then answers “can the controller reach something at this address”, which on a load-balanced service may be answered by a completely different host from the one you just deployed.

The general rule: check from where the answer means what you want it to mean. Application health, checked on the host. Traffic actually arriving, checked from outside. Those are two different questions and this part uses both.

The retry mechanism sits outside the module. until, retries and delay are task-level keywords implemented by the executor: it runs the module, evaluates the until expression against the registered result, and if false, sleeps delay seconds and runs the module again. The module knows nothing about it. This is why until works with any module that registers a result, and why a module failure — a connection refused — still participates in the loop rather than ending the task immediately.

The documentation for the retries keyword states that it is “Number of retries before giving up in an until loop. This setting is only used in combination with until.” The delay keyword carries the same sentence.

Executed on 2.21.3, that is not what happens. A failing task with retries and no until retries anyway:

TASK [retries without until] ***************************************************
FAILED - RETRYING: [h01]: retries without until (3 retries left).
FAILED - RETRYING: [h01]: retries without until (2 retries left).
FAILED - RETRYING: [h01]: retries without until (1 retries left).
FAILED - RETRYING: [h01]: retries without until (0 retries left).
fatal: [h01]: FAILED! => {"attempts": 4, "changed": false, "msg": "nope"}

retries: 4 produced four attempts; retries: 2 produced two; removing retries produced none; and a task that succeeds is not retried regardless. So the observed behaviour without until is an implicit “retry until the task succeeds”, bounded by retries.

Treat this as an inconsistency to avoid rather than a feature to use. Write the until you mean — a health check whose pass condition is “the module did not error” is a weaker gate than one that states its condition explicitly, and relying on undocumented behaviour in the step that protects your fleet is a poor trade for two saved lines.

What happens if the health check is flaky and fails one host in ten for no real reason?

The rollout stops on a healthy fleet, and — worse — the check gets disabled.

This is the failure mode that ends with an organisation having no health gate at all. The check is unreliable, so it stops good deploys. Somebody adds ignore_errors: true, or widens the status_code list, or increases the retries until it always passes. Each step is locally reasonable and the endpoint is the same: a gate that cannot fail.

The fix is to make the check deterministic, not to make it lenient:

Retry enough for a genuinely slow start — a service that takes 30 seconds needs retries: 12 with delay: 5, not retries: 3.
Set an explicit timeout on uri so a hung connection fails fast and is retried, rather than hanging the whole task.
Assert on something stable. A version string is stable; a request count or a latency figure is not.
Check the application directly rather than through a proxy, so unrelated infrastructure cannot fail your gate.

A check that has been made lenient to stop it complaining is worse than no check, because it still occupies the place where a real one would go and everyone believes it is doing something.

Knowledge check

Why this matters

The health check is the only part of a rolling deployment that can tell the difference between a change that ran and a change that worked.

Every other step reports on itself honestly and uselessly. The template task tells you the file changed. The handler tells you the unit restarted. Both are true, both are green, and neither has any way to know that the service has been returning 500 to every request since the restart. That is not a defect in those modules — it is the boundary of what they claim.

Which means the entire safety value of the pattern is concentrated in the assertions you write yourself, and those assertions are exactly the ones --check will not exercise. The gate that protects your fleet is the least-tested code in the playbook unless you deliberately test it.

So test it by breaking it. Once, on purpose, before you need it. Point it at a port with nothing behind it and watch the rollout stop at batch one. That five-minute exercise is the difference between a health check and the appearance of one, and the appearance of one is what most fleets are actually running.