Health gates that actually gate · Ansible

A health check and a health gate are different things, and the difference is not in the check.

A gate is a check whose failure changes what happens next. A monitor is a check whose failure is recorded somewhere. The same uri task is either one depending on three things around it: whether it can fail, whether the play stops when it does, and whether anything after it depends on it having passed.

Most health checks in deployment roles are monitors. They look identical to gates in review.

The three ways a gate stops being one

Read-only / Safea check that cannot fail— Read-only. ignore_errors means the play continues regardless of the result, so this task can only ever produce a log line.

# A monitor. Reads like a gate.
- name: Check the application is healthy
ansible.builtin.uri:
  url: 'http://{{ inventory_hostname }}:8080/healthz'
ignore_errors: true

ignore_errors: true is the obvious one. It is usually added the week after a flaky check failed a run, and it is never removed.

No failure policy on the play. A check that fails one host in a serial: 10 batch, on a play with no max_fail_percentage, stops that host and lets the other nine continue — and the next batch starts on schedule. The check failed and the deployment proceeded.

Placement after the last thing that matters. A health check as the final task of the play, after the last batch, is a report. There is nothing left for it to gate.

The retry arithmetic, which is not what it looks like

until reruns a task until its condition holds. retries and delay govern how many times and how far apart. The upstream defaults are retries: 3 and delay: 5 — small enough that an unset retries on a service that takes twenty seconds to start is a guaranteed failure.

The arithmetic people do is retries × delay. That is the sleeping time, not the elapsed time.

Read-only / Safea health gate with an honest budget— Read-only. Polls the health endpoint until it reports the deployed version. The worst case is thirty attempts, each able to consume the full request timeout, plus twenty-nine delays.

- name: Wait for the application to serve the version we deployed
ansible.builtin.uri:
  url: 'http://{{ ansible_host | default(inventory_hostname) }}:8080/healthz'
  return_content: true
  status_code: [200]
  timeout: 10
register: health
retries: 30
delay: 5
until:
  - health.status | default(0) == 200
  - health.json.version | default('') == myapp_version
  - health.json.database | default('') == 'connected'
changed_when: false

Worst case for that task:

30 attempts × 10s request timeout   = 300s
29 delays    ×  5s                  = 145s
                                    -----
                                      445s  ≈ 7.5 minutes, per host

The naive figure is 150 seconds. The real one is nearly three times that, and it is reached exactly when the service is not responding at all — the case where every attempt burns the full timeout rather than failing fast.

Two mechanical details worth knowing:

The registered variable gains an attempts key recording how many tries the task took. Logging it turns “the deployment felt slow tonight” into a number, and a service whose attempts has been creeping up over months is telling you something before it fails.

Since ansible-core 2.16, retries works without until — the task retries until it succeeds, at most retries times. That is useful for a genuinely flaky operation and is not what a health gate wants: a health gate should express its success condition explicitly, so the run log says what was being waited for.

`wait_for` on a port is not a readiness check

wait_for polls a TCP port and succeeds when something accepts a connection. That proves a process called listen(). It proves nothing about whether that process can serve a request.

The gap is not theoretical. A JVM application binds its port during start-up and then spends forty seconds initialising; a service with a cold connection pool accepts connections and returns errors; a process that crashed and was restarted by systemd binds the port immediately and fails every request.

Read-only / Safethe port check as a cheap first stage, not the gate— Read-only. Waits for the port to accept connections, then for the application to report itself ready. The first is fast and catches a service that did not start at all; the second is the actual gate.

- name: Wait for the listener to appear
ansible.builtin.wait_for:
  host: '{{ ansible_host | default(inventory_hostname) }}'
  port: 8080
  state: started
  timeout: 60

- name: Wait for the application to report itself ready
ansible.builtin.uri:
  url: 'http://{{ ansible_host | default(inventory_hostname) }}:8080/readyz'
  return_content: true
register: ready
retries: 20
delay: 3
until: ready.json.status | default('') == 'ready'
changed_when: false

Note the explicit host:. wait_for defaults host to 127.0.0.1, which means an unqualified wait_for polls a port on the host the task is running on — and under delegate_to: localhost it polls the controller. That default is correct for its commonest use and wrong for the one people write first.

Assert on content, not on 200

An HTTP 200 from a health endpoint means a handler returned. In a large share of real applications, that handler is a static return 200 that knows nothing about the state of anything.

Read-only / Safewhat to assert on instead— Read-only. Each condition is a claim the endpoint has to know something to satisfy, and each default is chosen so an absent field fails the condition.

  until:
  - health.status | default(0) == 200
  - health.json.version | default('') == myapp_version
  - health.json.database | default('') == 'connected'
  - health.json.migrations_pending | default(1) | int == 0
  - health.json.queue_depth | default(999999) | int < 1000

Every default() there is chosen to fail the condition. That is the discipline: default('') compared against a real value is false; default(1) compared against 0 is false; default(999999) fails a threshold. A field the endpoint stopped reporting must not read as good news.

The version condition is the one to add first if a role has only one. It is what distinguishes “the service is healthy” from “the service is healthy and is running what this play deployed” — and the second is the question a deployment is actually asking.

A gate that passes because you waited long enough is not measuring anything

The failure mode of generous retries: a check with retries: 200 and delay: 10 will eventually pass on almost anything, because almost anything comes good within half an hour, including a service that crash-looped forty times and finally started.

The symptom is a deployment that “works” and takes wildly variable amounts of time. The variance is the signal, and nobody looks at it because the run was green.

Set retries from what the service takes to start when it is working, plus a modest margin. If start-up is twenty seconds, retries: 20 with delay: 3 is a minute — generous. retries: 200 is not a safety margin; it is a decision to accept any behaviour at all.

And log attempts. A gate that normally passes on attempt two and passed on attempt eighty-three tonight is information you want before the incident, not after.

Under the hood

until is implemented in the task executor rather than in any module, and that placement explains its interactions.

For a task with until, the executor runs the action, evaluates the condition against the result, and if it is false sleeps delay seconds and runs the action again — up to retries times. The condition is “implicitly templated”, so it is written without {{ }} and evaluated as an expression against the registered variable.

Three consequences.

The task’s own failure and the until condition are separate. A uri task that gets a connection refused fails; until still applies and the task is retried. This is why a health gate works at all against a service that is not yet listening — the early attempts fail outright rather than returning a result the condition can inspect.

The result of the last attempt is what is registered. After the retries are exhausted, the registered variable holds the final attempt, and the task is reported failed. So the run output shows one failure with the last body, not thirty — which is usually what you want, and is occasionally frustrating, because the first failure sometimes had the more informative message. -vv shows the individual attempts.

changed_when: false matters more than it looks. A uri GET reports changed: false naturally, but a uri with a method that implies modification, or a command-based probe, reports changed: true on every attempt. Under a handler-driven deployment that means the health check itself notifies the restart handler, and the service is restarted because it was checked. Setting changed_when: false on every probe removes the possibility.

One thing until does not do: it does not apply to async tasks polled with poll: 0, and ansible.builtin.uri declares check_mode: none, so under --check the whole gate is skipped and the retry logic never runs.

What happens if the health endpoint itself is the thing that broke?

The gate fails, the deployment stops, and the service is fine. This is the false positive, and it is worth planning for because the response to it under pressure is usually to delete the gate.

The realistic instances: a new release renamed /healthz to /health and the role was not updated; the endpoint moved behind authentication; a JSON field was renamed, so health.json.database is now health.json.db and your default('') correctly fails a condition that is now asking about a field that does not exist.

In every case the service is serving users perfectly and the play has stopped the rollout.

The right response is not to loosen the gate. It is to recognise that the health contract is part of the deployment interface, and that changing it is a change to something the deployment depends on. Two things make that concrete: put the health endpoint’s shape in the application’s own tests, so a rename fails CI rather than a change window; and version the endpoint alongside the application, so the role can select the right one.

The wrong response, and the common one, is ignore_errors: true on the health gate. That converts the false positive into a permanent absence of a true positive, and it will be discovered on the night the service genuinely does not come back.

There is one legitimate escape hatch: a variable that disables the gate, defaulting to enabled, named so that using it is conspicuous in a run log — skip_health_gate: true passed on the command line during a known incident, never set in group_vars. The distinction is that somebody chose it for one run rather than the repository having quietly forgotten it.

Knowledge check

Why this matters

The health gate is where a deployment role stops describing state and starts making a claim about behaviour, and it is the only task in the role that can tell the difference between “everything was applied” and “it works”.

Everything before it is verifiable by inspection. The package is at the right version — checkable. The configuration matches the template — checkable. The service is active — checkable. All three can be true on a host that is returning 500 to every request, and a deployment that stops at “active” has verified the parts and not the whole.

That is also why the gate is the task most often weakened. It is the only one that fails for reasons outside the role’s control, so it is the only one that ever fails on a night when nothing is wrong, and the fix that removes tonight’s problem — ignore_errors, or two hundred retries — removes the check permanently.

The discipline is to keep the gate strict and make the contract stable: an endpoint that reports version and dependencies, tested by the application’s own CI, changed deliberately. A gate is only as good as the thing it is allowed to ask.

The three ways a gate stops being one

The retry arithmetic, which is not what it looks like

wait_for on a port is not a readiness check

Assert on content, not on 200

Knowledge check

`wait_for` on a port is not a readiness check