Retry loops for genuinely transient faults · Ansible

until is the third kind of loop. It does not iterate a list; it repeats one task until a condition holds or the retry budget runs out.

It is the right tool for a narrow class of problem and it is routinely used for a much wider one, which is how a fleet ends up with automation that “works, you just have to run it twice sometimes”.

The mechanics

Read-only / Safethe shape— Polls a health endpoint until it returns 200. uri with a GET changes nothing on the target, though the endpoint may log the request.

- name: Wait for the application to report healthy
ansible.builtin.uri:
  url: "http://{{ inventory_hostname }}:8080/health"
  status_code: 200
register: health
until: health.status == 200
retries: 12
delay: 5

until is a conditional evaluated after each attempt, in the same raw Jinja as when. It reads the registered result, which means register is mandatory — without it there is nothing for the condition to consult.

The defaults, confirmed by execution rather than assumed:

Read-only / Safethe default budget, timed— An until loop with neither retries nor delay set, against a condition that never becomes true. The attempts key in the result reports the count.

$ /usr/bin/time -f 'elapsed %e s' ansible-playbook retry2.yml

TASK [No retries or delay specified] *******************************************
FAILED - RETRYING: [localhost]: No retries or delay specified (2 retries left).
FAILED - RETRYING: [localhost]: No retries or delay specified (1 retries left).
FAILED - RETRYING: [localhost]: No retries or delay specified (0 retries left).
fatal: [localhost]: FAILED! => {"attempts": 3, "changed": false,
                              "stat": {"exists": false}}
...ignoring

elapsed 15.86 s

retries defaults to 3 and delay to 5 seconds. The result carries attempts: 3, and the 15.86 seconds of wall clock confirms three delays of five seconds rather than two — the delay is applied after each failed attempt, including the last one before giving up.

That last detail matters when you are budgeting: the worst case is retries x delay of pure waiting, not (retries - 1) x delay.

`retries: 2` means two attempts, not one attempt plus two retries

Read-only / Safecounting attempts— An until loop with retries set to 2. Two attempts are made in total, and the result reports attempts: 2.

$ ansible-playbook retry.yml

TASK [Wait for a path that never appears] **************************************
FAILED - RETRYING: [localhost]: Wait for a path that never appears (1 retries left).
FAILED - RETRYING: [localhost]: Wait for a path that never appears (0 retries left).
fatal: [localhost]: FAILED! => {"attempts": 2, "changed": false,
                              "stat": {"exists": false}}

retries: 2 produced attempts: 2. The name reads like “extra tries after the first” and it is not — it is the total attempt count.

Get this wrong in a wait loop and your timeout is off by one interval, which on a 60-second delay is a minute of difference between “the service was slow” and “the deployment failed”.

The condition has to be able to become true

An until whose condition can never become true is a way of spending retries x delay seconds to arrive at the same failure.

Three conditions that cannot resolve themselves:

Anything depending on state only this play changes, where the changing task comes after the wait.
A URL that is wrong. It will be wrong on attempt twelve too.
A credential that is rejected. Authentication failures are not transient, and retrying them is how accounts get locked out.

Before writing an until, answer: what event, external to this task, would make the condition become true? If you cannot name one, the retry loop is not going to help.

Good answers: a service is starting and will finish; a package repository mirror is briefly unavailable; a cloud API is rate-limiting; a cluster is electing a leader; a device is rebooting.

Waiting versus retrying

These are different intents and they want different tools.

Waiting is “the condition is not true yet and will become true”. The task is read-only, and repeating it is free.

Retrying is “the operation failed and might succeed if repeated”. The task changes something, and repeating it is only safe if the module is idempotent.

Read-only / Safewaiting, done with a read-only probe— Polls until the service is listening. wait_for opens a socket and reads nothing; it changes nothing on the target.

- name: Wait for the service to accept connections
ansible.builtin.wait_for:
  host: "{{ inventory_hostname }}"
  port: 8080
  state: started
  timeout: 60
  delay: 2

Where a purpose-built wait module exists — wait_for, wait_for_connection — prefer it. It is one task with one result and a single timeout, rather than a retry loop whose budget you compute in your head. until is for waits that no module covers.

For retrying a change, the requirement is stricter:

The diagnostic damage

Here is the operational case against casual retries, and it is the reason this lesson exists.

A task fails intermittently — one host in twenty, once a week. Someone adds retries: 5. The failure stops appearing. The run is green.

What actually happened:

The fault is still there. It now takes longer and reports success.

The evidence is gone. The failure was the only signal that something in the estate was degraded. A DNS resolver dropping one query in twenty is a real problem with a real cause, and now nothing will report it until it drops five in a row.

The blast radius grew quietly. The condition worsens over months. One in twenty becomes one in five. The retries still absorb it, until the day they do not, and the incident starts with “this has never failed before” — which is false, and would have been visibly false.

The runs got slower. Every affected host pays delay seconds on every run, and nobody attributes the slowdown to the retry because the retry is working.

The discipline: a retry loop is a statement that you have diagnosed the fault and it is transient. Not a way of avoiding the diagnosis.

Where you add one, make it visible:

Read-only / Safea retry that leaves evidence— Retries a read-only health check and reports when the first attempt did not succeed, so an increase in retries is visible rather than absorbed.

- name: Wait for the application to report healthy
ansible.builtin.uri:
  url: "http://{{ inventory_hostname }}:8080/health"
  status_code: 200
register: health
until: health.status == 200
retries: 12
delay: 5
# Transient: the app takes up to 60s to warm its cache after a
# restart. Confirmed against the application team, CHG-004417.

- name: Flag hosts that needed more than one attempt
ansible.builtin.debug:
  msg: >-
    {{ inventory_hostname }} needed {{ health.attempts }} attempts
    to become healthy - investigate if this is climbing
when: health.attempts | default(1) | int > 1

Two things there are the whole point. The comment naming why the fault is transient, with a ticket, so a reviewer in a year can tell whether the reason still applies. And the task that reports when the retry did work, so “the retry budget is being consumed more than it used to be” is a visible trend rather than an invisible one.

attempts is present on the result whenever a retry loop ran. On a first-attempt success it is absent, hence the default(1).

Budgeting

The worst case is retries x delay seconds per host, and hosts run in parallel up to forks.

For 400 hosts at forks: 20, with retries: 12 and delay: 5:

Worst case per host: 60 seconds of waiting.
Batches: 400 / 20 = 20.
Worst case for the play: 20 minutes of waiting, if every host exhausts its budget.

That is fine for a deployment with a maintenance window and it is not fine for a check that runs every fifteen minutes. Compute the number before you set the values, and remember that the case where every host exhausts its budget is exactly the case where something is badly wrong and you would rather find out quickly.

A shorter budget that fails fast is often the better operational choice: it turns a slow degraded run into a fast clear failure, and a fast clear failure is something you can act on.

Under the hood

until, retries and delay are task keywords handled by the task executor. ansible-doc -t keyword reports until as applying to Task and Handler, with retries and delay described as “only used in combination with until” — setting them without until does nothing at all, silently.

The executor runs the task, evaluates the until expression against the result, and repeats until the expression is true or the attempt count is exhausted. When it gives up, the last result is what gets registered, with attempts added.

That has a consequence for failed_when: it is evaluated on each attempt, so a failed_when that marks the task failed also feeds the retry decision. And because the last result is the one registered, a task that eventually succeeded registers a successful result with a raised attempts count — the earlier failures are visible only in the run log, not in the variable.

until has no interaction with loop. A task cannot iterate a list and retry each element with these keywords; until retries the task as a whole. Retrying per item means moving the retry inside an included task file that the outer loop calls.

What happens if the until condition references a key the failed result does not have?

The loop fails on the condition rather than on the task, and the error can be confusing because it names a template evaluation problem in what looks like a working retry.

until: health.status == 200

If the uri task fails in a way that produces no status key — a DNS failure, a connection refused before any HTTP response — then health.status is undefined and the until expression errors.

The safe form supplies a fallback in the condition itself:

until: health.status | default(0) == 200

Or, better, tests the result rather than indexing into it, using the tests from lesson 2:

until: health is succeeded

The second form is usually what you meant. The condition “the task succeeded” survives every failure mode, including the ones that produce no module keys at all — which is exactly the situation a retry loop exists to ride out.

Knowledge check

Why this matters

Retry loops are the most quietly corrosive feature covered in this part, because using them badly makes everything look better.

The failures stop. The dashboard goes green. The team stops getting paged. Every visible signal improves, and the only thing that got worse is your knowledge of your own estate — which is invisible until the day you need it.

A retry loop is a claim about the world: this fault is transient, an external event resolves it, and riding it out is correct. Where that claim is true, until is exactly right and the automation is better for it. Where it is untested, you have converted a reliable signal into a slow one, and you will find out which kind you had during an incident rather than before one.

Write the claim down in a comment next to the retry. If you cannot write it, do not add the retry.