When a retry makes it worse · Ansible

A retry is a claim: this will probably work if I wait. When the claim is true, a retry loop turns a flaky dependency into a non-event. When it is false, it converts a fast failure into a slow one, or — worse — repeats a change that was not safe to repeat.

The keywords:

Read-only / Safethe three keywords, as documented by the installed version— Executed against ansible-core 2.21.3. Reads local plugin documentation.

$ ansible-doc -t keyword until retries delay

delay:
applies_to:
- Task
- Handler
description: Number of seconds to delay between retries. This setting is only used
  in combination with `until`.
priority: 0
template: explicit
type: float
retries:
applies_to:
- Task
- Handler
description: Number of retries before giving up in a `until` loop. This setting
  is only used in combination with `until`.
priority: 0
template: explicit
type: int
until:
applies_to:
- Task
- Handler
description: This keyword implies a '`retries` loop' that will go on until
  the condition supplied here is met or we hit the `retries` limit.
priority: 0
template: implicit
type: list

until is the one that turns the loop on. retries and delay do nothing without it.

How many attempts you actually get

The word “retries” suggests attempts in addition to the first one. Measured on 2.21.3, it is the total.

A task with until: false — a condition that can never be satisfied — wrapped in a block whose rescue reports ansible_failed_result.attempts:

Read-only / Saferetries: 1, 2 and 3, measured— Executed. For each value of retries, the number of FAILED - RETRYING lines and the attempts field of the final failed result.

$ for n in 1 2 3; do ansible-playbook -i inventory.ini r$n.yml; done

--- retries: 1 ---
RETRYING lines: 1
retries=1 -> attempts=1

--- retries: 2 ---
RETRYING lines: 2
retries=2 -> attempts=2

--- retries: 3 ---
RETRYING lines: 3
retries=3 -> attempts=3

retries: N gives N attempts, not N + 1. And omitting retries while using until gives you three:

Read-only / Safethe default— Executed. until with no retries keyword. The final failed result reports three attempts.

$ ansible-playbook -i inventory.ini rdef.yml

default retries -> attempts=3

One display quirk to expect: the countdown in the retry message reaches zero before the task gives up, so the last line before the failure reads (0 retries left). That is cosmetic — the attempt count above is the number that matters.

Read-only / Safea retry loop that succeeds on the third attempt— Executed. The until condition becomes true when attempts reaches 3. Two retry messages, then ok, and the registered attempts field reads 3.

$ ansible-playbook -i inventory.ini failedwhen.yml

TASK [retries and until] *******************************************************
FAILED - RETRYING: [localhost]: retries and until (2 retries left).
FAILED - RETRYING: [localhost]: retries and until (1 retries left).
ok: [localhost]

TASK [show attempts] ***********************************************************
ok: [localhost] => {
  "msg": "attempts=3"
}

The attempts field is on the registered result, which makes it available for a report — “this node needed four attempts to come back” is a useful thing to log.

The honest retry

A retry is honest when the failure is transient by nature — the thing you are waiting for is genuinely in the process of becoming true.

- name: wait for the service to answer its health endpoint
  ansible.builtin.uri:
    url: "http://127.0.0.1:{{ webapp_listen_port }}/healthz"
    status_code: 200
  register: health
  until: health.status == 200
  retries: 12
  delay: 5
  changed_when: false

Sixty seconds of patience for a service that was just restarted. The condition is a state the system is moving towards, the action is read-only, and repeating it costs nothing.

The three properties that make this one honest:

The action is idempotent, ideally read-only. A GET on a health endpoint can run a hundred times.
The condition can become true without further intervention. The service is starting. Nobody has to do anything.
The total wait is bounded and stated. 12 × 5 = sixty seconds, in the file, where a reviewer can see whether that is the right number for this service.

Blast radius: retries multiply into wall-clock time across a fleet

retries: 30, delay: 10 is a five-minute ceiling per host. Under serial: 1 across 200 hosts, that is a worst case of over sixteen hours, and the operator who set it was thinking about one node.

Compute the product before you commit it:

worst case per host  = retries × delay
worst case per batch = retries × delay          (hosts in a batch run in parallel)
worst case for run   = retries × delay × number of batches

Then ask whether you would rather wait that long or fail after two minutes and be told. For a rolling deployment the answer is nearly always the second, because a five-minute stall per node is indistinguishable from a hung run.

The dishonest retries

Retrying a deterministic failure

The credential is wrong. The package name is misspelled. The path does not exist. None of these become true by waiting, and a retry loop around them does exactly one thing: it delays the alert by retries × delay.

The symptom is a play that used to fail in ten seconds and now fails in four minutes, with the operator watching a spinner. Worse, on a large fleet the delay is often long enough that somebody cancels the run, so the failure is never even reported — it is inferred from a Ctrl-C.

Test: if you cannot name the thing that will change between attempts, the retry is not doing anything.

Retrying a non-idempotent action

This is the one that causes damage rather than delay.

# Do not do this.
- name: create the reporting user
  ansible.builtin.command: /usr/local/bin/create-user reporting
  register: r
  until: r.rc == 0
  retries: 5
  delay: 10

If the command creates the user and then fails — on a slow post-creation step, on a notification to a downstream service, on a timeout after the work was done — the retry runs it again. Five times. You now have five reporting users, or one user and four confusing failures, or a downstream system that received five conflicting notifications.

The retry did not absorb a fault. It amplified one.

The command that succeeds and then times out

This is the specific shape to watch for, and it is common in anything that talks to a network service:

The command performs the change.
It waits for confirmation.
The confirmation times out.
It exits non-zero.

From the outside that is indistinguishable from “the change did not happen”, which is precisely why a retry looks reasonable. The change did happen.

If you must retry something like this, the retry has to be preceded by a check that makes it idempotent:

- name: does the reporting user already exist
  ansible.builtin.command: /usr/local/bin/user-exists reporting
  register: user_check
  changed_when: false
  failed_when: false

- name: create it only if it does not
  ansible.builtin.command: /usr/local/bin/create-user reporting
  when: user_check.rc != 0
  register: created
  until: created.rc == 0
  retries: 3
  delay: 5

And at that point, ask whether a module exists that does the whole thing idempotently, because one usually does. Part VIII made this argument in general; it applies with extra force whenever a retry is involved.

Retrying to paper over a race

A retry loop added because “sometimes the config file is not there yet” is a missing ordering constraint wearing a disguise. The fix is to make the producing task run first, or to wait for the specific thing with ansible.builtin.wait_for, which states what it is waiting for instead of retrying something that happens to fail while it waits.

`meta: clear_host_errors`

A host that fails is removed from the play and does not appear in subsequent plays of the same run. meta: clear_host_errors clears that state — and it is narrower than people expect.

Read-only / Safewhat the module documentation says— Executed. The relevant part is the second sentence.

$ ansible-doc ansible.builtin.meta | grep -A4 clear_host_errors

'clear_host_errors' (added in Ansible 2.1) clears the
failed state (if any) from hosts specified in the play's
list of hosts. This will make them available for
targeting in subsequent plays, but not continue

It does not resume the current play for the failed host. Verified: a two-host play where one fails, a meta: clear_host_errors immediately after, and a task following that:

Read-only / Safeclear_host_errors does not bring the host back into this play— Executed. node-a fails, the meta task runs, and the next task runs only for node-b.

$ ansible-playbook -i two.ini clearerr4.yml

TASK [node-a fails] ************************************************************
fatal: [node-a]: FAILED! => {"changed": false, "msg": "boom"}
skipping: [node-b]

TASK [reactivate failed hosts] *************************************************

TASK [both hosts should reach here] ********************************************
ok: [node-b] => {
  "msg": "node-b is in play"
}

PLAY RECAP *********************************************************************
node-a                     : ok=0    changed=0    unreachable=0    failed=1
node-b                     : ok=1    changed=0    unreachable=0    failed=0

What it does do is restore the host for the next play:

Read-only / Safewith and without, across two plays— Executed twice on the same two-host inventory. The only difference between the playbooks is the meta task at the end of play one.

$ ansible-playbook -i two.ini clearerr5.yml; ansible-playbook -i two.ini clearerr6.yml

=== WITH clear_host_errors ===
ok: [node-a] => { "msg": "node-a in play two" }
ok: [node-b] => { "msg": "node-b in play two" }
node-a  : ok=1  changed=0  unreachable=0  failed=1
node-b  : ok=1  changed=0  unreachable=0  failed=0

=== WITHOUT ===
ok: [node-b] => { "msg": "node-b in play two" }
node-a  : ok=0  changed=0  unreachable=0  failed=1
node-b  : ok=1  changed=0  unreachable=0  failed=0

node-a appears in play two only when the meta task ran. And note the recap in both cases: failed=1 for node-a. Clearing the error state does not erase the failure from the statistics, which is correct — the run did contain a failure and the recap says so.

The placement trap

A meta: clear_host_errors placed after a failing task in a single-host play never runs at all. The host was removed when it failed, and the meta task has no host left to execute on:

TASK [this fails] ****************
PLAY RECAP ***********************
localhost : ok=0 changed=0 unreachable=0 failed=1

The task does not appear in the output. Nothing warns.

For it to run, at least one host in the play must still be active. Put it in a rescue, or at the start of a subsequent play, or accept that on a single-host play it is not going to help you.

Where it genuinely earns its place: a first play that probes an estate and tolerates failures, followed by a second play that should still consider every host. Used to make a failing host “come back”, it is a way of continuing past a failure without deciding to — which is the pattern lesson 7 spent its length arguing against.

Under the hood

The retry loop lives in the task executor and wraps module execution. With until set, the executor loops up to retries times (default 3), stamping attempts onto the result each pass, evaluating the until condition after each, and sleeping delay seconds between them.

Two consequences worth carrying.

The loop is per host, and hosts are executed in parallel up to forks. So a batch of ten hosts each retrying twelve times with a five-second delay takes sixty seconds in total, not ten minutes — the delays overlap. It is across serial batches that the multiplication is real, because batches are sequential.

The until condition is evaluated against the module result and templated per attempt, which means it can reference the registered variable of the task itself. That self-reference is legal because the register happens before the condition is evaluated, and it is the mechanism behind until: result.status == 200.

meta: clear_host_errors is not a retry mechanism at all. It manipulates the play iterator’s failed-host set, and — as the executed output shows — the iterator has already advanced past the failed host for the current play. The set it clears is consulted when the next play builds its host list, which is exactly the scope the documentation states and no wider.

What happens if until references a variable the task did not register?

The task fails on the undefined variable, on the first attempt, and the retry loop never engages.

The commonest form is a copy-paste where the register: name and the until: name drifted apart:

- name: wait for the endpoint
  ansible.builtin.uri:
    url: http://127.0.0.1:8080/healthz
  register: health_check
  until: health.status == 200        # wrong name
  retries: 12
  delay: 5

health is undefined, so the condition raises rather than evaluating, and the task fails on the first attempt — under a third of a second, with no RETRYING lines at all:

fatal: [localhost]: FAILED! => {"msg": "Task failed: An 'until' expression
failed: Error while evaluating conditional: 'health' is undefined"}

That is the good outcome — it is loud and it points at the right line. The bad variant is a condition that references something which exists but is never going to change, such as a variable set once before the loop:

  until: service_should_be_up | bool     # a static variable

This evaluates cleanly, evaluates identically every attempt, and burns the full retries × delay before failing. Verify a new retry loop by running it once against a target you know is not ready and confirming it takes about the time you expect — a loop that fails instantly or waits the full ceiling regardless is telling you something about the condition.