AnsibleXLIV · Failure Modes and Partial Fleet FailureFailure Modes and Partial Fleet Failure
The seven things that can happen to a host
What you'll learn
- State what each of the seven recap counters proves about a host actual state
- Explain why failed and unreachable require different follow-up actions
- Recognise that ok silently absorbs changed and ignored counts
- Identify the two counters that report a real failure inside a zero-failure recap
Prerequisites
Verified against ansible-core 2.21.x · ansible (community package) 14.x · Python (controller) 3.12+ · ansible-lint 26.x · Molecule 26.x · Ubuntu 24.04 LTS · Debian 12 (Bookworm) · RHEL / Rocky / AlmaLinux 9.x · 2026-08-11
The PLAY RECAP looks like a scoreboard. It is not. It is seven
separate claims about seven different things, and only three of them
tell you what state a host is actually in.
You have already met the recap as a report format. This part uses it as an evidence source — the thing you read at 03:00 when a change went wrong across a fleet and someone needs to know what to do next. That reading requires a much more precise vocabulary than “220 passed, 80 did not”, because the eighty are not one population and treating them as one is how a partial failure becomes an outage.
The question each counter answers
Every counter answers one question: after this run, what do I know about this host?
| Counter | Did the module run on the host? | What you know about the host |
|---|---|---|
ok | Yes | It was already in the desired state, as far as that module can tell |
changed | Yes | It was not, and now it is |
skipped | No — a when was false | Untouched by design. State is old and known |
failed | It was attempted | The host may be half-changed. State is unknown in a specific place |
unreachable | No — nothing connected | Nothing was attempted. State is stale but consistent |
rescued | Yes, and it failed | A failure happened; a rescue block ran. State depends entirely on the rescue |
ignored | Yes, and it failed | A failure happened and you told Ansible to carry on |
Read down the third column. Three rows describe a host you can put in a sentence. Four describe a host you cannot, and each of those four needs a different response.
All seven in one run
$ ansible-playbook -i inv.ini seven.ymlTASK [This one is ok] **********************************************************
ok: [lh] => { "msg": "nothing changed here" }
TASK [This one reports changed] ************************************************
changed: [lh]
TASK [This one is skipped] *****************************************************
skipping: [lh]
TASK [This one fails and is ignored] *******************************************
fatal: [lh]: FAILED! => {"changed": false, "msg": "ignored failure"}
...ignoring
TASK [Inner failure] ***********************************************************
fatal: [lh]: FAILED! => {"changed": false, "msg": "rescued failure"}
TASK [Recovery] ****************************************************************
ok: [lh] => { "msg": "recovered" }
TASK [This one fails for real] *************************************************
fatal: [lh]: FAILED! => {"changed": false, "msg": "the real failure"}
PLAY RECAP *********************************************************************
lh : ok=4 changed=1 unreachable=0 failed=1 skipped=1 rescued=1 ignored=1
exit=2Six tasks were written. The recap sums to more than six, and that is the first thing worth understanding about it.
ok is not a category, it is a total
ok counts every task that finished without an unhandled failure. It
therefore includes the changed tasks and includes the ignored
failures. Both verified separately on 2.21.3:
$ ansible-playbook -i inv.ini okcount.yml ; ansible-playbook -i inv.ini ignonly.ymlPLAY RECAP *********************************************************************
lh : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
PLAY RECAP *********************************************************************
lh : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=1One changed task produced ok=1 changed=1. One failed and ignored
task also produced ok=1. So ok cannot be read as “number of tasks
that were fine”. A host with ok=40 ignored=6 had six failures in it.
This matters the moment anyone builds a dashboard or a chat notification out of recap numbers. The intuitive summary — “sum the ok column, that is our success rate” — counts ignored failures as successes, on every host, permanently.
failed versus unreachable: the distinction the rest of the part is built on
These two are the ones people merge, and merging them is the single most expensive mistake in fleet triage.
A failed host ran something. Ansible connected, pushed a module, the module executed on the managed node, and it returned a result saying the operation did not succeed. The host is somewhere between “nothing happened” and “the change is half applied”:
aptfailed to install a package — possibly after updating the package index, possibly after unpacking two of five packages.templatefailed validation — but the previous lesson in the play already restarted a service.commandreturned non-zero — after doing whatever the command does before its last line.
You cannot say what state that host is in without going and looking. What you can say is that something ran, so the change may be partially applied.
An unreachable host ran nothing. Ansible could not open a connection, so no module was ever transferred, so nothing executed. If the host was unreachable from the first task of the play, its configuration is exactly what it was before the run started.
That makes an unreachable host consistent — and stale. It is running last week’s configuration. It is not damaged. It is not half-anything. It is simply a host that did not get the change, and it will still be running the old configuration next Tuesday when something depends on the new one.
skipped is the good kind of untouched
A skipped task is the only “nothing happened” outcome that carries no
uncertainty at all. The when evaluated false, so Ansible deliberately
did not act. The host is in a known state: the state it was in, because
your conditional said it should be.
The reason to name this explicitly is that a skipped host and an unreachable host produce the same change on disk — none — but opposite confidence. Skipped is a decision. Unreachable is an accident.
There is one way to get this wrong, and it is common enough to name: a
when clause that references a fact. If facts were not gathered, or
were served from a stale cache, the conditional can skip a task on a
host that genuinely needed it. The recap will show skipped, which
reads as intentional, and nobody will look again.
- name: Install the agent on Debian-family hosts
ansible.builtin.apt:
name: monitoring-agent
state: present
when: ansible_facts['os_family'] == 'Debian'The defence is not to stop using conditionals. It is to make the
expected count explicit — if you believe 180 of 300 hosts are Debian
family, then skipped=120 is a pass and skipped=133 is a question.
rescued, and what a rescue actually promises
rescued counts a task that failed inside a block where a rescue
existed and ran. The recap then reports failed=0 for that host.
What the rescue promises is that your code ran after the failure. It promises nothing about the result. A rescue that logs the failure and moves on leaves the host exactly as broken as the failure left it, while converting a red recap into a green one.
The useful discipline is to ask, of every rescue block you write:
after this runs, can I describe the state of the host in one sentence?
A rescue that restores a backup and restarts the service can. A rescue
that prints a message cannot.
What to do with each state
This is the table to keep. The rest of the part elaborates every row.
| State | First action | Why |
|---|---|---|
ok / changed | Verify a sample against the intended outcome | Ansible reports what the module claimed, not what the service is doing |
skipped | Confirm the count matches your expectation | A wrong when looks identical to a right one |
failed | Investigate before rerunning. Do not retry blind | The host may be half-changed; a rerun may compound it |
unreachable | Add to a quarantine list. Establish why | It is stale, it will drift further, and it may be unreachable because of the change |
rescued | Read what the rescue did | The recap is green; the host may not be |
ignored | Read what you ignored, and why you thought it was safe | The ok count already absorbed it |
Knowledge check
Knowledge check · 4 questions
Q1. A patch run finishes and one host reports ok=40 changed=12 unreachable=0 failed=0 skipped=3 rescued=0 ignored=6. What actually happened on that host?
Q2. A host reported unreachable is in the same operational position as a host whose task was skipped by a when clause: neither received the change, so neither needs follow-up.
Q3. Which recap counters can be non-zero on a host where a task genuinely failed, while failed itself stays at zero? Select all that apply.
Q4. Why does the distinction between failed and unreachable change what you do next, rather than being a reporting detail?
Passing score: 75%. Answers are checked in this browser.