Skip to main content
RunBook Academy

AnsibleXLIV · Failure Modes and Partial Fleet FailureFailure Modes and Partial Fleet Failure

The seven things that can happen to a host

Advanced⏱ ~22 minansible-playbook

What you'll learn

  • State what each of the seven recap counters proves about a host actual state
  • Explain why failed and unreachable require different follow-up actions
  • Recognise that ok silently absorbs changed and ignored counts
  • Identify the two counters that report a real failure inside a zero-failure recap

Prerequisites

Verified against ansible-core 2.21.x · ansible (community package) 14.x · Python (controller) 3.12+ · ansible-lint 26.x · Molecule 26.x · Ubuntu 24.04 LTS · Debian 12 (Bookworm) · RHEL / Rocky / AlmaLinux 9.x · 2026-08-11

Not yet marked complete on this device.

The PLAY RECAP looks like a scoreboard. It is not. It is seven separate claims about seven different things, and only three of them tell you what state a host is actually in.

You have already met the recap as a report format. This part uses it as an evidence source — the thing you read at 03:00 when a change went wrong across a fleet and someone needs to know what to do next. That reading requires a much more precise vocabulary than “220 passed, 80 did not”, because the eighty are not one population and treating them as one is how a partial failure becomes an outage.

The question each counter answers

Every counter answers one question: after this run, what do I know about this host?

CounterDid the module run on the host?What you know about the host
okYesIt was already in the desired state, as far as that module can tell
changedYesIt was not, and now it is
skippedNo — a when was falseUntouched by design. State is old and known
failedIt was attemptedThe host may be half-changed. State is unknown in a specific place
unreachableNo — nothing connectedNothing was attempted. State is stale but consistent
rescuedYes, and it failedA failure happened; a rescue block ran. State depends entirely on the rescue
ignoredYes, and it failedA failure happened and you told Ansible to carry on

Read down the third column. Three rows describe a host you can put in a sentence. Four describe a host you cannot, and each of those four needs a different response.

All seven in one run

Read-only / Safeone play, one host, seven outcomes — executed on ansible-core 2.21.3
$ ansible-playbook -i inv.ini seven.yml
TASK [This one is ok] **********************************************************
ok: [lh] => { "msg": "nothing changed here" }

TASK [This one reports changed] ************************************************
changed: [lh]

TASK [This one is skipped] *****************************************************
skipping: [lh]

TASK [This one fails and is ignored] *******************************************
fatal: [lh]: FAILED! => {"changed": false, "msg": "ignored failure"}
...ignoring

TASK [Inner failure] ***********************************************************
fatal: [lh]: FAILED! => {"changed": false, "msg": "rescued failure"}

TASK [Recovery] ****************************************************************
ok: [lh] => { "msg": "recovered" }

TASK [This one fails for real] *************************************************
fatal: [lh]: FAILED! => {"changed": false, "msg": "the real failure"}

PLAY RECAP *********************************************************************
lh    : ok=4  changed=1  unreachable=0  failed=1  skipped=1  rescued=1  ignored=1

exit=2

Six tasks were written. The recap sums to more than six, and that is the first thing worth understanding about it.

ok is not a category, it is a total

ok counts every task that finished without an unhandled failure. It therefore includes the changed tasks and includes the ignored failures. Both verified separately on 2.21.3:

Read-only / Safetwo minimal plays, each with exactly one task — executed on 2.21.3
$ ansible-playbook -i inv.ini okcount.yml ; ansible-playbook -i inv.ini ignonly.yml
PLAY RECAP *********************************************************************
lh    : ok=1  changed=1  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0

PLAY RECAP *********************************************************************
lh    : ok=1  changed=0  unreachable=0  failed=0  skipped=0  rescued=0  ignored=1

One changed task produced ok=1 changed=1. One failed and ignored task also produced ok=1. So ok cannot be read as “number of tasks that were fine”. A host with ok=40 ignored=6 had six failures in it.

This matters the moment anyone builds a dashboard or a chat notification out of recap numbers. The intuitive summary — “sum the ok column, that is our success rate” — counts ignored failures as successes, on every host, permanently.

failed versus unreachable: the distinction the rest of the part is built on

These two are the ones people merge, and merging them is the single most expensive mistake in fleet triage.

A failed host ran something. Ansible connected, pushed a module, the module executed on the managed node, and it returned a result saying the operation did not succeed. The host is somewhere between “nothing happened” and “the change is half applied”:

  • apt failed to install a package — possibly after updating the package index, possibly after unpacking two of five packages.
  • template failed validation — but the previous lesson in the play already restarted a service.
  • command returned non-zero — after doing whatever the command does before its last line.

You cannot say what state that host is in without going and looking. What you can say is that something ran, so the change may be partially applied.

An unreachable host ran nothing. Ansible could not open a connection, so no module was ever transferred, so nothing executed. If the host was unreachable from the first task of the play, its configuration is exactly what it was before the run started.

That makes an unreachable host consistent — and stale. It is running last week’s configuration. It is not damaged. It is not half-anything. It is simply a host that did not get the change, and it will still be running the old configuration next Tuesday when something depends on the new one.

skipped is the good kind of untouched

A skipped task is the only “nothing happened” outcome that carries no uncertainty at all. The when evaluated false, so Ansible deliberately did not act. The host is in a known state: the state it was in, because your conditional said it should be.

The reason to name this explicitly is that a skipped host and an unreachable host produce the same change on disk — none — but opposite confidence. Skipped is a decision. Unreachable is an accident.

There is one way to get this wrong, and it is common enough to name: a when clause that references a fact. If facts were not gathered, or were served from a stale cache, the conditional can skip a task on a host that genuinely needed it. The recap will show skipped, which reads as intentional, and nobody will look again.

Read-only / Safea skip that hides a targeting bug
- name: Install the agent on Debian-family hosts
ansible.builtin.apt:
  name: monitoring-agent
  state: present
when: ansible_facts['os_family'] == 'Debian'

The defence is not to stop using conditionals. It is to make the expected count explicit — if you believe 180 of 300 hosts are Debian family, then skipped=120 is a pass and skipped=133 is a question.

rescued, and what a rescue actually promises

rescued counts a task that failed inside a block where a rescue existed and ran. The recap then reports failed=0 for that host.

What the rescue promises is that your code ran after the failure. It promises nothing about the result. A rescue that logs the failure and moves on leaves the host exactly as broken as the failure left it, while converting a red recap into a green one.

The useful discipline is to ask, of every rescue block you write: after this runs, can I describe the state of the host in one sentence? A rescue that restores a backup and restarts the service can. A rescue that prints a message cannot.

What to do with each state

This is the table to keep. The rest of the part elaborates every row.

StateFirst actionWhy
ok / changedVerify a sample against the intended outcomeAnsible reports what the module claimed, not what the service is doing
skippedConfirm the count matches your expectationA wrong when looks identical to a right one
failedInvestigate before rerunning. Do not retry blindThe host may be half-changed; a rerun may compound it
unreachableAdd to a quarantine list. Establish whyIt is stale, it will drift further, and it may be unreachable because of the change
rescuedRead what the rescue didThe recap is green; the host may not be
ignoredRead what you ignored, and why you thought it was safeThe ok count already absorbed it

Knowledge check

Knowledge check · 4 questions

  1. Q1. A patch run finishes and one host reports ok=40 changed=12 unreachable=0 failed=0 skipped=3 rescued=0 ignored=6. What actually happened on that host?

  2. Q2. A host reported unreachable is in the same operational position as a host whose task was skipped by a when clause: neither received the change, so neither needs follow-up.

  3. Q3. Which recap counters can be non-zero on a host where a task genuinely failed, while failed itself stays at zero? Select all that apply.

  4. Q4. Why does the distinction between failed and unreachable change what you do next, rather than being a reporting detail?

Passing score: 75%. Answers are checked in this browser.