Break/Fix: 200 hosts reported a successful agent rollout and none of them has the agent · Break/Fix · Ansible

Reported symptoms

A security agent was rolled out to 200 hosts. The rollout run was green: 200 hosts, failed=0, no errors, change record closed.

Three weeks later an audit queries the agent management console and finds 200 hosts missing. Not misconfigured, not stale - absent. The agent has never been installed on any of them.

Everything about the automation appears correct:

The playbook names the right package.
Running the install task on its own, by hand, produces a clear failure that explains itself immediately.
The nightly converge has run every night since and reported success every time.
No dashboard, alert or pipeline has ever flagged anything.

The only oddity, once somebody goes back to the log, is that the install task reports ok rather than changed - which on an idempotent task is what you would expect on a second run and is entirely unremarkable on a first.

Evidence provided

Read-only / Safethe failure is in the log, followed by the word that ends the story

$ grep -A3 'Install the security agent' logs/rollout.log | head -6

TASK [Install the security agent] **********************************************
fatal: [app001]: FAILED! => {"changed": false, "msg": "Failed to update apt cache: unauthorized"}
...ignoring

Read-only / Safefailed=0, and ignored=1 in a column nobody reads

$ tail -3 logs/rollout.log

PLAY RECAP *********************************************************************
app001                     : ok=12   changed=2    unreachable=0    failed=0    skipped=1    rescued=0    ignored=1
app002                     : ok=12   changed=2    unreachable=0    failed=0    skipped=1    rescued=0    ignored=1

Read-only / Safeone line

$ grep -n -A6 'Install the security agent' roles/security/tasks/main.yml

21:- name: Install the security agent
22:  ansible.builtin.package:
23:    name: vendor-agent
24:    state: present
25:  ignore_errors: true

Read-only / Safeeight months ago, for a reason that lasted one afternoon

$ git log -1 --format='%h %ad %s' --date=short -L21,25:roles/security/tasks/main.yml | head -2

b4419c7 2025-12-14 security: tolerate a flaky host during the December change

Read-only / Safethe actual problem, and it is not an Ansible problem

$ ansible app001 -i inventory -b -m ansible.builtin.command -a 'apt-get update' | tail -2

E: Failed to fetch https://repo.example.com/vendor/dists/stable/InRelease  401  Unauthorized
E: Some index files failed to download.

Read-only / Safenothing in the role ever checks that the agent exists

$ grep -c 'assert\|command:.*vendor-agent\|service_facts' roles/security/tasks/main.yml

Work the evidence before reading on

The failure is in the log. It has been in the log every night for three weeks. The run still ends failed=0.

Read the three lines of the task output in order. What is the third one, and what does it do to the second?
Look at the play recap columns. Which one is non-zero, and where in the line is it?
The role installs something. What does it do afterwards to establish that the thing is there?

Before continuing: what did the commit eight months ago intend to tolerate, and what does the line actually tolerate?

Root cause

1. `ignore_errors` discards every error, not the one you had in mind

ignore_errors: true means the task cannot fail the play. Whatever goes wrong - a transient network blip, an expired credential, a missing package, a full disk, a repository that no longer exists - the result is downgraded to ignored and the play continues.

The commit that added it was addressing a real problem: one host failed transiently during an unrelated change in December, and the line made the run go green that afternoon. It was never removed, because nothing ever went wrong afterwards - or rather, nothing ever appeared to.

When the vendor repository credential expired three weeks ago, the task started failing on every host. The failure is not transient, it is not host-specific, and it is exactly the kind of thing a rollout must stop for. It was ignored, along with everything else.

2. The signals exist and nothing consumes them

Ansible does report this. The task output ends with ...ignoring, and the play recap carries an ignored=1 column.

Both are easy to miss for structural reasons rather than careless ones. The ...ignoring line is one line among hundreds and contains no word that anybody greps for. The ignored column is the last field of a recap line that begins with the number everybody looks at, which is failed=0.

Pipelines consume the exit code, which is 0. Dashboards count failed hosts, which is 0. The one number that is not zero is in neither.

3. Nothing asserted the outcome

The role installs the agent and then moves on. There is no task afterwards that checks whether the package is present, whether the service is running, or whether the agent has registered with its console.

Without that, the play asserts only that Ansible executed. The question the rollout actually needed answered - is the agent on this host - was never asked by anything until an auditor asked it three weeks later.

Under the hood

Ansible offers four distinct ways to handle a task that might fail, and they differ in precision rather than in strength.

ignore_errors: true accepts every failure. It is the blunt instrument and it is almost never what anybody means.

failed_when redefines what failure means for that task, so a specific outcome is tolerated and every other outcome still fails:

- name: Install the security agent
  ansible.builtin.package:
    name: vendor-agent
    state: present
  register: agent_install
  failed_when:
    - agent_install is failed
    - "'already the newest version' not in agent_install.msg | default('')"

retries with until handles genuine transience, which is what the December commit was actually trying to address - a bounded number of attempts, with a delay, and a real failure at the end if none succeeds.

block with rescue handles a failure by doing something about it, which is the right shape when there is a fallback or a cleanup, and it records the outcome in the rescued column rather than hiding it.

Only the first of the four discards information. The other three all leave the play able to fail, which is the property that matters.

One more detail is worth knowing: ignore_errors does not suppress unreachable hosts. A host that cannot be connected to is still reported as unreachable and still counts, which is ignore_unreachable and a separate setting. That is a small mercy and it is the reason this failure mode does not also hide connectivity problems.

A line added to make a run go green outlives its reason

The December commit is entirely sympathetic. A host was flaky, a change window was closing, and one line made the run finish. Everybody has written that line.

What makes it dangerous is that it has no expiry and leaves no reminder. The flaky host was fixed within a day. The line stayed for eight months and then converted a fleet-wide failure into a fleet-wide success report.

Two habits limit the damage. First, when a suppression is added under time pressure, add the ticket to remove it in the same commit - and write in the comment exactly which error is being tolerated, so a future reader can tell whether it still applies. Second, prefer the precise tool: retries for transience, failed_when for a named acceptable outcome. Both address the actual problem and neither hides anything else.

The same reasoning applies to every other suppression in a repository: changed_when: false on a task that does change things, no_log used to quieten noisy output rather than to protect a secret, a when: added to skip a host that was failing. Each of them was a reasonable five-minute decision and each of them is load-bearing forever.

Resolution

Raise the exposure first. 200 hosts have been without a security agent for three weeks while the compliance record said otherwise; that belongs with whoever owns the control, not in an Ansible ticket.
Fix the underlying failure. The repository credential has expired, which is not an Ansible problem and would have been obvious on day one if the task had been allowed to fail.
Remove ignore_errors: true from the task. If a transient failure genuinely needs tolerating, express it with bounded retries or with a failed_when naming the specific condition.
Add a verification task after the install that asserts the package is present and the service is running, so a future silent failure fails the run.
Roll out to a canary host and confirm both the install and the verification behave, before the fleet.
Deploy to the fleet and confirm from the agent management console that all 200 hosts have registered, rather than from the Ansible run.
Audit the repository for every other ignore_errors, and for the related suppressions - changed_when: false on tasks that change things, and conditions added to skip failing hosts.
Add a CI rule that any new ignore_errors must carry a comment naming the specific error being tolerated, so the next December commit has to explain itself.

Verification

The verification task can fail. Remove the agent from a scratch host, break the repository credential, and run the play: it must report a failure. This is the check that would have caught the incident on day one and it had never existed.
The play fails when the install fails. With ignore_errors removed, confirm on that same scratch host that the run stops rather than continuing.
The agent is present on every host. Query the package and the service state across the fleet, and confirm the count is 200.
The agents are reporting in. Confirm from the management console, which is independent of both the hosts and Ansible, and is the check the auditor performed.
The recap is clean. ignored=0 across the fleet, and a grep of the run log for ignoring returns nothing.
The fleet is converged. Run the play once more and require changed=0, so the state is stable rather than newly repaired.
The audit of other suppressions is complete, with each remaining one either removed or annotated with the specific error it tolerates and the reason it still applies.

Prevention

Do not use ignore_errors: true on a task whose outcome matters. It trades every future failure for one past one.
Use the precise tool instead. Bounded retries with until for transience; failed_when for a named acceptable outcome; block and rescue when there is something to do about the failure.
Assert the end state after anything that installs, configures or starts something:

- name: Confirm the security agent is installed and running
  ansible.builtin.assert:
    that:
      - ansible_facts.packages['vendor-agent'] is defined
      - ansible_facts.services['vendor-agent.service'].state == 'running'
    fail_msg: "The security agent is not installed and running on {{ inventory_hostname }}."

Alert on ignored and rescued in the recap, not only on failed. They are the columns that record failures the play decided to survive.
Fail CI on a new ignore_errors without a comment naming the specific error and the reason it is acceptable. The comment is what makes the decision reviewable later.
Pair every suppression added under time pressure with a ticket to remove it, in the same commit. The urgency ends the same day; the line does not.
Verify rollouts at an independent system where one exists. The agent console knew the truth for three weeks and nobody was comparing it against the automation.

What if the task had used failed_when or retries instead?

Both would have stopped this on day one, and they would have stopped it differently, which is worth seeing.

With bounded retries, the task attempts the install several times with a delay. A genuinely flaky host - the December problem - succeeds on the second or third attempt and the run is green for the right reason. An expired credential fails all the attempts and then fails the task, so the rollout stops on the first host and somebody reads a 401 within minutes.

With failed_when naming the acceptable condition, the specific outcome somebody decided to tolerate is tolerated and everything else fails. Writing it forces the question the blanket suppression avoids: which error, exactly, is acceptable here? An unauthorised repository is plainly not on that list, and putting the list in the file is what makes that obvious.

The worst variant is worth naming too. ignore_errors: true on a task followed by a task that depends on its result produces a cascade: the first task fails silently, the second uses a registered result that does not contain what it expects, and the error surfaces several tasks later in a form that has nothing to do with the cause. That is the version that takes a day to diagnose rather than three weeks to notice, and it comes from the same line.

Why this matters

Three weeks of 200 unmonitored hosts, a closed change record, a green pipeline every night, and the failure printed in full in every log the whole time.

The specific lesson is about ignore_errors, and it is simple: suppressing errors is not the same as handling them, and a blanket suppression trades every future failure for one that already happened. The more useful lesson is about what a green run means. Ansible reports that it executed. It does not report that the outcome you wanted exists, and it cannot, unless a task asks.

So the discipline is to end every play that installs, configures or starts something with an assertion about the result - and to check that assertion against something outside the automation where one is available. A console, a scan, a client request. In this incident such a check existed, and it was operated by an auditor, three weeks late.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. ignore_errors discards every error, not the one you had in mind

2. The signals exist and nothing consumes them

3. Nothing asserted the outcome

Resolution

Verification

Prevention

1. `ignore_errors` discards every error, not the one you had in mind