Validation plays that fail loudly · Ansible

The gap this course keeps returning to:

template succeeded. systemctl restart succeeded. The HTTP health check fails. The play reported success.

Every task did what it was asked. The file was written, the unit was restarted, systemd reports the service as active, and the application returns 500 because the new config has a typo in the database host. Nothing in the play noticed, because nothing in the play asked.

A validation play is what asks. It runs after a change, it tests operational health rather than task outcomes, and its failure is a change failure.

The six checks

Almost every validation play is some subset of these, and the value is in running the whole subset rather than the first one.

Check	Mechanism	What it catches that the previous one does not
Package installed at the expected version	`package_facts`	A deploy that did not deploy
Config file correct	`slurp` or `stat` plus an assertion	A template that rendered wrong
Service running	`service_facts`	A unit that failed after starting
Port listening	`wait_for`	A service that is running but not bound
Endpoint healthy	`uri`	A process that is bound but broken
Certificate valid and not near expiry	`x509_certificate_info`	A working service about to stop working

The ordering is deliberate: each row is closer to what a user experiences, and each row can pass while the one below it fails.

Read-only / Safevalidate.yml— Read-only throughout. Run after every change, and on a schedule. Nothing here writes to a target.

- name: Validate the application after a change
hosts: appservers
gather_facts: true
tasks:
  - name: Collect package and service state
    ansible.builtin.package_facts:
      manager: auto
    check_mode: false

  - name: Collect service state
    ansible.builtin.service_facts:
    check_mode: false

  - name: The expected version is installed
    ansible.builtin.assert:
      that:
        - "'acme-app' in ansible_facts.packages"
        - ansible_facts.packages['acme-app']
          | map(attribute='version') | first is version(expected_version, '==')
      fail_msg: >-
        {{ inventory_hostname }} has
        {{ ansible_facts.packages['acme-app'] | default([]) | map(attribute='version') | join(',') }},
        expected {{ expected_version }}
      quiet: true

  - name: The service is running and enabled
    ansible.builtin.assert:
      that:
        - ansible_facts.services['acme-app.service'].state == 'running'
        - ansible_facts.services['acme-app.service'].status == 'enabled'
      fail_msg: "acme-app is not running and enabled on {{ inventory_hostname }}"
      quiet: true

  - name: The service is listening
    ansible.builtin.wait_for:
      port: 8080
      host: 127.0.0.1
      state: started
      timeout: 30

  - name: The endpoint is healthy
    ansible.builtin.uri:
      url: "http://127.0.0.1:8080/healthz"
      status_code: 200
      return_content: true
      timeout: 10
    register: health

  - name: The endpoint reports the version we deployed
    ansible.builtin.assert:
      that:
        - health.json.version == expected_version
      fail_msg: >-
        Endpoint reports {{ health.json.version | default('no version') }},
        expected {{ expected_version }}
      quiet: true

service_facts entries are not uniform, and the key is not the service name you typed

Verified by execution on ansible-core 2.21.3. Entries carry a source, and the shape depends on it:

A systemd entry is keyed with its unit suffix — apparmor.service — and carries name, source, state and status.
A sysv entry is keyed without a suffix — apparmor — and has no status key at all. On the host used for this lesson, 23 sysv entries lacked it.

Two consequences for a validation play. An assertion on ansible_facts.services['app'].status raises an undefined error on any host where the service is reported through sysv, and the failure looks like a broken play rather than a finding. And the same service can appear twice under two keys, so a lookup by the bare name may find the sysv entry while the unit you manage is the systemd one.

Key on the full unit name, and guard the status test with | default('unknown') if the estate is mixed.

The last assertion is the one that closes the gap in the opening sentence. The service is running, it is bound, it returns 200 — and it is running the previous release because the restart raced the symlink swap. Only a check that compares what the endpoint reports against what was deployed catches that.

`assert`, `fail`, `failed_when`

Three mechanisms, and choosing correctly is mostly about who reads the message.

Use	When	What it gives you
`assert`	Testing a condition that should hold	`that`, `fail_msg`, `success_msg`, `quiet`
`fail`	Ending deliberately, usually inside a `when`	A single message, full control of the condition
`failed_when`	Reinterpreting another task’s result	Turns a task’s success into a failure, or the reverse

assert is the default for validation because fail_msg puts the diagnosis in the failure. quiet: true suppresses the per-assertion success output, which on a 300-host run is the difference between a readable result and forty screens.

The `failed_when` trap

This is the defect the curriculum flags for this part, and it is worth seeing executed because the reasoning that produces it is entirely sensible.

A compliance author wants a task to fail if any of several conditions indicates a problem, and writes them as a list — the way assert takes a list in that, where a list means “all of these must hold”.

Read-only / Safea list given to failed_when is joined with implicit and— Executed on ansible-core 2.21.3. Probe A has one true condition and one false; probe B has two true. Only B fails.

$ ansible-playbook failed-when.yml

TASK [Probe A - one condition true, one false] *********************************
ok: [localhost] => {
  "msg": "compliance probe"
}

TASK [Probe B - both conditions true] ******************************************
[ERROR]: Task failed: Action failed: A 'failed_when' expression evaluated to 'True'.
fatal: [localhost]: FAILED! => {
  "msg": "compliance probe"
}
...ignoring

TASK [Report] ******************************************************************
ok: [localhost] => {
  "msg": "A.failed=False B.failed=True"
}

PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=1

A.failed=False. One of its two conditions was true and the task passed.

Now put that in a compliance role:

# This check can essentially never fire.
- name: Assess the host against the baseline
  ansible.builtin.command:
    argv: [/usr/local/bin/assess-baseline]
  register: assessment
  changed_when: false
  failed_when:
    - assessment.rc != 0
    - "'FAIL' in assessment.stdout"
    - assessment.stdout_lines | length > 3
    - "'CRITICAL' in assessment.stdout"

The author’s intent was “fail if any of these indicates a problem”. The semantics are “fail only if all four are simultaneously true”. Add a fifth condition and it becomes even less likely to fire. Every host reports compliant.

The fix is to write the disjunction explicitly:

Read-only / Safean or, written as an or— A single expression. The or is visible in the code, so a reviewer reads what the play will do.

- name: Assess the host against the baseline
ansible.builtin.command:
  argv: [/usr/local/bin/assess-baseline]
register: assessment
changed_when: false
check_mode: false
failed_when: >-
  assessment.rc != 0
  or 'FAIL' in assessment.stdout
  or 'CRITICAL' in assessment.stdout

There is a stronger version of the same advice: prefer assert for compliance checks and reserve failed_when for reinterpreting a task’s result. assert takes a list in that where “all must hold” is the natural reading, so writing each requirement as a positive assertion puts the semantics and the intent in the same direction.

- name: The baseline assessment passed
  ansible.builtin.assert:
    that:
      - assessment.rc == 0
      - "'FAIL' not in assessment.stdout"
      - "'CRITICAL' not in assessment.stdout"
    fail_msg: "Baseline assessment failed on {{ inventory_hostname }}"
    quiet: true

Same three requirements, expressed as things that must be true rather than things that must all be wrong, and now the conjunction is what the author wants.

Why a list means and, in both keywords, and why one reads wrong

The behaviour is consistent and it is the framing that differs.

Ansible evaluates a list of conditions as a conjunction wherever it accepts one: when, failed_when, changed_when, assert’s that. That is a single rule and it is the right rule — when with a list of conditions meaning “run this if all of these hold” is exactly what anybody wants.

The mismatch is semantic, not mechanical:

when: [a, b] — run if all hold. Conjunction is intuitive.
that: [a, b] — all of these must be true. Conjunction is stated.
failed_when: [a, b] — fail if all hold. But the author was thinking about failure symptoms, and symptoms are naturally disjunctive. Any one of them means trouble.

The keyword is a predicate over failure, and failure conditions are the one place where a human enumerating a list means “or”. Nothing about failed_when signals that, and the list form looks identical to the assert form directly above it in the same file.

Two properties make the defect durable once written. It never errors, so nothing surfaces it. And its symptom is that everything passes, so nobody investigates. A failed_when list is worth grepping for across an existing compliance repository:

grep -rn -A3 'failed_when:$' roles/

Any hit where the following lines are a YAML list of failure symptoms is a check that fires only when every symptom is present at once.

What happens if validation fails on 3 hosts out of 300?

The rule is that a validation failure is a change failure. Applying it to three hosts out of three hundred needs more than the rule.

What must not happen is the default: note it, finish the window, and put it in a follow-up ticket. Three hosts running a half-applied change are the partial population from Part XLVIII, and they are the hosts that behave strangely for the next three weeks.

The response depends on when the failure arrives, which is the argument for validating between batches rather than at the end:

In the canary or an early batch, three failures out of the batch is an abort. The change is wrong or the fleet is not what the change assumed, and both answers say stop.
In a late batch, with earlier batches validating clean, the three hosts are more likely to be individually different. Stop the rollout, and either roll those three back or investigate them while the rest hold at their current state.

Either way, three things are non-negotiable: the rollout does not continue past unexplained validation failures; the three hosts are named in the report with the assertion that failed; and they end the window in a known state, which usually means rolled back rather than left mid-change.

The tempting fourth option — re-running the change on those three to see if it works this time — is occasionally right and should be a decision rather than a reflex. A transient failure re-runs clean. A real one re-runs identically, and you have spent the window’s remaining time finding that out.

Knowledge check

Why this matters

Validation is the part of a change that is cut when the window runs late, and it is the part whose absence is invisible until much later.

Skipping the validation play saves four minutes. What it costs is that the change is recorded as successful, so when the problem surfaces on Thursday nobody connects it to Tuesday’s window — the change worked, it says so in the record. The investigation starts from the wrong week.

The failed_when trap is the same failure with a different cause. The validation play ran, took its four minutes, and reported success because the check was constructed so that it could not report anything else. That is worse than skipping it, because now there is positive evidence for a conclusion nobody tested.

Both come down to one habit worth keeping: for any check you rely on, be able to say what would make it fail. If you cannot describe the failing case, you have not verified that the check can fail — and a check that cannot fail is indistinguishable from no check at all, except that it produces a green result.

The six checks

assert, fail, failed_when

The failed_when trap

Knowledge check

`assert`, `fail`, `failed_when`

The `failed_when` trap