Verify the outcome, not the task result · Ansible

ok is a statement about a module, not about a host.

When ansible.builtin.systemd_service reports ok for state: started, it means the module asked systemd for the unit’s state, found it active, and had nothing to do. That is a real fact and it is narrower than the sentence people read it as. It does not mean the service is answering requests, that it loaded the configuration you rendered, that it is listening on the port you expect, or that it will still be running in thirty seconds.

The gap between “the module was satisfied” and “the thing works” is where verification lives, and it is the difference between a test suite and a re-run of the deployment.

The chain of things that can be true separately

Take a web server role. Here is what can be independently true:

The package is installed.
The configuration file exists with the content you rendered.
The configuration is valid — the daemon would accept it.
The unit is enabled.
The unit is active.
The process is listening on the expected port.
The endpoint answers.
The endpoint answers with the content this role is responsible for.

A converge that reports ok on every task establishes 1, 2, 4 and 5. It establishes nothing about 3, 6, 7 or 8 — and every real incident this role can cause lives in that second list.

Item 3 deserves special attention because it is the one that catches people. A template task reports changed when it writes new content. It has no opinion about whether that content is a valid configuration file. A rendered nginx.conf with a typo in a directive is written successfully, reported as changed, and refuses to load.

Verification probes state

The rule for writing a verify.yml, or any verification play, is simple: do not read the results of the tasks that made the change. Ask the host.

A verification that starts by registering the converge tasks and asserting on their changed values is asserting that Ansible did what Ansible said it did, which was never in doubt.

Read-only / Safemolecule/default/verify.yml — probing, not re-reading— Every task here asks the instance a question. Nothing reads a result produced during converge.

---
- name: Verify
hosts: molecule
gather_facts: true
tasks:
  - name: Collect service state from the host
    ansible.builtin.service_facts:

  - name: The unit is enabled and running
    ansible.builtin.assert:
      that:
        - "'nginx.service' in ansible_facts.services"
        - ansible_facts.services['nginx.service'].state == 'running'
        - ansible_facts.services['nginx.service'].status == 'enabled'
      fail_msg: 'nginx.service is not both enabled and running'
      success_msg: 'nginx.service is enabled and running'

  - name: The daemon accepts its own configuration
    ansible.builtin.command:
      cmd: nginx -t
    changed_when: false
    register: nginx_config_test

  - name: Something is listening on the expected port
    ansible.builtin.wait_for:
      port: 8080
      host: 127.0.0.1
      timeout: 10
      state: started

  - name: Fetch the health endpoint
    ansible.builtin.uri:
      url: http://127.0.0.1:8080/health
      return_content: true
    register: health

  - name: The health endpoint returns the body this role publishes
    ansible.builtin.assert:
      that:
        - "'ok' in health.content"
      fail_msg: >-
        /health answered with status {{ health.status }} and a body that
        does not contain "ok". Body was: {{ health.content | truncate(200) }}

Five probes, four different kinds, each answering a different question.

service_facts gathers the service manager’s own view. It is better than running systemctl is-active for two reasons: it works across service managers, and it gathers everything at once so several assertions cost one round trip.

nginx -t asks the daemon to parse its own configuration. This is the item-3 check, and no Ansible module can substitute for it — only the program knows what its configuration language means. changed_when: false is required, because command reports changed unconditionally and a verify play that reports changes will break the idempotence signal it sits next to.

wait_for with state: started waits for a TCP connection to succeed. It is a genuine probe of item 6, and its timeout is doing work: a service that takes eight seconds to bind is a service that passed this check and would have failed a naive one-shot connection test.

uri with return_content, followed by an assert on the body, is item 8. The status code is already covered — uri fails on anything other than 200 unless you tell it otherwise — so the assertion’s job is the content, which is what catches the case where something answered and it was not your service.

Keeping the assertion in its own task rather than folding it into failed_when is deliberate. failed_when replaces the module’s own failure determination rather than adding to it, so a failed_when that looks only at the body would also decide what counts as an HTTP failure, which is not what the author intended and is not obvious from reading it.

Where the probe runs is a real choice, and the default is often wrong

wait_for and uri are ordinary modules: they run on the target host, over the connection Ansible established. So url: http://127.0.0.1:8080/health in a play whose hosts: is the instance means “the instance asks itself”, which tests the service and not the network path to it.

That is usually what you want in a role test, and it is emphatically not what you want when the question is “can clients reach this?”. For that, delegate_to: localhost moves the probe to the controller, and the URL must then be the target’s real address rather than a loopback one.

Getting this backwards produces a check that passes in every environment and proves nothing about reachability — the same shape of mistake as using debug to test whether a host is up.

Assertions that diagnose

ansible.builtin.assert takes more than that:

Read-only / Safeansible-doc -s ansible.builtin.assert— Executed against ansible-core 2.21.3.

$ ansible-doc -s ansible.builtin.assert

- name: Asserts given expressions are true
assert:
    fail_msg:              # The customized message used for a failing
                           # assertion.
    quiet:                 # Set this to 'true' to avoid verbose output.
    success_msg:           # The customized message used for a successful
                           # assertion.
    that:                  # (required) A list of string expressions of the
                           # same form that can be passed to the 'when'
                           # statement.

The default failure message names the expression that failed, which is better than nothing and worse than a sentence. Compare:

Read-only / Safetwo assertions, same condition— Both assume service_facts has been gathered. The second one tells whoever reads the CI log what to do next, and that person is frequently not the author.

# Fails with: Assertion failed
- name: Check nginx
ansible.builtin.assert:
  that:
    - ansible_facts.services['nginx.service'].state == 'running'

# Fails with a sentence naming the expectation and what was found
- name: The web server unit is running
ansible.builtin.assert:
  that:
    - ansible_facts.services['nginx.service'].state == 'running'
  fail_msg: >-
    Expected nginx.service to be running; it is
    {{ ansible_facts.services['nginx.service'].state }}.
    If it failed to start, the usual cause is a rendered config the
    daemon rejects - run nginx -t on the host.

Writing fail_msg for every assertion is tedious and worth it for the ones that fail intermittently, because those are the ones somebody will read at an inconvenient hour.

quiet: true suppresses the per-expression output on success. In a verify play with twenty assertions this is the difference between a log you skim and a log you scroll.

The same idea outside Molecule

None of this is Molecule-specific. The same pattern belongs at the end of a production deployment play, where it is worth more:

Configuration changepost-deployment verification in a real play— Runs against production hosts after the change. Badged CONFIGURATION because it is part of a play that changes things; the verification tasks themselves are read-only.

- name: Verify the tier is serving after the change
hosts: web
gather_facts: false
tasks:
  - name: The service port accepts connections
    ansible.builtin.wait_for:
      port: 8080
      host: "{{ ansible_host }}"
      timeout: 30
      state: started
    delegate_to: localhost

  - name: The health endpoint answers
    ansible.builtin.uri:
      url: "http://{{ ansible_host }}:8080/health"
      return_content: true
    register: health
    delegate_to: localhost
    retries: 6
    delay: 5
    until:
      - health.status | default(0) == 200
      - "'ok' in (health.content | default(''))"

Both probes are delegated to the controller here, because the question in production is whether the load balancer’s view of the host is healthy, not whether the host can reach itself. retries and until give the service time to come up without turning the check into a fixed pause.

This is also what makes a rolling deployment safe: with serial, a verification play that fails stops the rollout at the current batch instead of continuing through the fleet. The rolling-update part builds on exactly this.

Under the hood

changed and ok are fields in the JSON a module returns. Nothing inspects the host to corroborate them.

For a module like copy, ok is a genuinely strong claim: the module checksummed the destination, compared it to the source, and found them equal. For systemd_service with state: started, ok means systemd reported the unit active at the moment it was asked. For command, the result fields are whatever changed_when and failed_when evaluated to, which is to say whatever you told it.

So the strength of an ok varies per module, and there is nothing in the recap that distinguishes a strong one from a weak one. Forty-two ok results are forty-two module-specific claims of forty-two different strengths, and the recap presents them identically.

Outcome verification sidesteps the whole question. A TCP connection that succeeds, an HTTP body containing the expected string, a configuration parser that exits zero — these are facts about the host that do not depend on trusting any module’s self-report. That is why a small number of outcome probes is worth more than a large number of task-result assertions.

What happens if the verification is written by the person who wrote the role, immediately after writing it?

It tends to assert the things the role does, in the order the role does them, and to miss the thing the role forgot.

That is not a criticism of the author; it is a property of writing both halves from the same mental model. The role’s author knows it renders site.conf, so the verification checks that site.conf was rendered. Nobody checks whether the daemon was reloaded afterwards, because the mental model contains a reload and the code may not.

Two cheap correctives. Write the verification from the requirement — “this host serves the health endpoint on 8080” — rather than from the tasks. And use side_effect in the Molecule sequence: stop the service, delete the config, then converge and verify. A verification that only passes on a freshly built instance is not testing convergence, it is testing a fresh build.

What a passing verification still does not prove

That it works under load. One request answered says nothing about a thousand concurrent ones.
That it works on production data. The health endpoint answers on an empty database too.
That it survives a restart. Unless you restarted it and checked again — which is what side_effect is for, and what a container cannot do for anything involving the boot path.
That the configuration is correct, only that it is valid. nginx -t accepts a syntactically perfect configuration that proxies to the wrong upstream.
That the hosts you did not target are fine. Verification runs against the hosts in the play.

Those five sentences are the agenda for the remaining two lessons.

The chain of things that can be true separately

Verification probes state

Assertions that diagnose

The same idea outside Molecule

What a passing verification still does not prove

Knowledge check