The second run is a test

Run the play. Run it again. The second run must report changed=0.

That is the whole test. It takes no test framework, no assertion library, and no test cases — the play is the test — and in most repositories it finds more genuine defects on its first execution than any suite anyone writes afterwards.

Part XII established why an accurate changed=0 is a trustworthy signal about an estate. This lesson is about turning that property into a gate that runs before code merges, and about what to do with the results, which are usually uncomfortable the first time.

What the second run is actually asserting

An Ansible task declares a desired state. If the module is honest, the first run reaches that state and reports changed, and every subsequent run finds the state already correct and reports ok.

A second run that still reports changed therefore says one of three things, and they need different fixes:

1. The task does not converge. It changes something back and forth, or it writes a file whose content differs on every render — a template containing a timestamp, ansible_date_time, or a randomly ordered dictionary. This is a real bug: the host is being modified on every run forever, and if a handler is attached, the service restarts forever too.

2. The task cannot detect its own outcome. command, shell, script and raw have no idea whether they changed anything, so they report changed unconditionally. The fix is changed_when, or better, replacing the task with a module that knows.

3. The task’s outcome legitimately differs every time. state: latest against a mirror that publishes daily. A vendor CLI that rewrites its own config file on invocation. These exist, and they are the minority — treating every case as this one is how a repository stops having an idempotency property at all.

The diagnosis order matters. Assume 1, check for 2, and only conclude 3 when you can write down the mechanism.

The gate

The crude version is a shell script, and it is worth having on day one:

set -euo pipefail

INVENTORY=inventory.test.ini

ansible-playbook -i "$INVENTORY" site.yml

ansible-playbook -i "$INVENTORY" site.yml | tee second-run.log

if grep -qE 'changed=[1-9]' second-run.log; then
echo "FAIL: the second run reported changes; the play is not idempotent"
exit 1
fi

echo "OK: second run reported no changes"

Three properties of that script are load-bearing, and one of them is a weakness.

set -euo pipefail means a failed first run fails the job rather than proceeding to a second run that fails confusingly.

The pattern changed=[1-9] matches changed=1 through changed=9 and the leading digit of anything larger, and does not match changed=0.

The weakness is that it parses human-readable text. The recap is produced by the default stdout callback plugin, and that plugin’s output format is a presentation choice rather than an interface. It also tells you a host changed without telling you which task changed — which is exactly the thing you need next.

The version that tells you what changed

The obvious idea — set a JSON stdout callback and parse it — does not work on ansible-core 2.21. There is no json stdout callback: the callbacks that ship with ansible-core are default, junit, minimal, oneline and tree. The default callback does have a result_format option with json and yaml choices, and its own documentation is unusually direct about why that is not what you want:

These formats do not cause the callback to emit valid JSON or YAML formats. The output contains these formats interspersed with other non-machine parsable data.

The mechanism that is built for this is the junit callback, which has an option named for exactly this test:

$ ansible-doc -t callback ansible.builtin.junit

   fail_on_change  Consider any tasks reporting "changed" as a junit
                 test failure
      set_via:
        env:
        - name: JUNIT_FAIL_ON_CHANGE
      default: false
      name: JUnit fail on change

 output_dir  Directory to write XML files to.
      set_via:
        env:
        - name: JUNIT_OUTPUT_DIR
      default: ~/.ansible.log
      name: JUnit output dir

junit is an aggregate callback rather than a stdout callback, so it is enabled alongside the normal screen output and writes an XML file per play:

set -euo pipefail

INVENTORY=inventory.test.ini

ansible-playbook -i "$INVENTORY" site.yml

ANSIBLE_CALLBACKS_ENABLED=ansible.builtin.junit \
JUNIT_OUTPUT_DIR=./junit \
JUNIT_FAIL_ON_CHANGE=true \
ansible-playbook -i "$INVENTORY" site.yml

Here is the XML that produces, from a two-task play where the first task reported changed and the second did not:

$ cat junit/second-run.xml

<?xml version="1.0" ?>
<testsuites disabled="0" errors="0" failures="1" tests="2" time="0.0139">
<testsuite disabled="0" errors="0" failures="1" name="second-run" skipped="0" tests="2" time="0.0139">
	<testcase classname="second-run.yml:7" name="[localhost] Simulated second run: Render the site configuration" time="0.0074">
		<failure message="stand-in for a template task" type="failure">{
  "changed": true,
  "changed_when_result": true
}</failure>
	</testcase>
	<testcase classname="second-run.yml:12" name="[localhost] Simulated second run: Ensure the service is enabled" time="0.0065">
		<system-out>{
  "changed": false,
  "changed_when_result": false
}</system-out>
	</testcase>
</testsuite>
</testsuites>

Every changed task becomes a named failing test with its host, its playbook line number and its result payload. That turns a red build into a diagnosis rather than an invitation to reproduce it by hand.

Molecule already has this

Molecule’s default test sequence includes an idempotence action, and it is one of the reasons Molecule is worth the setup cost. The action’s own help text, from Molecule 26.6.0:

$ molecule idempotence --help

Usage: molecule idempotence [OPTIONS] [ANSIBLE_ARGS]...

Use the provisioner to configure the instances.

After parse the output to determine idempotence.

Molecule re-runs the converge playbook and parses the result; the default test_sequence places idempotence immediately after converge, so a role that is not idempotent fails the scenario without anyone writing an assertion. There is no separate playbook to author for this step.

If you already run Molecule, you already have this gate, and the shell script above is for repositories that are not there yet. Lesson 4 builds the scenario.

Reading a failure

Suppose the gate fails and names one task:

$ ansible-playbook -i inventory.test.ini site.yml

TASK [webserver : Render the site configuration] ********************************
changed: [test01.example.com]

PLAY RECAP *********************************************************************
test01.example.com         : ok=18   changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The diagnostic move is --diff, which shows the module the content it would write against the content already present:

ansible-playbook -i inventory.test.ini site.yml \
--diff --limit test01.example.com \
--tags webserver

In practice the diff usually shows one of four things:

• A timestamp or a hostname in the rendered output. Templates that embed ansible_date_time or a build number differ on every render. Move the volatile part out of the managed file, or accept that the file is regenerated and stop notifying a handler from it.
• Dictionary ordering. A for loop over a dict without a sort filter can emit keys in a different order between runs on different Python builds. Sort it.
• Trailing whitespace or a missing final newline. The file is identical to a human and different to a checksum.
• Nothing at all. The diff is empty and the task still reports changed. That is category 2 above: the module is a command or shell in disguise, or a module whose changed detection is genuinely broken for this option combination.

Legitimate exceptions, recorded rather than silenced

Some operations really are not idempotent. A database schema migration that applies a versioned delta. A vendor appliance CLI with no query mode. A licence activation call.

The wrong response is changed_when: false on the task, which makes the noise stop by lying: the task now reports ok while still doing something, and every future reader believes it is a no-op.

The right response has three parts, all of them visible in the repository:

# The vendor CLI has no query subcommand, so the task cannot ask whether
# the licence is already applied. The creates: guard is what makes this
# idempotent: the CLI writes the stamp file on success, so a converged
# host skips the task entirely.
#
# Reviewed 2026-08-11. If the vendor ships a query mode, delete this
# guard and use it instead.
- name: Apply the appliance licence
ansible.builtin.command:
  cmd: /opt/vendor/bin/apply-licence --token REPLACE_ME
  creates: /var/lib/vendor/licence.stamp
no_log: true

creates: is the honest fix here: the task is skipped when the stamp file exists, so the second run reports ok because it genuinely did nothing, not because it was told to say so.

Where no guard is possible, the exception belongs in the scenario’s documentation and in a review comment on the gate — “this role has one known non-idempotent task, here is why, here is what would remove it” — rather than in a skip_list that hides it.

What it does not prove

Being explicit keeps the test honest:

• It does not prove the declaration is correct. A role that configures the wrong TLS ciphers converges cleanly and passes the second run every time.
• It does not prove the service works. changed=0 means the declared resources match; it says nothing about whether anything is listening. That is lesson 6.
• It does not cover what the play does not declare. Only the attributes you named are compared.
• It is only as fast as the container it ran in. A role that is idempotent against a container and not against a VM is a real possibility, because the container never exercised the code path that is not idempotent. That is lesson 3.

Knowledge check