Making an unavoidable shell task idempotent · Ansible

Sometimes there is no module. A vendor’s installer, a licence activation, a database migration tool, a proprietary CLI that manages its own state — occasionally the honest answer is that the work has to be done by running a program.

That is a legitimate position. What is not legitimate is leaving the task reporting changed on every run, because that result is consumed by machinery that assumes it is true.

Why an inaccurate `changed` is expensive

A changed result is not decoration. Three separate systems read it:

Handlers. notify fires on changed. A read-only probe written as a command task will restart the service it was probing, every run, forever. On a fleet with serial batching, that is a rolling restart nobody asked for.
The recap. changed=0 on a converged fleet is the single most useful number in Ansible. It is how an operator knows a run did nothing, which is how they know the estate matches its definition. A play with six always-changed tasks reports changed=6 on a fleet that is perfectly converged, and the number stops meaning anything.
The audit trail. “What changed on these servers last Tuesday” is a question somebody will ask you. If the answer is “the log says everything changed, every night, for two years”, you cannot answer it.

Under the hood

The module cannot help you here, and it is worth being precise about why. command.py sets the result before it runs anything:

r['changed'] = True

# actually executes command (or not ...)
if not module.check_mode:
    r['rc'], r['stdout'], r['stderr'] = module.run_command(args, ...)

There is no inspection of the command, no comparison, no heuristic that occasionally guesses wrong. An arbitrary command has no declared desired state, so there is nothing for the module to compare against.

Everything below is therefore about supplying that missing comparison yourself, from outside the module. There are exactly three places to put it: before the task runs (creates), in a separate task (when), or after the task runs (changed_when).

Technique 1: `creates` and `removes`

The best fix is the one where the work does not happen at all.

creates names a path; if it exists, the module exits without running. removes is the inverse: if the path does not exist, the module exits without running. Both are evaluated by the module on the managed node, and both work under --check, which is the only check-mode support command and shell have.

Configuration changethe cheapest guard there is— The installer runs once. On every subsequent run the marker exists and the module exits before executing anything.

- name: Run the vendor installer
ansible.builtin.command:
  cmd: /opt/vendor/install.sh --unattended
  creates: /opt/vendor/.installed

- name: Extract the release archive
ansible.builtin.command:
  cmd: /usr/bin/tar -xzf /tmp/app-1.4.2.tgz -C /opt/app
  creates: /opt/app/releases/1.4.2/VERSION

Read-only / Safewhat a guarded task returns— Executed against 2.21.3 with -v. The module reports ok, not changed, and says in plain text why it did not run.

$ ansible-playbook -i localhost, idem.yml -v

TASK [Technique 1 - creates guard, work already done] **************************
ok: [localhost] => {"changed": false, "cmd": ["/bin/echo", "would-have-run"],
"msg": "Did not run command since '/etc/hostname' exists",
"rc": 0, "stdout": "skipped, since /etc/hostname exists"}

The judgement call is which path to name. Two rules that hold up:

Name something the work itself produces. A version-stamped directory, a generated config, a binary. Not a marker file you touch yourself in a following task — that decouples the guard from the work, and the first failed run leaves you with a marker and no installation.
Name something version-specific when the command is version- specific. creates: /opt/app/current never runs again after 1.4.2, including for 1.5.0. creates: /opt/app/releases/{{ app_version }}/VERSION runs once per version, which is what you meant.

Technique 2: a read-only probe and a `when`

When there is no file whose existence answers the question, ask the question explicitly in a separate task — one that reads state and changes nothing — and drive the work from its result.

Configuration changeprobe, decide, act— The probe is read-only and declares itself so. The work runs only when the probe says it is needed.

- name: Read the currently applied schema version
ansible.builtin.command: /usr/local/bin/appmigrate --current-version
register: schema_probe
changed_when: false          # correct here: this task genuinely changes nothing

- name: Apply outstanding migrations
ansible.builtin.command: /usr/local/bin/appmigrate --apply
when: schema_probe.stdout | trim != app_schema_version
notify: Restart app

Read-only / Safethe converged second run— Executed against 2.21.3. The probe reports ok, the work task is skipped, and the recap reports zero changes on an already-converged host.

$ ansible-playbook -i localhost, idem.yml

TASK [Technique 2a - read-only probe] ******************************************
ok: [localhost]

TASK [Technique 2b - do the work only when the probe says so] ******************
skipping: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=1

Three properties make this the technique to reach for when creates does not fit:

The probe is honest about itself. changed_when: false on a task that provably reads and does not write is a true statement, not a suppression. That distinction is the whole point of the last section of this lesson.
The decision is visible in the output. A skipped task with false_condition in the verbose result tells the next engineer exactly why nothing happened.
The work task can notify a handler truthfully, because it only runs when there is something to do.

Where a purpose-built read-only module exists, use it instead of a command probe. ansible.builtin.stat is the common one:

Read-only / Safestat instead of a command probe— stat supports check mode properly and returns structured data, so the whole pattern survives a dry run.

- name: Look for the certificate the renewal would replace
ansible.builtin.stat:
  path: /etc/pki/tls/certs/app.pem
register: cert

- name: Renew the certificate
ansible.builtin.command:
  cmd: /usr/bin/certtool --renew --out /etc/pki/tls/certs/app.pem
when: not cert.stat.exists or (cert.stat.mtime | int) < (renew_before_epoch | int)

Technique 3: `changed_when` from the command’s own output

Last resort, and still a real technique: run the command, then decide from what it said.

Configuration changereading the tool's own report— Only correct when the command is safe to run repeatedly and reports what it did.

- name: Converge the firewall ruleset
ansible.builtin.command: /usr/sbin/fwctl apply --config /etc/fw/rules.conf
register: fw_apply
changed_when: "'no changes required' not in fw_apply.stdout"
notify: Reload firewall

- name: Apply outstanding database migrations
ansible.builtin.command: /usr/local/bin/appmigrate --apply
register: migrate
changed_when: migrate.stdout is not search('0 migrations applied')

Read-only / Safethe derived result— Executed against 2.21.3 with -v. changed_when_result records the judgement, so a wrong expression is visible rather than mysterious.

$ ansible-playbook -i localhost, idem.yml -v

TASK [Technique 3 - changed_when derived from the command output] **************
ok: [localhost] => {"changed": false, "changed_when_result": false,
"cmd": ["/bin/echo", "nothing to do"], "rc": 0,
"msg": "", "stdout": "nothing to do"}

Two conditions have to hold before this is honest:

The command must be safe to run every time. changed_when is evaluated after execution. It changes the report, never the behaviour. If running the command twice is harmful, this technique does not apply and you need technique 1 or 2.
The output must be a reliable signal. A string match against a message that a vendor can reword in a point release is a guard with a maintenance cost. Prefer a documented exit code, a structured output mode (--format json piped through from_json), or a counter.

The thing this is not

Blanket changed_when: false is a reporting change, not a fix

Adding changed_when: false to a task that genuinely modifies the system does not make it idempotent. It makes it dishonest. The task still does the work on every run; it simply stops saying so.

The damage is precise:

Handlers that should fire no longer fire, because there is no changed to notify on. A config task that stops restarting its service is a config change that never takes effect, and nothing in the output hints at it.
changed=0 becomes meaningless as a convergence signal, and every future reader of that playbook believes a run did nothing when it did a great deal.
The audit trail records no change on a host that was changed.

changed_when: false is correct on exactly one kind of task: one that reads and does not write. The probe in technique 2 is the archetype. Ask “if I run this a hundred times, does the host differ?” — if the answer is yes, changed_when: false is the wrong tool.

The anti-patterns part of this course treats this as its own case study, because it is the most common way a mature Ansible codebase quietly stops meaning anything.

Choosing between them

Situation	Technique
The work produces a file or directory	`creates`
The work removes something	`removes`
State is readable but is not a file	Probe task plus `when`
The command reports what it did, and is safe to rerun	`changed_when` from output
The command is not safe to rerun and state is unreadable	Redesign. There is no reporting fix for this.

Blast radius

An always-changed task is a blast-radius problem in itself, because it propagates. On a play with serial: 10 and a notify: Restart app, one dishonest probe task restarts the service on every host in the fleet, in batches, every night — an outage schedule created by a reporting bug.

Before trusting a play as converged, run it twice and read the second recap:

Configuration changethe two-run test— The second run of a correct play changes nothing. This is the single most useful test in Ansible and it takes one extra minute.

ansible-playbook -i inventories/staging site.yml --limit staging-web01.example.com
ansible-playbook -i inventories/staging site.yml --limit staging-web01.example.com

Any task reporting changed in the second run is either a genuine drift source or a task that needs one of the three techniques above. There is no third explanation.

Why an inaccurate changed is expensive

Technique 1: creates and removes

Technique 2: a read-only probe and a when

Technique 3: changed_when from the command’s own output