Lab: The second-run test — take a play from six changes to zero · Lab · Ansible

Objective

By the end of this lab a supplied play that reports six changes on every run will report zero on its second run, and you will be able to justify each fix from evidence. Crucially, you will have rejected at least two “fixes” that would have made the recap clean while making the play less truthful than it was before.

Architecture

One managed node and a small application deployment. The play is deliberately representative: it uses command, shell, lineinfile, a copy with a generated timestamp, a git checkout and a package install — which between them cover every common cause of a false change.

controller ──ssh──▶ node1   (Debian 12 or Ubuntu 24.04)
                     ├── /opt/reporting/          <- deployment target
                     ├── /etc/reporting.conf      <- config file
                     └── reporting.service        <- systemd unit

Requirements

A controller with ansible-core 2.21.x.
One managed node running a real init system. This lab installs a package and manages a systemd unit, so a container without systemd as PID 1 will report a service task as changed on every run for a reason that has nothing to do with your play — and you will chase it. Declare B-nested and use a VM.
SSH key access to the node and a become path. No out-of-band access requirement: this lab does not touch SSH, the firewall or networking, so a mistake cannot lock you out. It can leave a service stopped, which Cleanup addresses.
The node must have internet access, or a local package mirror, for the package task.

Scenario

A play that deploys an internal reporting tool reports changed=6 on every run, including runs where nothing has changed on either side. The team has learned to ignore the recap, which means they have lost the only cheap signal they had about whether a run did anything.

Somebody has proposed adding changed_when: false to all six. Your job is to fix it properly and to be able to explain why that proposal is worse than the problem.

Tasks

Task 1: Capture the starting state

# Substitute your own values before running:
NODE=node1
WORKDIR="$HOME/ansible-idempotency-lab"

mkdir -p "$WORKDIR"
cd "$WORKDIR"

inventory.yml:

reporting:
  hosts:
    node1:
      ansible_host: 192.0.2.11
  vars:
    ansible_user: operator

Capture what the node looks like before anything runs. Cleanup uses this:

# capture.yml
- name: Capture pre-lab state
  hosts: reporting
  gather_facts: true
  become: true
  tasks:
    - name: Record whether the config file already exists
      ansible.builtin.stat:
        path: /etc/reporting.conf
      register: pre_conf

    - name: Back up the config file if it exists
      ansible.builtin.copy:
        src: /etc/reporting.conf
        dest: /etc/reporting.conf.pre-lab
        remote_src: true
        mode: preserve
      when: pre_conf.stat.exists

    - name: Record the installed package list
      ansible.builtin.package_facts:

    - name: Save the pre-lab state to the controller
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          captured: {{ ansible_date_time.iso8601 }}
          reporting.conf existed: {{ pre_conf.stat.exists }}
          jq installed: {{ 'jq' in ansible_facts.packages }}
        dest: "{{ playbook_dir }}/pre-lab-{{ inventory_hostname }}.txt"
        mode: '0644'
      delegate_to: localhost
      become: false

Configuration changecontroller— Writes one backup file on the node and one record on the controller

$ ansible-playbook -i inventory.yml capture.yml

Task 2: Run the offending play twice

site.yml — the play as inherited:

- name: Deploy the reporting tool
  hosts: reporting
  become: true
  gather_facts: true

  tasks:
    - name: Install jq
      ansible.builtin.package:
        name: jq
        state: present

    - name: Ensure the deployment directory exists
      ansible.builtin.file:
        path: /opt/reporting
        state: directory
        mode: '0755'

    # 1. shell with no change detection
    - name: Ensure the reporting user owns the directory
      ansible.builtin.shell: chown -R reporting:reporting /opt/reporting || true

    # 2. command with no change detection
    - name: Regenerate the report index
      ansible.builtin.command: /usr/bin/find /opt/reporting -name '*.json'

    # 3. content that includes a timestamp
    - name: Write the deployment marker
      ansible.builtin.copy:
        content: |
          deployed_at: {{ ansible_date_time.iso8601 }}
          deployed_by: ansible
        dest: /opt/reporting/DEPLOYED
        mode: '0644'

    # 4. lineinfile with a regexp that never matches what it inserts
    - name: Set the log level
      ansible.builtin.lineinfile:
        path: /etc/reporting.conf
        regexp: '^log_level'
        line: 'log_level = warn'
        create: true
        mode: '0644'

    # 5. a check that reports changed because it ran
    - name: Verify the config parses
      ansible.builtin.shell: |
        set -o pipefail
        grep -c '^log_level' /etc/reporting.conf
      args:
        executable: /bin/bash

    # 6. an unconditional restart
    - name: Restart the reporting service
      ansible.builtin.systemd_service:
        name: reporting
        state: restarted
      failed_when: false

Run it twice and record both recaps:

ansible-playbook -i inventory.yml site.yml | tee run1.txt | tail -3
ansible-playbook -i inventory.yml site.yml | tee run2.txt | tail -3

Configuration changecontroller— The second run. Every one of these changes is false.

$ ansible-playbook -i inventory.yml site.yml | tail -3

PLAY RECAP *********************************************************************
node1                      : ok=8    changed=6    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Illustrative output

The first run’s changed=6 might be honest. The second run’s is not: nothing changed on the node between them, so a truthful play would report changed=0.

Task 3: Classify before you fix

Every false change has one of four causes. Getting the classification right determines the fix; guessing produces changed_when: false everywhere.

Cause	What it looks like	Correct fix
A. Wrong module	`command`/`shell` doing something a module does	Use the module
B. No change detection possible	A genuinely necessary command with no module	`changed_when` derived from its output
C. Content is not stable	A timestamp, a UUID, a sorted-differently render	Remove the unstable part from the managed content
D. Unconditional state change	`state: restarted`, `state: absent` then `present`	Gate it on an actual change, via a handler

Fill in classification.md for all six tasks before writing any YAML.

Task 4: Fix each one at its own layer

Task 1 — chown -R in a shell. Cause A. The file module does this with real change detection:

- name: Ensure the reporting user owns the directory
  ansible.builtin.file:
    path: /opt/reporting
    state: directory
    owner: reporting
    group: reporting
    recurse: true
    mode: '0755'

recurse: true reports changed only when something actually changed ownership, because the module stats each path first. The shell version could not know.

Task 2 — find for an index. Cause B, and also a task that changes nothing at all. find only reads. A read that reports changed is the purest form of the defect:

- name: Regenerate the report index
  ansible.builtin.command: /usr/bin/find /opt/reporting -name '*.json'
  changed_when: false
  register: report_index

Task 3 — the timestamp. Cause C. The content changes every second because you put a clock in it. The fix is not to suppress the change, it is to stop generating unstable content:

- name: Write the deployment marker
  ansible.builtin.copy:
    content: |
      deployed_version: {{ reporting_version }}
      deployed_by: ansible
    dest: /opt/reporting/DEPLOYED
    mode: '0644'

If the deployment time genuinely must be recorded, record it where a change is meaningful — a log line, or a fact file written only when the version changes:

- name: Record the deployment time when the version changed
  ansible.builtin.copy:
    content: "deployed_at: {{ ansible_date_time.iso8601 }}\n"
    dest: /opt/reporting/DEPLOYED_AT
    mode: '0644'
  when: marker.changed

Task 4 — the lineinfile regexp. Cause A in disguise. Check the file after a run:

# Substitute your own values before running:
NODE=node1

ansible -i inventory.yml "$NODE" -b -m command -a 'cat /etc/reporting.conf'

If the file has accumulated repeated log_level = warn lines, the regexp is not matching the line the module inserts. ^log_level does match log_level = warn, so the more common cause is a different one — the file has log_level=warn without spaces from another source and line: writes the spaced form, so the module rewrites it every time. Either way, lineinfile on a config file with a defined format is the wrong tool:

- name: Set the log level
  ansible.builtin.template:
    src: reporting.conf.j2
    dest: /etc/reporting.conf
    mode: '0644'
  notify: Restart reporting

A template owns the whole file and compares the whole file. There is no regexp to get subtly wrong.

Task 5 — the verification shell. Cause B. It must run, and it reads only. Derive the change decision — and the failure decision — from its output:

- name: Verify the config parses
  ansible.builtin.shell: |
    set -o pipefail
    grep -c '^log_level' /etc/reporting.conf
  args:
    executable: /bin/bash
  register: config_check
  changed_when: false
  failed_when: config_check.stdout | int != 1

Two improvements over the original. It never reports changed, because it cannot change anything. And it now actually verifies — the original ran grep -c and ignored the answer, so a config with zero or five log_level lines passed silently.

Task 6 — the unconditional restart. Cause D, and the one with real production consequences. state: restarted restarts the service every run, whether or not anything changed. On a fleet with a rolling deployment that is an outage per host per run.

  handlers:
    - name: Restart reporting
      ansible.builtin.systemd_service:
        name: reporting
        state: restarted

…notified from the tasks that write configuration, and nowhere else. Add a state-not-restart task to make sure the service is running without bouncing it:

- name: Ensure the reporting service is enabled and running
  ansible.builtin.systemd_service:
    name: reporting
    state: started
    enabled: true

state: started is idempotent — it reports changed only if the service was not running. state: restarted is not, and never can be.

Task 5: Prove the fix

cd "$HOME/ansible-idempotency-lab"

ansible-playbook -i inventory.yml site.yml | tee fixed-run1.txt | tail -3
ansible-playbook -i inventory.yml site.yml | tee fixed-run2.txt | tail -3

The second run must report changed=0. Then prove you did not cheat:

# Every changed_when in the play, with its task
grep -n -B12 'changed_when' site.yml | grep -E 'name:|changed_when|command:|shell:'

For each changed_when: false, you must be able to say in one sentence why the command cannot change state. If you cannot, it is suppression.

Task 6: Confirm the play still does its job

A play that reports changed=0 because it stopped doing anything is worse than the original. Verify the end state independently:

# verify.yml
- name: Verify the deployment
  hosts: reporting
  become: true
  gather_facts: false
  tasks:
    - name: The config file has exactly one log_level line
      ansible.builtin.shell: |
        set -o pipefail
        grep -c '^log_level' /etc/reporting.conf
      args:
        executable: /bin/bash
      register: levels
      changed_when: false
      failed_when: levels.stdout | int != 1

    - name: The deployment directory is owned by the reporting user
      ansible.builtin.stat:
        path: /opt/reporting
      register: dir

    - name: Assert ownership
      ansible.builtin.assert:
        that:
          - dir.stat.exists
          - dir.stat.pw_name == 'reporting'
        success_msg: "deployment directory ownership correct"

Validation

run2.txt (before the fix) reports changed=6.
fixed-run2.txt reports changed=0.
fixed-run1.txt reports a non-zero change count on a node that was not already converged — a play that reports zero on the first run has stopped working.
classification.md assigns each of the six tasks to cause A, B, C or D.
Every changed_when: false in the final play is on a task that reads only. The play contains at most two of them.
verify.yml passes.
ansible-playbook -i inventory.yml site.yml --check on a converged node also reports changed=0, and you can explain any task where it does not.

Expected Outcome

ansible-idempotency-lab/
├── classification.md
├── fixed-run1.txt, fixed-run2.txt
├── inventory.yml
├── pre-lab-node1.txt
├── run1.txt, run2.txt
├── site.yml
├── templates/reporting.conf.j2
└── verify.yml

On the node: /opt/reporting owned by the reporting user, one /etc/reporting.conf with exactly one log_level line, a running reporting service that was restarted once during the fixing and not since. Second and subsequent runs report changed=0.

Troubleshooting

changed=1 on the package task on every run. The package is not actually being installed — check state: present rather than state: latest. latest compares against the repository on every run and reports changed whenever an update is available, which is honest but unhelpful in a convergence play.

changed=1 on the systemd task in a container. Expected, and it is why this lab needs a VM. Without systemd as PID 1 the module cannot read service state and reports what it did rather than what changed.

The template task reports changed every run. Diff it: ansible-playbook ... --check --diff --tags config. A trailing newline, a {% for %} over an unordered dictionary, or a Jinja whitespace-control difference will all produce a one-character diff every run. Sort dictionary iterations with | dictsort.

changed=0 on the second run, but the third run changes again. Two tasks are fighting: one writes a value the other overwrites. Run with --diff and look for the same file appearing twice.

lineinfile keeps appending duplicates. The regexp does not match the line it inserts. The module searches with regexp and, finding no match, appends line — forever. Test by making regexp match the exact line you are inserting, or stop using lineinfile for this file.

Cleanup

This lab installed a package, created a directory tree, wrote a config file and restarted a service. All of that has to come back.

Step 1. Read the pre-lab capture, so you know what to restore rather than what to delete:

cd "$HOME/ansible-idempotency-lab"
cat pre-lab-node1.txt

Step 2. Restore or remove, branching on what the capture recorded:

# cleanup.yml
- name: Restore the pre-lab state
  hosts: reporting
  become: true
  gather_facts: false
  tasks:
    - name: Look for the pre-lab config backup
      ansible.builtin.stat:
        path: /etc/reporting.conf.pre-lab
      register: backup

    - name: Restore the original config when one existed
      ansible.builtin.copy:
        src: /etc/reporting.conf.pre-lab
        dest: /etc/reporting.conf
        remote_src: true
        mode: preserve
      when: backup.stat.exists

    - name: Remove the config when the lab created it
      ansible.builtin.file:
        path: /etc/reporting.conf
        state: absent
      when: not backup.stat.exists

    - name: Remove the backup marker
      ansible.builtin.file:
        path: /etc/reporting.conf.pre-lab
        state: absent

    - name: Remove the deployment directory the lab created
      ansible.builtin.file:
        path: /opt/reporting
        state: absent

    - name: Leave the service in a defined state
      ansible.builtin.systemd_service:
        name: reporting
        state: stopped
        enabled: false
      failed_when: false

Do not remove the package blindly

jq is a common utility and something else on the node may depend on it. The pre-lab capture recorded whether it was already installed; remove it only if it was not:

grep 'jq installed' pre-lab-node1.txt

If that line says False, and only then:

ansible -i inventory.yml reporting -b -m package -a 'name=jq state=absent'

Uninstalling a package that was already present, on a shared node, is a larger change than anything this lab made.

Step 3. Verify the node is back:

ansible -i inventory.yml reporting -b -m stat -a 'path=/opt/reporting' \
  | grep '"exists"'
ansible -i inventory.yml reporting -b -m stat -a 'path=/etc/reporting.conf' \
  | grep '"exists"'

Step 4. Keep the classification and remove the working directory:

rm -rf deletes the whole lab directory, including anything you added

The next block removes ~/ansible-idempotency-lab and everything under it, recursively and without confirmation. If you saved notes, an inventory you want to keep, or a playbook you extended, copy it out first — the step above does that for the lab’s own deliverables and knows nothing about yours.

The path is written in full and anchored to $HOME deliberately. A relative rm -rf run from the wrong directory is how people delete work they meant to keep, and the shell gives no second chance.

Confirm what you are about to delete before you delete it:

ls -la "$HOME/ansible-idempotency-lab"

mkdir -p "$HOME/ansible-lab-deliverables"
cp -a "$HOME/ansible-idempotency-lab/classification.md" \
      "$HOME/ansible-lab-deliverables/idempotency-classification.md"

rm -rf "$HOME/ansible-idempotency-lab"

Under the hood

changed is not something Ansible observes; it is something a module returns. Every module decides for itself, and the quality of that decision is a property of the module rather than of Ansible.

file stats the path, compares mode, owner, group and type against what was requested, and returns changed: true only if it altered one of them. copy and template checksum the rendered content against the destination and return changed only on a mismatch. package asks the package manager what is installed. These are all comparisons, which is why they are idempotent.

command and shell have nothing to compare. They execute an opaque string and cannot know what it did, so they return changed: true unconditionally — the honest answer to “did something change?” from a module that cannot tell is “assume yes”. changed_when is the mechanism for supplying the comparison the module could not make, using rc, stdout or stderr from the result.

This also explains why creates: and removes: are the only check-mode support command has. Both turn the opaque execution into a stat comparison, which is something the module can evaluate without running anything.

What You Learned

The second run is the test. Two invocations and a recap tell you more about a play’s quality than any amount of reading it.
Four causes, four different fixes. Wrong module, no detection possible, unstable content, unconditional state change — and only one of them is fixed by changed_when.
changed_when: false is honest on a read and dishonest on a write. You applied it to find and to a grep -c, and refused it on the restart, the timestamp and the chown.
A timestamp in managed content is a design bug, not a reporting bug. The fix was to stop putting a clock in a file whose contents are supposed to be stable.
state: restarted can never be idempotent. It became a handler notified by the tasks that actually write configuration, and state: started took over the “make sure it is running” job.
A changed=0 play still has to be verified independently. The separate verify.yml is what distinguishes a converged play from one that quietly stopped doing its job.

Lab: The second-run test — take a play from six changes to zero

Objectives

Prerequisites