Reducing three hundred hosts to one reproduction

Nothing in this part is usable against three hundred hosts at once. The first job in any Ansible investigation is to get down to one host, one task, one repeatable failure — and then everything else becomes straightforward.

The tools for doing that are all pre-flight options you have already met. What this lesson adds is the order, and the traps in two of them.

The order matters

Narrow along the axis that costs you the least information.

--limit first, always. Reducing the host count costs you nothing except the ability to observe host-to-host variation, and if the failure is host-specific you already know which host.

--start-at-task is fourth rather than second for a reason developed below: it discards the state that earlier tasks would have established, which is frequently what the later task depends on.

Step 1: one host

# confirm the limit resolves to exactly the host you mean
ansible-playbook -i inventory/prod site.yml \
--limit app-047.example.com --list-hosts

# then work only against it
ansible-playbook -i inventory/prod site.yml \
--limit app-047.example.com --check --diff

Choose the host deliberately. If thirty hosts failed, pick one that is representative of the largest sub-population, and keep a second one untouched as a control — you will want a host in the failed state to compare against once you have changed something.

Step 2: the relevant section only

$ ansible-playbook -i inv.ini tasks.yml --list-tasks ; ansible-playbook -i inv.ini tasks.yml --list-tags

playbook: tasks.yml

play #1 (local): Patch wave	TAGS: []
  tasks:
    Drain from the load balancer	TAGS: [drain]
    Apply package updates	TAGS: [patch]
    Restart the service	TAGS: [patch, restart]
    Health check	TAGS: [verify]
    Return to service	TAGS: [drain]

playbook: tasks.yml

play #1 (local): Patch wave	TAGS: []
    TASK TAGS: [drain, patch, restart, verify]

--list-tasks is the map. Run it before selecting anything, because a tag that matches nothing produces an empty run that looks like a fast success, and --list-tags is how you find out that the tag you were about to use is spelled differently.

Step 3: check mode, before real execution

--check --diff against the single host tells you what each task believes it needs to do without doing it. On a debugging run this is frequently the whole answer: a task that reports changed in check mode against a host you believe is already correct is telling you the host is not correct, and the diff shows how.

The limits apply as always: command, shell, raw and script skip in check mode unless given check_mode: false, so a play built mainly from those returns little.

Step 4: --start-at-task, and the trap

$ ansible-playbook -i inv.ini tasks.yml --start-at-task 'Health check'

TASK [Health check] ************************************************************
ok: [lh] => { "msg": "healthy" }

TASK [Return to service] *******************************************************
ok: [lh] => { "msg": "inservice" }

PLAY RECAP *********************************************************************
lh    : ok=2  changed=0  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0

The second problem is state. Tasks before the start point frequently establish things the later task needs: a set_fact, a registered variable, gathered facts, a package that must be installed before the service can be configured. Starting after them produces a failure that is an artefact of your narrowing rather than a reproduction of the original.

Step 5: --step

--step prompts before each task with (N)o/(y)es/(c)ontinue. It is the right tool when you want to stop just before the failing task and go and look at the host in another terminal.

It is interactive, so it belongs to a human at a keyboard against one host, never in a pipeline. Combined with --limit to a single host it turns a play into a guided walk.

Step 6: the debug strategy

The last resort, and the most powerful: an interactive debugger that activates at the point of failure, with the task’s variables in scope.

$ ansible-doc -t strategy ansible.builtin.debug

> STRATEGY ansible.builtin.debug

Task execution is 'linear' but controlled by an interactive debug
session.

- name: Reproduce the failure interactively
hosts: app-047.example.com
strategy: debug
gather_facts: true
tasks:
  - name: The task that fails
    ansible.builtin.template:
      src: app.conf.j2
      dest: /etc/app/app.conf
      mode: '0640'

When a task fails, you get a prompt. The commands that matter:

p task.args is the one that earns the whole mechanism. It shows the arguments after templating, which is the value that actually reached the module — and a discrepancy between that and what you expected is a templating problem, not a module problem.

The narrowing, applied

A worked sequence for “the deploy fails on about ten percent of hosts, different ones each time”:

1. Classify. Are the failing hosts random, or do they share something? Pull the failed list from three runs and intersect them. An empty intersection points at timing or contention; a stable set points at host state.
2. --limit to one failing host, and keep a second failing host untouched as a control.
3. --check --diff on it. If check mode already shows something unexpected, you have the answer without running anything.
4. --tags down to the smallest coherent section that still fails.
5. -vvv on that section, with fake values if it handles secrets.
6. ssh -vvv and sudo -n true if it looks like transport or escalation.
7. strategy: debug and p task.args if the arguments are the suspect.

Steps 1 to 3 resolve the majority. The point of the ordering is that each step is cheaper and safer than the one after it.

Knowledge check