Partitioning a run so it can be resumed

A run that takes four minutes and a run that takes four hours are not the same operation performed at different speeds. They have different failure profiles, and the difference is not gradual.

In four minutes, essentially nothing about the estate changes. The inventory you parsed at the start is still true at the end. Your SSH agent is still unlocked. Your laptop has not slept. The change window has not closed.

In four hours: hosts are built and destroyed, the on-call engineer changes, a credential expires, a network maintenance starts, somebody reboots a host for an unrelated reason, and the person who started the run has gone home. The run’s environment is no longer a constant. It is a thing that drifts underneath the run.

What a long run has to survive

Concretely, over four hours:

• Inventory drifts. Hosts appear and disappear in the source of truth. The set Ansible parsed at second zero is what it will use for the whole run — new hosts are not picked up, and destroyed hosts stay in the target list until they go unreachable.
• Credentials expire. A vaulted token, a Kerberos ticket, an SSH certificate with a lifetime.
• The controller can die, taking the entire in-memory record of progress with it.
• The window closes. A change approved until 06:00 that is still running at 06:00 has become an unapproved change.
• A human loses patience and presses Ctrl-C, which is the most common way long runs end.

None of these are exotic. They are the normal weather of a large estate, and a single monolithic ansible-playbook invocation has no defence against any of them.

Ansible does not remember where it got to

This is the fact that surprises people, so it is worth being blunt: on an interrupted run, Ansible retains nothing. The per-host statistics that become the recap are accumulated in the process’s memory and die with it. There is no state file, no journal, no “hosts completed” list.

Two mechanisms look like they might help. Neither does.

The retry file

ansible-playbook can write a .retry file listing hosts to re-run, and --limit @file accepts it. But:

$ ANSIBLE_RETRY_FILES_ENABLED=true ansible-playbook -i inventory.ini serial.yml; cat serial.retry

PLAY RECAP *********************************************************************
web01                      : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

web01

One line. The retry file lists the hosts that failed, not the hosts that were not reached. Re-running with --limit @serial.retry would retry that single host and leave the other nine in their original state forever, while the operator believes the run has been resumed.

Two further points about it:

• RETRY_FILES_ENABLED defaults to false on 2.21.3, so unless somebody turned it on, there is no retry file at all.
• A Ctrl-C interrupt does not produce one either.

--start-at-task

The other tempting option is to restart the play from the task it stopped at. It does work — and it silently discards everything the earlier tasks established.

$ ansible-playbook -i inventory.ini probe.yml --start-at-task 'use the probe'

[ERROR]: Task failed: Finalization of task args for 'ansible.builtin.debug'
failed: Error while resolving value for 'msg': 'version' is undefined
fatal: [web01]: FAILED! => {"msg": "Task failed: ... 'version' is undefined"}

An undefined-variable error is the good outcome, because it stops. The bad outcome is a play where the skipped task set a variable that has a default() elsewhere, or a when condition that quietly evaluates false — so the play continues and does something different from what it would have done, with no error at all.

--start-at-task is a troubleshooting aid for a play you are actively debugging. It is not a resumption mechanism for production, and it addresses the wrong axis anyway: a fleet run stops part-way through the hosts, not part-way through the tasks.

Partition the run yourself

The design that works is boring: many small runs instead of one large one, driven by something that records completion.

A partition should be:

• Fixed in membership, so that “partition 7” means the same hosts today as yesterday. Write the host lists to files; do not re-derive them from a dynamic query on each attempt.
• Independently valid. Completing partitions 1–5 and not 6–20 must leave the estate in a state that is odd but safe.
• Small enough to complete inside the window, with room for a retry.
• Recorded on completion, outside the controller’s memory.

set -euo pipefail

ansible-playbook -i inventory/ site.yml \
--limit wave3_domain_a --list-hosts \
| awk 'NF==1 && /\./ {print $1}' > wave3.hosts

split -l 200 -d --additional-suffix=.hosts wave3.hosts part-

wc -l wave3.hosts part-*.hosts

set -uo pipefail

mkdir -p state
COMMIT="$(git rev-parse HEAD)"

for part in part-*.hosts; do
marker="state/${part}.done"
if [ -f "$marker" ]; then
  echo "skip $part (completed at $(cat "$marker"))"
  continue
fi

echo "=== $part : $(wc -l < "$part") hosts"
if ansible-playbook -i inventory/ site.yml --limit "@$part"; then
  printf '%s %s\n' "$(date -Is)" "$COMMIT" > "$marker"
else
  echo "FAILED on $part - stopping. Fix, then re-run this script."
  exit 1
fi
done

The properties that make this worth the twenty lines:

• Re-running the script after any interruption resumes from the right place, because the marker files say what completed.
• Each partition is a separate process, so a controller restart costs at most one partition.
• Failure stops the programme rather than continuing into the next partition — the wave-gate discipline, enforced mechanically.
• The commit SHA is recorded per partition, so a later investigation can see that partitions 1–5 got one version of the content and 6–20 got another.

Re-entry is safe because the play is idempotent

The driver above re-runs whole partitions, including hosts that already succeeded within a failed partition. That is only acceptable because running the play twice against a converged host is a no-op.

Which makes idempotence a prerequisite for resumability, not a nice-to-have. If the play is not idempotent, re-entry is a second deployment, and the partitioning scheme has made things worse rather than better.

The operations that genuinely cannot be idempotent — reboots, one-shot data migrations, write-only API calls — need fencing so that re-entry skips them. The usual fence is a state marker the operation itself writes, checked by creates: or a stat guard.

- name: run the one-shot schema migration
ansible.builtin.command:
  cmd: /usr/local/bin/migrate-schema --to 14
  creates: /var/lib/app/.migrated-to-14

Knowledge check