Estimating runtime before you commit to a window · Ansible

“How long will this take?” is asked at every change advisory board and answered, almost universally, with a guess. The guess is usually derived from a run against a handful of hosts, multiplied by nothing in particular, and rounded up to a number that sounds cautious.

Then the run is still going when the window closes, and the choice is between an unapproved change and an interrupted one.

The estimate is not difficult. It is four numbers, one correction that most people miss, and a check against reality.

The model

wall-clock  ~=  (hosts / effective concurrency) * per-host play time
                + fixed overhead

Written out as quantities you can actually obtain:

Quantity	How you get it
Host count	`--list-hosts`, per wave
Effective concurrency	`min(forks, serial batch size)` — and lower if any task carries a `throttle`
Per-host play time	Measured: sum of per-task cost over the play
Fixed overhead	Inventory parse, fact gathering, connection setup, handler flush

Two of those are commonly got wrong, and both make the estimate too optimistic.

Effective concurrency is not `forks`

forks is a ceiling. The number that governs is the smallest of:

forks on the controller,
the serial batch size, if set,
the throttle value on the task currently executing,
and, in practice, whatever the shared dependency will tolerate.

A play with forks: 100, serial: 50 and a throttle: 10 on the package task runs that task ten hosts at a time. If that task is most of the play’s duration, the effective concurrency for planning purposes is ten, not a hundred.

Batches are also strictly sequential. serial: 50 over 2,000 hosts is forty batches, one after another, each paying its own connection setup and fact gathering.

Sum of maxima, not maximum of sums

Here is the correction that matters, and it follows directly from how the default strategy works.

linear synchronises at every task boundary: the play does not move to task n+1 on any host until task n has finished on every host in the batch. So the batch’s duration is not the slowest host’s total time — it is the sum, over tasks, of the slowest host for that task.

An illustration with three hosts and three tasks, in seconds:

Task	host A	host B	host C	Batch pays
1	2	2	9	9
2	7	1	1	7
3	1	6	1	6
Total	10	9	11	22

The slowest host takes 11 seconds. The batch takes 22. Estimating from “the slowest host we measured” understates it by a factor of two, and the factor grows with batch size, because a larger batch is more likely to contain an outlier for any given task.

Measuring per-task cost with what core ships

profile_tasks is the usual recommendation and it is not in ansible-core — it lives in the ansible.posix collection. The callbacks that ship in core are exactly five: default, junit, minimal, oneline and tree.

If you cannot install collections on the controller, junit gives you per-task, per-host timings:

Configuration changecapture per-task timings with a core callback— Runs the play for real against a measurement group and writes a JUnit XML file with a time attribute on every host-task pair.

ANSIBLE_CALLBACKS_ENABLED=ansible.builtin.junit \
JUNIT_OUTPUT_DIR=./timings \
ansible-playbook -i inventory/ site.yml --limit measurement_group

Read-only / Safewhat the file contains— Executed against ansible-core 2.21.3. Each testcase is one host-task pair with a time attribute in seconds. Note that the testsuite time is the sum across hosts, not wall-clock.

$ head -6 timings/*.xml

<?xml version="1.0" ?>
<testsuites disabled="0" errors="0" failures="0" tests="10" time="24.97292804718017">
<testsuite disabled="0" errors="0" failures="0" name="wf" skipped="0" tests="10" time="24.97292804718017">
	<testcase classname="/path/to/site.yml:5" name="[web03] web: per-host jitter" time="0.2070465087890625">

The time on testsuites is the sum of every host-task duration, not wall-clock: that run finished in about three and a half seconds of real time. Which is a useful reminder in itself — the aggregate you want for a budget is per-task maxima, and you have to compute it.

Read-only / Safeextract the per-task maximum from the JUnit output— Groups host-task timings by task and prints the slowest host for each, which is what a linear batch actually pays.

python3 - <<'PY'
import glob, collections, xml.etree.ElementTree as ET
worst = collections.defaultdict(float)
for path in glob.glob('timings/*.xml'):
  for tc in ET.parse(path).iter('testcase'):
      task = tc.get('name').split(': ', 1)[-1]
      worst[task] = max(worst[task], float(tc.get('time')))
for task, t in sorted(worst.items(), key=lambda kv: -kv[1]):
  print(f'{t:8.2f}s  {task}')
print(f'{sum(worst.values()):8.2f}s  TOTAL per batch')
PY

A worked budget

A patching change over wave 3: 1,600 hosts, forks: 60, serial: 200, one throttle: 20 task.

Step	Working	Value
Batches	`1600 / 200`	8
Effective concurrency, normal tasks	`min(60, 200)`	60
Per-batch time, measured non-throttled tasks	sum of per-task maxima	210 s
Throttled download task	200 hosts / 20 at a time × 25 s each	250 s
Fact gathering per batch	measured	40 s
Per batch	`210 + 250 + 40`	500 s
Run time	`8 × 500 s`	66 min
Inventory parse, once	measured	1 min
Estimated wall-clock		~67 min

Then the part that is usually left out entirely:

Programme step	Value
Wave 1 canary run and verification	20 min
Gate: human review of canary evidence	30 min
Wave 2 run and soak	24 h
Wave 3 run (above)	67 min
Contingency at 30%	20 min

A change advisory board asking “how long?” usually means the whole programme, and the answer is dominated by the soak, not by Ansible.

Checking the extrapolation

An estimate built from twenty hosts and applied to two thousand is a linear model of a system that is not linear. Check it at an intermediate size — 200 hosts — before committing.

The places the extrapolation breaks:

Controller saturation. Beyond the controller’s fork ceiling, throughput drops and failures appear that look target-side.
Shared dependency saturation. The mirror is fine at 20 concurrent clients and refuses at 200.
Tail growth. A 200-host batch contains outliers a 20-host sample did not.
Fact-gathering cost, which scales with host count and is often the single largest line.

If the 200-host check lands within about 20% of the model, the model is good enough for a window. If it does not, the difference is telling you something specific, and it is worth finding out what before running against 2,000.

Under the hood

The lockstep behaviour comes from the linear strategy plugin, which is the default. Its run loop takes the next task, dispatches it to the worker pool for every host still active in the play, waits for all of them, then advances. That barrier is what makes serial batches meaningful, what makes handler notification ordering deterministic, and what makes the slowest host set the pace.

There are exactly four strategy plugins in ansible.builtin: linear, free, host_pinned and debug. free removes the barrier — hosts run ahead independently — which removes the tail penalty and also removes the guarantee that all hosts have completed task n before any begins task n+1.

That guarantee is not a formality. A rolling deployment that drains a host, changes it, verifies it and re-adds it depends on the ordering, and under free the hosts will drift out of step. Choosing free to fix a runtime estimate is a correctness decision disguised as a performance one.

What happens if the estimate says it does not fit?

Four levers, in the order that costs least.

Cut task count. Use tags to run only what the change needs. Most site playbooks contain far more than any single change requires, and running the whole thing is a habit rather than a decision.

Cut fact gathering. gather_facts: false where the play reads no facts, or gather_subset to collect only what the roles use. On a large fleet this is frequently the biggest single saving and it costs nothing in risk once you know which facts are read.

Raise concurrency, carefully. Bounded above by the controller’s real capacity and by what the shared dependencies tolerate. This is the lever people reach for first and the one most likely to make things worse.

Cut scope. Fewer hosts this window, the rest next window — which the partitioning work makes a normal operation rather than an improvisation.

What is not on the list: starting anyway and hoping. A run that overruns its window has to be either abandoned mid-fleet, leaving a mixed estate that nothing has recorded, or continued without approval. Both are worse than moving two hundred hosts to next Tuesday.