forks and the controller real limits · Ansible

The configuration part of this course covered what forks is, that its default is 5, and the argument that it is functioning as an accidental blast-radius limit which must be replaced before it is removed. All of that still holds and none of it is repeated here.

This lesson answers the performance question that comes next: how high can it usefully go, and what decides that?

The short version is that “more forks is faster” is true over a narrow range and false outside it, and the range is smaller than most people assume.

The curve, measured

24 hosts, gathering facts, on a controller with 12 cores and 46 GiB of RAM. Every host uses a local connection, so this measures controller-side work with the network removed — which isolates the ceiling this lesson is about.

Read-only / Safewall clock against fork count, two passes— ansible-core 2.21.3. Fact gathering is real CPU and process work on the controller, which is what makes it a fair proxy for a fork-bound workload.

$ for f in 1 2 4 8 12 16 24 32; do /usr/bin/time -f '%e s' ansible-playbook -i facts.ini facts.yml -f $f; done

pass 1                          pass 2
forks=1    14.49 s
forks=2     7.44 s
forks=4     4.54 s              forks=4     4.61 s
forks=8     2.37 s              forks=8     2.92 s
                              forks=12    2.84 s
forks=16    2.83 s              forks=16    2.28 s
                              forks=24    2.60 s
forks=32    2.46 s              forks=32    2.60 s

Read the shape rather than the individual numbers.

One to eight: near-linear. 14.49 to 7.44 to 4.54 to around 2.6. Each doubling roughly halves the time. This is the region where “more forks is faster” is simply true.

Eight to thirty-two: flat. Every value from 8 upward lands between 2.28 and 2.92 seconds, and the two passes disagree about which fork count is fastest. The run-to-run spread is about half a second — larger than the entire difference between forks: 8 and forks: 32.

The honest conclusion from this data is not “16 is optimal”. It is: past roughly the core count, there is no measurable gain, and anyone reporting one from a single run is reporting noise.

This is one controller and one workload

12 cores, 46 GiB, 24 hosts, a purely controller-bound workload with no network. Your numbers will be different, and the differences matter:

A network-bound run — real SSH to real hosts across a real network — spends most of each fork’s life waiting, so it sustains a fork count well above the core count before the controller becomes the constraint.
A memory-constrained controller hits its ceiling long before its CPU does, and the symptom is swapping rather than saturation.
A module that returns large output — a find over a big tree, a slurp of a real file, a command producing megabytes — multiplies memory per fork by the result size.

The transferable thing is the method, not the number. Nobody can tell you your fork ceiling, and anyone who offers a figure without asking about your controller is guessing.

What each fork costs

A fork is a real operating-system process, and the accounting is concrete. Per host being worked on, the controller pays for:

A Python process with the Ansible libraries loaded. Tens of megabytes of resident memory before your task does anything.
An SSH client process and connection, plus a persistent control socket when ControlPersist is in use — which it is by default, since ssh_args defaults to -C -o ControlMaster=auto -o ControlPersist=60s.
File descriptors for that connection, its control path, and the pipes between the worker and the parent.
Memory to hold the module payload and the result. The result is the variable one: a task returning a large stdout holds all of it in the worker and again in the parent when it is collected.

Multiply by forks and the arithmetic stops being abstract. At forks: 200, that is 200 Python processes, 200 SSH clients, and 200 result buffers, concurrently, on one machine.

Under the hood

File descriptors deserve their own mention because the failure is abrupt and the error does not mention Ansible.

Each concurrent host needs several descriptors: the SSH connection, the control socket, and the pipes the worker uses to talk to the parent. The default soft limit on a Linux login session is commonly 1024, which is comfortable at forks: 20 and is not obviously comfortable at forks: 300.

Check the controller’s limits before raising forks into the hundreds:

ulimit -Sn            # soft limit for this shell
ulimit -Hn            # hard limit
cat /proc/sys/fs/file-max

When you run out, the run fails with Too many open files from somewhere unhelpful — inside a connection plugin, or from ssh itself — and the first suspicion is always the network.

The systemd unit that runs your scheduled playbooks has its own LimitNOFILE, independent of your login shell, which is why the same playbook can work interactively and fail from a timer. Resource limits, ulimit and how to raise them properly belong to the Linux course; what belongs here is knowing that forks consumes them and that the resulting error will not say so.

Three ceilings, not one

The controller is the ceiling people think about. It is rarely the first one you hit in production.

1. The controller

CPU, memory and file descriptors, as above. The signature is a controller that is visibly busy — load average above core count, memory pressure, swap — while the managed nodes are idle.

2. The rate at which your fleet accepts connections

Every fork opens an SSH connection. Two hundred forks means two hundred near-simultaneous connection attempts, and the things in the path have opinions about that:

sshd on each target enforces MaxStartups, which by default begins randomly dropping unauthenticated connections past a threshold.
A bastion or jump host multiplexing your whole fleet is one machine receiving all of them.
Firewalls and load balancers with connection-rate limits treat a burst of SSH from one source as exactly what it looks like.

The signature here is intermittent, random unreachable hosts — a different set each run — with a healthy controller and healthy targets.

This is the one that defeats the whole premise.

Twenty hosts running apt-get update in parallel against one repository mirror do not finish twenty times faster. They finish about as fast as the mirror can serve twenty concurrent clients, which may be slower than serving five.

The same applies to a licence server, an internal artefact store, a configuration API, a shared NFS mount, and a database that every host is about to migrate against. And on each individual host, a package manager holds an exclusive lock — so a play that runs two package tasks against the same host gains nothing from parallelism it was never going to have.

The signature is that the run does not speed up at all, and errors start arriving from the shared dependency rather than from Ansible: HTTP 429, mirror timeouts, Could not get lock.

Controller exhaustion presents as target-side failure

The break/fix worth recognising by shape:

A playbook that ran in 4 minutes at forks: 20 takes 11 minutes at forks: 100, with intermittent unreachable hosts. A different handful of hosts fails each run.

Every instinct points at the fleet. The error says UNREACHABLE. It names managed nodes. The obvious investigation is network, sshd, or those specific machines — and every one of those hosts, checked individually, is fine.

The cause is the controller. At forks: 100 it is swapping or saturated, its SSH clients cannot complete handshakes within timeout (default 10 seconds), and a connection that does not complete in time is reported as an unreachable host. The controller’s own slowness is being attributed to the machines it failed to reach.

The diagnostic is to watch the controller during the run, not the targets:

# on the controller, in another shell, while the slow run is in progress
vmstat 2 10
uptime
ps -eo rss,comm --sort=-rss | head -20

Load average well above core count, si/so columns non-zero in vmstat, or a wall of Python processes is the answer. The confirming test is cheaper still: re-run at the old fork count. If it goes back to 4 minutes, the fleet was never the problem.

Finding your own number

Read-only / Safethe fork-ceiling experiment— Run against a non-production group. --check keeps it read-only but changes the profile; use it to learn the shape and confirm the chosen value with a real staging run.

for f in 5 10 20 40 80; do
for pass in 1 2 3; do
  printf 'forks=%-4s pass=%s ' "$f" "$pass"
  /usr/bin/time -f '%e s' ansible-playbook -i inventory/staging site.yml \
    --check -f "$f" >/dev/null
done
done

Three passes per value, because of the noise demonstrated above. While it runs, watch the controller in another shell — the point where load average climbs past core count, or vmstat shows swap activity, is more informative than the wall-clock number.

Then apply the rule this course keeps returning to:

Raising forks without adding a real limit removes protection

The measurement tells you what the controller can sustain. It says nothing about what your service can sustain.

forks: 5 is currently limiting how many hosts are in a half-configured state at once. Raising it to 40 because the controller can cope multiplies simultaneous exposure eightfold, and the measurement gave you no reason to believe that is safe.

Raise forks and introduce the deliberate limit in the same change:

- name: Roll the web tier
  hosts: web:&live
  serial: 4               # blast radius, expressed in service terms
  max_fail_percentage: 0

forks becomes the ceiling on what the controller is willing to do; serial becomes the limit on what the play is allowed to do. Those are different questions with different right answers, and conflating them is how a performance change becomes an incident.

What happens if forks is higher than the number of hosts in the batch?

Nothing bad, and nothing useful. forks is a ceiling, not a target: at most one worker starts per host in the current batch, so forks: 50 against 12 hosts runs 12 workers.

Two consequences follow.

A generous forks costs nothing on small runs, so “it will slow down my one-host runs” is not an argument against raising it. The entire case for and against lives in the large runs.

And serial wins. A play with serial: 4 never has more than four hosts in flight regardless of forks, because the batch has four hosts in it. The keyword documentation for throttle states the same relationship from the other direction: it “cannot be set higher than those limits”, so forks: 10 with throttle: 15 still operates on at most ten hosts in parallel. That was measured in this part’s throttle lesson: throttle: 4 with forks: 1 took the fully serialised time.

The mental model that keeps all three straight: forks is the controller’s capacity, serial is the play’s blast radius, throttle is a shared dependency’s tolerance. The smallest one governs.

A number, and the reason for it, in the same place

Whatever value you settle on, write down what it was measured against.

[defaults]
# Measured 2026-08-11 on ctrl-1 (12 cores, 46 GiB): near-linear to 8,
# flat from 8 to 32 with run-to-run noise larger than the difference.
# 16 chosen for headroom. Re-measure if the controller is resized or if
# fleet size changes materially. Blast radius is set by serial in the
# plays, not by this value.
forks = 16

A bare forks = 200 is indistinguishable from somebody having typed a big number, and it will be inherited by whoever resizes the controller down next year. The comment is the difference between a measured setting and a guess that survived.

Knowledge check

Why this matters

Fork tuning is the first thing people reach for and one of the last things that turns out to matter. The measured curve here flattened at roughly the controller’s core count, and beyond that the differences were smaller than the noise — on a workload deliberately chosen to be fork-bound. On a real playbook, where fact gathering or one slow task usually dominates, the flattening arrives even earlier.

Meanwhile the costs of overshooting are real and they arrive disguised. A saturated controller reports unreachable managed nodes. A burst of connections trips MaxStartups on hosts that are perfectly healthy. Twenty hosts hammering one mirror go no faster and start failing in ways that look like package problems.

So: measure your own ceiling, write the number down with the reason, and remember that the number you found describes what your controller can do — not what your service can survive. That second limit is serial, and it has to move in the same commit.