throttle: protecting shared dependencies · Ansible

The previous lesson named three ceilings on a run: the controller, the rate at which the fleet accepts connections, and the shared things every host depends on. forks is the lever for the first. throttle is the lever for the third, and it is the only one of the concurrency keywords that is about a machine which is not in your play.

The shape it addresses:

Two hundred hosts run a package task. All two hundred fetch from one internal mirror. The mirror serves five concurrent clients comfortably, forty badly, and two hundred not at all — and the errors come back as timeouts that look like a network problem.

Lowering forks would fix it and would slow down every other task in the run. throttle lowers concurrency for the tasks that need it and leaves the rest alone.

What it does, measured

Four hosts, one task that takes two seconds each, run four ways:

Read-only / Safethrottle bounds concurrency and cannot raise it— ansible-core 2.21.3. wait_for with only a timeout sleeps and touches nothing; all four hosts use local connections.

$ /usr/bin/time -f 'elapsed %e s' ansible-playbook -i thr.ini thr.yml -f <N> [-e thr=<N>]

-- forks 5, no throttle --
elapsed 2.48 s

-- forks 5, throttle 1 --
elapsed 8.91 s

-- forks 1, no throttle --
elapsed 8.92 s

-- forks 1, throttle 4 (cannot raise above forks) --
elapsed 8.91 s

Four hosts times two seconds is eight seconds of work. Done in parallel it takes 2.5 seconds; serialised it takes 8.9. throttle: 1 at forks: 5 produced the serialised time, which is the keyword working.

The fourth measurement is the one worth keeping. throttle: 4 with forks: 1 still took 8.9 seconds — asking for more concurrency than forks allows achieves nothing. The keyword documentation states it directly:

Read-only / Safethe definition on your own controller— ansible-doc reads the installed ansible-core, so this is the definition your runs are using.

$ ansible-doc -t keyword throttle

throttle:
applies_to:
- Play
- Role
- Block
- Task
- Handler
description: Limit the number of concurrent task runs on task, block and playbook
  level. This is independent of the forks and serial settings, but cannot be set
  higher than those limits. For example, if forks is set to 10 and the throttle
  is set to 15, at most 10 hosts will be operated on in parallel.
priority: 0
template: explicit
type: int

throttle only ever lowers concurrency. It is a floor-ward adjustment, never a ceiling-ward one, which makes it safe to add: the worst case of a wrong value is a run that is slower than it needed to be.

Where to put it

The keyword applies to Play, Role, Block, Task and Handler, and the level you choose changes what is protected.

On the task

The narrow, common case. One task touches the shared thing; everything else in the play runs at full concurrency.

Configuration changeone task, one mirror— A package installation is a configuration change on every targeted host. The throttle limits how many hit the mirror at once; it does not limit how many hosts are changed by the run.

- name: Install the agent from the internal mirror
ansible.builtin.package:
  name: monitoring-agent
  state: present
throttle: 5

On a block

The case people miss. If a sequence of tasks holds the shared resource, throttling one of them leaves the others racing.

Service impact possiblea whole critical section, serialised— Checking out a licence, using it, and returning it must happen as a unit. Throttling the block keeps one host inside the section at a time.

- name: Licensed reconfiguration
block:
  - name: Check out a licence seat
    ansible.builtin.uri:
      url: "https://licence.example.com/api/checkout"
      method: POST
    register: seat

  - name: Apply the licensed configuration
    ansible.builtin.command: /opt/vendor/bin/reconfigure --seat {{ seat.json.id }}

  - name: Return the seat
    ansible.builtin.uri:
      url: "https://licence.example.com/api/release/{{ seat.json.id }}"
      method: POST
throttle: 1

With throttle: 1 on the individual checkout task instead, host A would check out a seat, and while it was running reconfigure, host B would be free to check out the next one. The pool would empty exactly as fast as it would have without the keyword.

On a handler

Handlers inherit nothing from the tasks that notified them, so a handler that restarts a service which registers with a shared discovery system needs its own throttle:

handlers:
  - name: restart app
    ansible.builtin.systemd:
      name: app
      state: restarted
    throttle: 4

On a play or role

Blunt but occasionally right — a role that exists entirely to talk to one API, for example. Prefer the narrowest level that covers the resource, because a play-level throttle slows down every task in the play including the ones that touch nothing shared.

throttle is not a blast-radius control

throttle: 1 looks like the safest possible setting and it is not a safety control at all.

It bounds how many hosts run a task simultaneously. It does not bound how many hosts the run changes. A play over 400 hosts with throttle: 1 on every task changes 400 hosts — one at a time, slowly, and completely.

That distinction matters because the slowness is reassuring in a way the behaviour does not justify. An operator watching a throttle: 1 run tick through hosts one by one has plenty of time to notice something wrong, and that is a real benefit. But if nobody is watching, the outcome is identical to the unthrottled run and it took eleven hours.

The blast-radius controls remain what they have been throughout this course: the host pattern, --limit, serial, and max_fail_percentage. serial: 4 stops the play after a failed batch. throttle: 4 does not.

`throttle`, `serial` and `run_once` are three different questions

They get confused constantly, and the confusion produces real defects.

Keyword	Question it answers	Effect on a 400-host play
`forks`	How much can the controller do at once?	Ceiling on workers, fleet-wide
`serial`	How many hosts may be mid-change at once, and where do we stop?	Batches; failure policy is evaluated at batch boundaries
`throttle`	How many hosts may run this task at once?	Lowers concurrency for the throttled scope only
`run_once`	Should one host do this on behalf of the batch?	One execution per batch; see the delegation part

The two substitutions that go wrong:

run_once where throttle: 1 was meant. Each host needs to register itself with a catalogue, and the catalogue tolerates one writer. run_once registers one host and leaves 399 unregistered — a correctness bug, and a quiet one, because the task reports ok.

throttle: 1 where serial: 1 was meant. The intent was “change one host at a time, and stop if one fails”. throttle: 1 gives the first half and not the second: with the default failure handling, other hosts continue after one fails, and there is no batch boundary at which max_fail_percentage is evaluated.

Under the hood

throttle is templated — the keyword definition says template: explicit and type: int — so the value can come from a variable. That is more useful than it first appears, because the correct value is a property of the dependency, not of the play.

# inventory/production/group_vars/all.yml
mirror_max_concurrency: 5
licence_pool_size: 1

# inventory/staging/group_vars/all.yml
mirror_max_concurrency: 20     # staging mirror is not shared with anyone
licence_pool_size: 3

- name: Install from the mirror
  ansible.builtin.package:
    name: monitoring-agent
    state: present
  throttle: "{{ mirror_max_concurrency | int }}"

Two benefits. The number lives next to the environment it describes, so staging is not throttled to production’s limits. And when the mirror is upgraded, one variable changes instead of every role that touches it.

The | int is not decoration. Variables arriving from group_vars are often strings, and the keyword is typed; making the conversion explicit avoids a class of confusing type error at parse time.

One limitation to be aware of: because the value is resolved per task rather than negotiated with anything, throttle is a static guess about what the dependency tolerates. It cannot react to the mirror being busy. If the shared resource has a real queueing or rate-limit protocol, using it — until with retries on a 429, a token bucket in a wrapper script — is better than picking a number, and throttle becomes the backstop rather than the mechanism.

What happens if you set throttle higher than the number of hosts?

Nothing. Like forks, it is a ceiling rather than a target: three hosts with throttle: 20 run three at a time, subject to forks and serial.

The more interesting case is throttle interacting with serial, because both are lowering concurrency and the smaller one wins.

With serial: 3 and throttle: 5, at most three hosts are in the batch, so at most three run the task. The throttle has no effect. With serial: 10 and throttle: 2, the batch has ten hosts and the throttled task runs two at a time within it — while every other task in that play runs at ten.

That combination is the normal one in a rolling deployment, and reading it correctly matters: serial is setting the blast radius, and throttle is protecting a dependency inside it. They are not competing, and neither is a substitute for the other.

Find the number from the dependency, not from the run

The right throttle value is a property of the shared thing, and it is usually documented or measurable there rather than discoverable from Ansible:

A package mirror or artefact store — the web server’s worker or connection limit, and whatever the operations team says it is sized for.
A rate-limited API — the documented limit. If it is 10 requests per second and each task takes 200 ms, a throttle of 2 keeps you under it.
A licence server — the seat count. This one is exact.
A database you are migrating against — the connection pool size, minus whatever the application needs to stay alive.

The wrong method is to keep halving the number until the errors stop. That finds a value that works on a quiet afternoon and fails during a busy one, because the dependency was already serving other clients you did not account for.

Where you cannot find a number, throttle: 1 on the critical section and a measurement of how long the run then takes is a defensible starting point. Slow and correct is a better place to negotiate from than fast and intermittently broken.

Knowledge check

Why this matters

Most concurrency tuning is about the controller and the fleet, both of which you own and can measure. throttle is about the things in between — the mirror, the licence server, the internal API, the database — which are usually owned by somebody else, sized for a steady load, and not expecting two hundred simultaneous clients from one source at 02:00 on a Sunday.

Those dependencies produce the least informative failures in the whole system. A mirror timing out looks like a network problem. An API returning 429 looks like an authentication issue to anyone reading the module’s error. A licence server refusing a checkout looks like a broken host. Every one of them sends the investigation to the wrong place.

One line, at the narrowest scope that covers the resource, with the number taken from the dependency rather than guessed. It costs a little wall-clock time and removes a class of failure that is genuinely hard to diagnose from the Ansible side.