Scheduling and the thundering herd

The first fleet-wide patch run most organisations do is also the first time they take down their own package mirror.

The playbook is fine. The change is fine. Two thousand hosts request four hundred megabytes of packages within the same ninety seconds, the mirror’s connection limit is reached, half the hosts get a timeout, and the run reports several hundred failures that look like a package problem and are actually a load problem you created.

That is the thundering herd, and at fleet scale it is not an edge case. It is the default outcome of running the same thing on every host at once.

What a fleet run concentrates onto

Every fleet-wide run has a small number of choke points. Naming them before the run is most of the work.

The last row is the one people forget, and Part XXXIV covers it properly. The others are external, and the run is the load event.

Three tools, three different jobs

They are routinely confused, and they do not substitute for each other.

serial shapes how many hosts are inside the play at a time. It is a play-level keyword and it applies to the whole play — every task, for the whole batch.

throttle shapes how many hosts run one particular task at a time. It applies at play, role, block, task and handler level, and it can only lower concurrency, never raise it: it cannot exceed forks or serial.

Jitter shapes when a scheduled run starts on each host, or when a particular task begins. It is the only one of the three that helps when the concentration comes from many independent runs rather than one big one.

- name: patch the web tier
hosts: wave3_domain_a
serial: 200
tasks:
  - name: refresh package metadata and upgrade
    ansible.builtin.package:
      name: nginx
      state: latest
    throttle: 20          # never more than 20 clients on the mirror

  - name: render the configuration
    ansible.builtin.template:
      src: nginx.conf.j2
      dest: /etc/nginx/nginx.conf
      mode: '0644'
    notify: reload nginx

The distinction is worth stating plainly: dropping serial to 20 to protect the mirror also slows every other task in the play by a factor of ten. throttle on the one task that touches the shared resource costs you nothing anywhere else.

Jitter that is stable per host

The standard idiom is to give each host a random offset. The naive version — a fresh random number every run — has a real drawback: a host’s slot moves every night, so you can never say “web014 patches at 02:07”, and correlating an incident with a run becomes guesswork.

Seeding the random filter on the hostname fixes that. Verified on 2.21.3:

$ ansible-playbook -i inventory.ini jitter.yml

ok: [web01] => {"msg": "web01 -> 181s"}
ok: [web02] => {"msg": "web02 -> 249s"}
ok: [web03] => {"msg": "web03 -> 3s"}
ok: [web04] => {"msg": "web04 -> 125s"}
ok: [web05] => {"msg": "web05 -> 176s"}
ok: [web06] => {"msg": "web06 -> 162s"}
ok: [web07] => {"msg": "web07 -> 54s"}
ok: [web08] => {"msg": "web08 -> 188s"}
ok: [web09] => {"msg": "web09 -> 249s"}
ok: [web10] => {"msg": "web10 -> 76s"}

The expression is {{ 300 | random(seed=inventory_hostname) }}: an integer from 0 to 299, derived from the hostname, identical on every run.

Two ways to use it, and they are not equivalent.

# (a) inside a play: stagger the moment each host hits the mirror
- name: stagger before touching the shared mirror
ansible.builtin.wait_for:
  timeout: "{{ 300 | random(seed=inventory_hostname) }}"

# (b) on the schedule: spread independently-triggered runs
- name: nightly convergence, staggered across the hour
ansible.builtin.cron:
  name: ansible-converge
  minute: "{{ 60 | random(seed=inventory_hostname) }}"
  hour: '2'
  job: /usr/local/bin/converge.sh

Windows, and why “02:00” is not a time

On a fleet in one building, 02:00 is a quiet moment. On a fleet across three regions it is three different moments, and scheduling by UTC or by local time are meaningfully different decisions:

• By UTC, every region runs simultaneously. The herd is maximal and the shared dependencies see the whole fleet at once — but the change lands everywhere at a single, easily-recorded instant.
• By local time, load is naturally spread across the day, but the fleet is in a mixed state for many hours, and “is the change live?” has no single answer while it rolls.

Neither is wrong. What is wrong is not choosing, because the default — cron entries in each host’s local time, applied by a role that assumed one time zone — gives you the mixed-state downside without anyone having decided to accept it.

Additional scheduling constraints worth writing down explicitly:

• Business hours for the service, which may not be the business hours of the team running the change.
• Change freeze periods: quarter end, retail peaks, election nights, regulatory reporting dates.
• Backup windows, which compete for the same I/O and network the run needs.
• Other automation. A patching run and a compliance-scan run that both start at 02:00 are two herds, and neither team knows about the other.

Knowledge check