Rolling a deployment through a load balancer · Ansible

Part XXXII built the rolling loop: drain, change, health-check, return, soak. This lesson is about composing that loop with the deployment content of this part, and about two keywords whose semantics are per batch rather than per run — which is the source of the two most expensive mistakes in a rolling deploy.

The composition

Service impact possiblea rolling application deploy, whole— Removes each batch from the load balancer, deploys, health-checks and returns it. Every host in the batch is out of the pool simultaneously, so the serial value is a capacity decision.

- name: Roll the application through the fleet
hosts: appservers
become: true
serial: '{{ deploy_serial | default([1, 2, "25%"]) }}'
max_fail_percentage: 0
tasks:
  - name: Deploy this host, returning it to the pool whatever happens
    block:
      - name: Drain this backend from the load balancer
        ansible.builtin.uri:
          url: >-
            https://lb.example.com/api/v1/pools/{{ app_pool }}/members/{{ inventory_hostname }}
          method: PATCH
          body_format: json
          body:
            state: drain
          headers:
            Authorization: 'Bearer {{ lb_api_token }}'
          status_code: [200, 204]
        delegate_to: localhost
        throttle: 2
        changed_when: true

      - name: Wait for in-flight requests to complete
        ansible.builtin.wait_for:
          host: '{{ ansible_host | default(inventory_hostname) }}'
          port: 8080
          state: drained
          timeout: 120

      - name: Converge package, configuration and service
        ansible.builtin.include_role:
          name: myapp

      - name: Prove this host serves the version we deployed
        ansible.builtin.include_tasks: health-gate.yml

    always:
      - name: Return healthy hosts, announce the rest
        ansible.builtin.include_tasks: undrain.yml

  - name: Soak before the next batch
    ansible.builtin.pause:
      seconds: '{{ soak_seconds | default(120) }}'
    run_once: true

Three structural decisions in there are worth naming.

The drain and the undrain are delegate_to: localhost. They are API calls to the load balancer, made from the controller, about the current host. The host being deployed may be unreachable at that moment, and it certainly has no business talking to the balancer’s control plane.

The whole per-host sequence is a block with an always. The drain creates an obligation; the always discharges it however the block ended, including when a failure policy is about to stop the play.

The soak is run_once — correct here, because one pause per batch is exactly the intent.

`throttle`: fewer workers for one task

throttle limits the number of workers for a single task or block. Upstream describes the use case as restricting “tasks that may be CPU-intensive or interact with a rate-limiting API”, which is precisely what a load-balancer control plane is.

The constraint is stated just as plainly: “If you have already restricted the number of forks or the number of machines to execute against in parallel, you can reduce the number of workers with throttle, but you cannot increase it.”

So throttle is a one-way valve. It can narrow, never widen.

Setting	Effect on a `serial: 10` batch with `forks: 5`
`throttle: 2`	2 hosts run the task at a time
`throttle: 5`	no change; already capped at 5 by `forks`
`throttle: 20`	no change; cannot exceed `forks` or the batch

`run_once` is per batch, and the migration is the casualty

Upstream: “When used together with serial, tasks marked as run_once will be run on one host in each serial batch.”

This was verified on ansible-core 2.21.3, with serial: [1, 3, 6] over ten hosts and a run_once debug task:

Read-only / Saferun_once under serial, executed on 2.21.3— Read-only. Ten local-connection hosts; only the debug and assert modules ran. The task marked run_once printed three times, once per batch.

$ ansible-playbook -i inv10.ini batchstop.yml

PLAY [Validation failure stops the next batch] **********************************
"msg": "BATCH h01 of 10 total"

PLAY [Validation failure stops the next batch] **********************************
"msg": "BATCH h02,h03,h04 of 10 total"

PLAY [Validation failure stops the next batch] **********************************
"msg": "BATCH h05,h06,h07,h08,h09,h10 of 10 total"

For a soak, three executions is the intent. For a schema migration it is a production incident, and the mechanism produces it silently: the migration runs on batch one, succeeds; runs again on batch two against an already-migrated database, and either errors — stopping the deployment mid-fleet — or, worse, succeeds and applies something twice.

Read-only / Safethe two idioms— Read-only. The first runs once per batch. The second is true for exactly one host in exactly one batch, because ansible_play_hosts_all holds the whole play host list regardless of batching.

# Once per batch. Correct for a soak or a per-batch banner.
- name: Soak before the next batch
ansible.builtin.pause:
  seconds: 120
run_once: true

# Once per run. Correct for anything fleet-wide.
- name: Announce the deployment start
ansible.builtin.uri:
  url: 'https://chat.example.com/api/messages'
  method: POST
  body_format: json
  body:
    text: 'Deploying {{ myapp_version }} to {{ app_pool }}'
  headers:
    Authorization: 'Bearer {{ chat_token }}'
delegate_to: localhost
when: inventory_hostname == ansible_play_hosts_all[0]

ansible_play_hosts_all is documented as every host the play targets, unaffected by batching, so testing inventory_hostname against its first element is true once.

A schema migration does not belong in the batched play at all

The when: inventory_hostname == ansible_play_hosts_all[0] idiom makes a task run once. It does not make it run at the right time.

A migration inside a rolling play runs during batch one, which means it runs while batches two onward are still serving the old application version against the new schema. Whether that is survivable is a property of the migration, not of Ansible, and it is what the expand-and-contract discipline exists to guarantee.

The structurally correct shape is three plays in one playbook:

- name: Migrate the schema
  hosts: db_primary
  tasks: [...]        # unbatched, one host, gated on a verified backup

- name: Roll the application
  hosts: appservers
  serial: [1, 2, '25%']
  tasks: [...]

- name: Contract the schema
  hosts: db_primary
  tasks: [...]        # after every app host runs the new version

Three plays, three different host groups, no batching where batching would be wrong. The database lesson later in this part is about what goes inside the first and third.

There is one more trap in the idiom itself: if the first host of the play has failed and left the active set, the condition is true for nobody and the task silently never runs. For an announcement that is an annoyance. For “release the deployment lock” it is a lock nobody holds and nobody released.

Delegation: whose variables, whose facts

delegate_to changes where the task executes, not which host the task is about. That distinction produces the two recurring confusions.

Variables come from the original host. In the drain task above, inventory_hostname, app_pool and ansible_host all resolve from the host being deployed, even though the task runs on the controller. This is what makes the task expressible at all — it is a statement about web07 executed from localhost.

Facts, by default, do not transfer. A task delegated to the load balancer host that gathers facts assigns them to the original host unless delegate_facts: true is set, which is the source of a memorable bug: ansible_facts.hostname on web07 becomes the load balancer’s hostname, and it stays that way for the rest of the play.

Read-only / Safedelegation without contaminating the original host's facts— Read-only. Gathers facts about the load balancer and stores them against the load balancer, not against the host currently being deployed.

- name: Gather facts about the load balancer itself
ansible.builtin.setup:
  gather_subset: ['!all', '!min', 'network']
delegate_to: '{{ lb_host }}'
delegate_facts: true
run_once: true

delegate_to: localhost also means the task uses the controller’s Python, the controller’s installed libraries and the controller’s network position. A uri task delegated to localhost needs the controller to be able to reach the load balancer API, which is an assumption worth checking before the change window rather than during it.

Under the hood

Why throttle cannot raise parallelism, mechanically.

The linear strategy maintains a worker pool sized by forks. For each task it takes the current batch’s host list and dispatches work items to free workers. throttle sets a smaller ceiling for that one task: the strategy dispatches at most throttle items concurrently and waits.

There is no path by which the strategy can exceed the pool it was given. forks is the pool; serial bounds the host list; throttle bounds dispatch within those. Each is a narrower constraint applied later, and narrowing is the only operation available.

Two practical consequences.

Throttle costs wall-clock time proportionally. A throttle: 1 task in a serial: 25 batch runs twenty-five times sequentially. If the task takes eight seconds, that is over three minutes added per batch, and the deployment’s duration is now dominated by the one task you throttled. Throttle to the largest value the downstream service tolerates, not to 1 out of caution.

Throttle does not serialise across batches. It bounds concurrency within the task’s dispatch for the current batch. Combined with the per-batch play restart, that means a throttle: 2 on a drain task in a serial: 10 play produces at most two concurrent API calls at a time within each batch, and each batch starts fresh.

run_once is implemented in the same layer and by a related mechanism: the strategy selects the first host of the current batch, runs the task there, and copies the result and any facts to every other host in the batch. That copying is why run_once combined with register makes the registered variable available on every host in the batch — a genuinely useful property — and why “the batch” rather than “the play” is the unit throughout.

What happens if the load balancer API is unreachable when the play starts?

The first drain task fails, and where it fails decides whether you have a stopped deployment or a fleet with a hole in it.

If the drain is the first task inside the block, the failure occurs before any host has been changed. The always runs, attempts the undrain, and that fails too — but it fails trying to return a host that was never removed, so nothing is out of the pool. With max_fail_percentage: 0 the play stops on batch one, the fleet is untouched, and the diagnosis is a controller that cannot reach the balancer.

That is the good arrangement, and it is worth designing for explicitly with a pre-flight check in an unbatched play:

- name: Confirm the controller can reach the load balancer API
  ansible.builtin.uri:
    url: 'https://lb.example.com/api/v1/pools/{{ app_pool }}'
    headers:
      Authorization: 'Bearer {{ lb_api_token }}'
  delegate_to: localhost
  run_once: true
  changed_when: false

The bad arrangement is a balancer that is reachable at drain time and unreachable at undrain time — a token that expired mid-run, a control plane that restarted, a network partition. Then hosts are drained, the always cannot return them, and capacity drops silently with every batch.

Two defences. Make the undrain failure loud: an always section that emits a CAPACITY WARNING line naming every host left out of service, so the run log contains the fact rather than only its absence. And keep a census playbook — something you can run at any time that lists every backend not in the pool. Part XXXII makes the case for it; this is the failure that needs it.

Knowledge check

Why this matters

The rolling deploy is the shape most organisations converge on, and the two keywords in this lesson are where it goes wrong quietly rather than loudly.

throttle set too low turns a twenty-minute deployment into a ninety-minute one, and the cost is paid in change windows compressed under pressure, which is how batches get skipped. run_once on a migration produces an incident whose cause is genuinely hard to see from the run output, because the task did what it says on the label.

Both come from the same root: serial restarts the play, so anything whose scope is “the play” is really “the batch”. run_once, pre_tasks, post_tasks, the handler flush, gather_facts — all of them.

Once that is internalised, the design rule follows without needing to be memorised. Anything that must happen once per fleet goes in an unbatched play. Anything that must happen once per batch goes in the batched play with run_once. Anything that must happen per host is an ordinary task.

Three categories, and putting a task in the wrong one is how a rolling deployment becomes a fleet-wide event.

The composition

throttle: fewer workers for one task

run_once is per batch, and the migration is the casualty

Delegation: whose variables, whose facts

Knowledge check

`throttle`: fewer workers for one task

`run_once` is per batch, and the migration is the casualty