Return to service and the soak interval · Ansible

The drain created an obligation. This is where it is discharged.

Two things have to be true before the batch is finished: the host is back in the load balancer, and it is actually serving traffic. Those are different claims, and only the first one is easy.

The enable call is not the verification

Service impact possibleputting the host back— Re-enables the backend. This returns real user traffic to the host, so it is the point of no return for the batch.

    - name: Return this host to the load balancer pool
    ansible.builtin.uri:
      url: >-
        https://lb.example.com/api/v1/pools/{{ app_pool }}/members/{{ inventory_hostname }}
      method: PATCH
      body_format: json
      body:
        state: active
      headers:
        Authorization: 'Bearer {{ lb_api_token }}'
      status_code:
        - 200
        - 204
    delegate_to: localhost
    changed_when: true

A 204 means the load balancer accepted the request. It does not mean the backend is in the pool, that its health check has passed, or that a single request has reached it.

Most load balancers will not send traffic to a backend until their own health check has passed against it, and that check runs on the balancer’s schedule — commonly every five to thirty seconds, sometimes requiring several consecutive successes. So there is a window, after your enable call returns 204, during which the host is enabled and receiving nothing.

If the play proceeds to drain the next batch during that window, capacity is lower than you think. Do it across enough batches and you can drain your way into an outage while every task reports success.

Read-only / Safewaiting for the balancer to agree the host is healthy— Read-only: a GET against the balancer API, retried until the member reports healthy and in service. This is the verification the enable call does not provide.

    - name: Wait for the balancer to mark this backend healthy and in service
    ansible.builtin.uri:
      url: >-
        https://lb.example.com/api/v1/pools/{{ app_pool }}/members/{{ inventory_hostname }}
      method: GET
      headers:
        Authorization: 'Bearer {{ lb_api_token }}'
      return_content: true
    delegate_to: localhost
    register: member
    retries: 20
    delay: 3
    until:
      - member.json.state | default('') == 'active'
      - member.json.health | default('') == 'healthy'
    changed_when: false

Again the default() values are chosen to fail the condition. An absent field must not read as success.

Proving traffic is arriving

The strongest form asks the host itself whether requests are reaching it. Any service that exposes a request counter can answer:

Read-only / Safethe host confirms it is being asked for things— Read-only. Records a baseline counter, waits, then asserts the counter moved. Proves real traffic is arriving rather than that a configuration API returned 204.

    - name: Record the request counter before waiting
    ansible.builtin.uri:
      url: 'http://{{ ansible_host | default(inventory_hostname) }}:8080/metrics'
      return_content: true
    register: metrics_before
    changed_when: false

  - name: Give the balancer time to route requests here
    ansible.builtin.pause:
      seconds: '{{ traffic_settle_seconds | default(20) }}'

  - name: Confirm the request counter has moved
    ansible.builtin.uri:
      url: 'http://{{ ansible_host | default(inventory_hostname) }}:8080/metrics'
      return_content: true
    register: metrics_after
    retries: 10
    delay: 5
    until: >-
      (metrics_after.json.requests_total | default(0) | int)
      > (metrics_before.json.requests_total | default(0) | int)
    changed_when: false

The step that must never be skipped

Part XXXI established that when a failure policy triggers, the play stops at the end of the failing task and the surviving hosts run no further tasks. A return-to-service step written as an ordinary final task is exactly such an unreached task.

Service impact possiblethe whole batch in a block, with the return in always— The always section is dispatched however the block ended. Healthy hosts go back into the pool; unhealthy ones are recorded as out of service so the capacity loss is announced rather than silent.

    - name: Deploy this host, returning it to service whatever happens
    block:
      - name: Drain, deploy, restart and health-check
        ansible.builtin.include_tasks: deploy-one-host.yml

    always:
      - name: Return a healthy host to the load balancer
        ansible.builtin.uri:
          url: >-
            https://lb.example.com/api/v1/pools/{{ app_pool }}/members/{{ inventory_hostname }}
          method: PATCH
          body_format: json
          body:
            state: active
          headers:
            Authorization: 'Bearer {{ lb_api_token }}'
          status_code:
            - 200
            - 204
        delegate_to: localhost
        changed_when: true
        when: health.json.version | default('') == release_version

      - name: Announce a host left out of service
        ansible.builtin.debug:
          msg: >-
            CAPACITY WARNING: {{ inventory_hostname }} failed its health
            check and remains OUT OF SERVICE in pool {{ app_pool }}.
            The fleet is running with reduced capacity until this host is
            repaired or manually returned.
        when: health.json.version | default('') != release_version

The soak interval

The last step of the loop is doing nothing, on purpose.

Read-only / Safethe deliberate pause between batches— Read-only. Runs once per batch at the end of the task list, so it delays the next batch rather than each host.

    - name: Soak before starting the next batch
    ansible.builtin.pause:
      seconds: '{{ soak_seconds | default(120) }}'
    run_once: true
    when: not (skip_soak | default(false) | bool)

The health check answers “is this host working now”. A whole class of failures does not present immediately:

A memory leak in the new version that takes minutes to exhaust the heap, then crashes the process.
A connection pool that is fine until it fills, which depends on traffic volume over time.
A cache that starts cold and is fine until the first cache-expiring request pattern arrives.
A scheduled job the new version runs on a timer, which fails the first time it fires.
A slow resource leak — file descriptors, threads, temporary files.

Without a soak, a rolling deploy can put the new version on the entire fleet before any of these has had time to appear on the first host. The canary was healthy when you checked it, and by the time it fell over, all 200 hosts were running the same code.

The soak is what converts the canary from “it started successfully” into “it survived a realistic period of production traffic”.

Choosing the length. Long enough for the failure mode you are most afraid of, and no longer. Two minutes is a reasonable default for a web application, and it is a default rather than an answer — if the leak you have seen before takes fifteen minutes to manifest, a two-minute soak does not protect you against it.

Soak after the canary, not after every batch

A uniform soak on every batch is expensive: 120 seconds across eleven batches is 22 minutes of deliberate waiting, and long change windows get compressed under pressure.

The information is not uniformly distributed either. The first batch tells you the most — it is the first exposure of new code to production traffic. By batch eight you have had the new version running for half an hour on a quarter of the fleet, and another two-minute pause tells you almost nothing.

Weight it accordingly. There is no “batch number” variable, but the first batch is identifiable: it is the one containing the first host of the play.

    - name: Soak, longest after the canary batch
      ansible.builtin.pause:
        seconds: >-
          {{ 300 if ansible_play_hosts_all[0] in ansible_play_batch else 60 }}
      run_once: true

Verified on ansible-core 2.21.3 with serial: [1, 2, 3] over six hosts:

batch=['h01']               first_batch=True   soak=300
batch=['h02', 'h03']        first_batch=False  soak=60
batch=['h04', 'h05', 'h06'] first_batch=False  soak=60

This relies on ansible_play_hosts_all keeping the full play host list regardless of batching, which Part XXXI established. It also relies on the first host of the play being in the first batch, which holds because batching slices the host list positionally — and stops holding under order: shuffle, one more reason not to shuffle a play that changes things.

Under the hood

run_once on the soak task is doing precise work, and it depends on the Part XXXI finding.

run_once executes on the first host of the current batch, not the first host of the play — verified there, where serial: 2 over six hosts ran a run_once task three times, on h01, h03 and h05. For most tasks that behaviour is a trap. For a soak it is exactly right: you want one pause per batch, not one per host, and not one per play.

Without run_once, the pause would execute once per host in the batch. Whether that serialises depends on the strategy — under the default linear strategy, all hosts in the batch run the task before the play advances, so a 120-second pause on a five-host batch does not become ten minutes. But it does produce five pause events, and with the interactive form of pause it would prompt repeatedly. run_once makes the intent explicit rather than relying on strategy behaviour.

The pause module is one of the few that genuinely does nothing on the managed node — it is an action plugin that sleeps on the controller. That is why it works with gather_facts: false and against hosts that would otherwise be unreachable, and it is why a pause is not a substitute for a wait_for: it observes nothing about the host it is nominally running against.

What happens if the load balancer forgets the host is enabled?

Some balancers reload their configuration from a file and discard runtime state, which silently undoes your enable call.

The scenario: your play enables the backend through the runtime API. An hour later, an unrelated change causes the balancer to reload from its on-disk configuration — which still lists that backend as disabled, because your play never wrote it there. The host drops out of the pool and nothing reports it.

This is a real class of bug with HAProxy runtime socket commands and similar mechanisms, where runtime state and configuration state are separate.

Two defences. Write both, if the balancer distinguishes them — update the configuration file and apply the runtime change, so a reload converges to the same state. Or make the configuration the source of truth and have the deploy modify the file and reload the balancer, accepting the extra reload per batch.

The general principle is the one the drift part covers: if your automation puts a system into a state that its own configuration does not describe, that state has an expiry date nobody has written down.

Knowledge check

Why this matters

The return to service is where rolling deployments fail quietly, and quietly is the operative word.

A failed health check is loud. A failed package install is loud. A host that was drained and never returned makes no sound at all: it is up, it is healthy, its metrics look normal because it is doing nothing, and the only system that knows it is missing is the load balancer, which nobody opens after reading an Ansible log.

The named incident for this part is a capacity drop of a third at 03:00 that nobody noticed until the morning peak. It was not caused by a bug in the deployment. It was caused by a deployment that failed in an ordinary way and left a step undone, in a design where no path guaranteed that step would run.

So the discipline is: the drain incurs an obligation, the obligation is discharged in an always, and the discharge is verified rather than assumed. And because even always does not survive a killed controller, something outside the run has to be able to answer “is any host drained right now” — on a calm afternoon, before you need it.