Draining a host from the load balancer · Ansible

Draining is two operations that are easy to confuse: telling the load balancer to stop sending new requests, and waiting for the requests already in flight to finish.

The first is instant and the load balancer will confirm it. The second takes as long as your slowest request, and nothing confirms it unless you ask.

Implementations that skip the second step restart a service while it is mid-response to real users. The requests fail, the users see errors, and the run log records a successful deployment — because from Ansible’s point of view the disable call returned zero and the restart returned zero.

Telling the load balancer

The task runs against the load balancer while the play is iterating over application servers, which is what delegate_to is for.

Service impact possibledraining via a control command on the balancer— Takes a production host out of rotation. delegate_to sends the task to the load balancer; inventory_hostname still refers to the host being deployed, which is what makes the command target the right backend.

    - name: Remove this host from the load balancer pool
    ansible.builtin.command: >-
      /usr/local/bin/lb-ctl disable {{ inventory_hostname }}
    delegate_to: '{{ load_balancer_host }}'
    changed_when: true

For a balancer with an HTTP API, uri is the better instrument, and it runs from wherever you delegate it:

Service impact possibledraining via the balancer API— Same effect through an API. The token comes from a vault-encrypted variable; it is never written into the playbook. status_code accepts both a fresh change and a host that was already disabled.

    - name: Set this backend to draining
    ansible.builtin.uri:
      url: >-
        https://lb.example.com/api/v1/pools/{{ app_pool }}/members/{{ inventory_hostname }}
      method: PATCH
      body_format: json
      body:
        state: draining
      headers:
        Authorization: 'Bearer {{ lb_api_token }}'
      status_code:
        - 200
        - 204
      validate_certs: true
    delegate_to: localhost
    changed_when: true

Two details in that example are deliberate.

delegate_to: localhost because the API call goes over the network from wherever it runs, and the controller is a sensible place to run it from. Delegating an API call to the load balancer host itself is pointless — you would be connecting to the balancer over SSH so that it can make an HTTP request to itself.

changed_when: true on both examples. Neither command nor uri knows whether the state actually changed, and a drain is a change worth reporting. The alternative — leaving uri to report ok — makes the run log claim nothing happened.

The load balancer belongs in inventory

delegate_to will happily connect to a hostname that is not in inventory, and upstream is explicit that doing so does not add it to inventory.

What you lose is everything inventory provides: no group_vars, no host_vars, no facts, and connection variables that fall back to defaults rather than to anything you configured. A play that works from your workstation, where the SSH config supplies the right user and key, fails in CI where it does not — and nothing in the playbook explains why.

Put the balancer in inventory, in its own group, with its connection variables and credentials where every other host has them. Then delegate_to: '{{ load_balancer_host }}' resolves through the same precedence rules as everything else, and the failure modes are the familiar ones.

Why “sleep 10” is not a drain

The pattern that appears in a great many playbooks:

Service impact possiblethe anti-pattern— Disables the backend then guesses at how long requests take. This is a deployment that works until traffic patterns change.

    - name: Remove from the pool
    ansible.builtin.command: /usr/local/bin/lb-ctl disable {{ inventory_hostname }}
    delegate_to: '{{ load_balancer_host }}'
    changed_when: true

  # Do not do this.
  - name: Wait for connections to finish
    ansible.builtin.pause:
      seconds: 10

It is wrong in both directions at once.

Too short, and you restart mid-request. Ten seconds is fine until a report endpoint takes forty, or a client uploads a large file, or the database is slow that night. The failure is intermittent, correlates with load, and looks like a network problem.

Too long, and every deployment costs the padding. Ten seconds per host across 200 hosts in batches of five is a little over six minutes of pure waiting. Somebody will eventually shorten it, under time pressure, without knowing why it was ten.

Above all: it asserts nothing. After the pause you know exactly as much as you did before it. A verification that cannot fail is not a verification.

Verifying the drain

wait_for with state: drained checks for active TCP connections on a port and returns when there are none.

Read-only / Safewaiting for the connections to actually go away— Read-only: wait_for inspects connection state and changes nothing. It fails if the host has not drained within the timeout, which converts an assumption into an assertion.

    - name: Wait for in-flight requests to complete
    ansible.builtin.wait_for:
      host: '{{ ansible_host | default(inventory_hostname) }}'
      port: 8080
      state: drained
      timeout: 90
      sleep: 2
      exclude_hosts: '{{ drain_exclude_hosts }}'

The difference from the pause version is the whole point: this task can fail. A host that still has connections after 90 seconds fails the task, which under max_fail_percentage: 0 stops the rollout — and that is correct, because a host that will not drain is a host you should not be restarting.

It also finishes early. A host that drains in 300 milliseconds costs 300 milliseconds, not ten seconds.

`exclude_hosts` is not optional

state: drained counts every active connection to the port. In production that includes things that will never go away:

the monitoring system polling /healthz every ten seconds
the load balancer own health checks, which usually continue against a draining backend
a metrics scraper
another application server holding a keep-alive connection

Any one of these makes the drain never complete, and the task fails at the timeout on a host that drained perfectly well.

Read-only / Safeexcluding the pollers— Read-only. Define the exclusion list once in group_vars rather than per task, so it stays correct when the monitoring infrastructure changes.

# group_vars/appservers.yml
drain_exclude_hosts:
- 192.0.2.10        # load balancer health checks
- 192.0.2.11        # load balancer, second node
- 198.51.100.25     # monitoring poller

Getting this list wrong is the most common reason a correct drain implementation fails on its first production run. Work it out before the change window by looking at what is actually connected:

Read-only / Safefinding out who is connected before you write the exclusion list— Read-only: ss reports socket state, and changed_when: false keeps the probe out of the change signal.

    - name: List everything currently connected to the application port
    ansible.builtin.command: ss -Htn state established '( sport = :8080 )'
    changed_when: false
    register: connections

  - name: Show the connected peers
    ansible.builtin.debug:
      msg: '{{ connections.stdout_lines }}'

TIME_WAIT is counted, and it lingers

The active_connection_states option controls which TCP states count as active, and its default is:

ESTABLISHED, FIN_WAIT1, FIN_WAIT2, SYN_RECV, SYN_SENT, TIME_WAIT

TIME_WAIT is the one that surprises people. A socket enters TIME_WAIT after it has been closed cleanly, and it stays there for twice the maximum segment lifetime — commonly 60 seconds on Linux.

So on a busy host, connections that have already finished continue to count as active for up to a minute after the last real request. The drain has completed in every sense that matters, and wait_for is still waiting.

If your drains time out on healthy hosts and ss shows nothing but TIME_WAIT, narrow the state list rather than lengthening the timeout:

    - name: Wait for genuinely in-flight requests only
      ansible.builtin.wait_for:
        host: '{{ ansible_host | default(inventory_hostname) }}'
        port: 8080
        state: drained
        timeout: 60
        active_connection_states:
          - ESTABLISHED
          - FIN_WAIT1

This is a real trade-off rather than a free fix. TIME_WAIT is in the default list because a socket in that state can still receive a retransmitted segment, and excluding it means accepting a smaller window of risk in exchange for a drain that completes in seconds rather than minutes. For an HTTP application behind a load balancer, that trade is usually right; for a protocol where a late duplicate would be harmful, think about it first.

When the port is the wrong question

wait_for counts TCP connections, which is a proxy for “is this host serving traffic”. Sometimes it is the wrong proxy.

Persistent connections never drain. An application using long-lived keep-alive or WebSocket connections holds them open for hours. The connection count will not fall to zero because the clients have no reason to disconnect. Draining here means asking the load balancer how many requests it has routed recently, not counting sockets.

The load balancer knows better than the host. A balancer that tracks active sessions per backend can answer “is this backend still serving anyone” authoritatively, and its answer accounts for session affinity that the host cannot see.

Read-only / Safeasking the balancer instead of counting sockets— Read-only: a GET against the balancer API, retried until the backend reports no active sessions. The retry loop is what makes this a wait rather than a single check.

    - name: Wait until the balancer reports no active sessions for this backend
    ansible.builtin.uri:
      url: >-
        https://lb.example.com/api/v1/pools/{{ app_pool }}/members/{{ inventory_hostname }}
      method: GET
      headers:
        Authorization: 'Bearer {{ lb_api_token }}'
      return_content: true
    delegate_to: localhost
    register: member
    retries: 30
    delay: 3
    until: member.json.active_sessions | default(1) == 0
    changed_when: false

Note default(1) rather than default(0). If the field is missing — because the API changed, or the response was an error — the condition must not be satisfied. Defaulting an absent value to the one that means “safe to proceed” is how a broken check becomes an invisible check.

Under the hood

wait_for with state: drained runs on the managed node and reads the kernel connection table, filtering for sockets whose local port matches and whose state appears in active_connection_states. It polls every sleep seconds until the count reaches zero or timeout expires.

Two consequences follow from it being a local view.

It cannot see connections the load balancer intends to send but has not yet opened, so there is a small race between the balancer accepting your disable call and stopping new connections. Most balancers close this by completing the disable synchronously; the ones that do not need a short settle before the drain check, which is the one legitimate use of a pause in this pattern.

It also cannot distinguish a client that is waiting on a slow response from one that is idle inside a keep-alive window. Both are ESTABLISHED. This is why the socket count is a proxy rather than an answer, and why an application with persistent connections needs the balancer view instead.

The check_mode attribute of wait_for reports support: none, which means the module is skipped entirely under --check rather than simulated. A --check run of a rolling play therefore proves nothing about the drain step: it does not run, cannot fail, and reports as skipped. The health-check lesson returns to this, because the same is true of uri, and between them they are most of the verification in this pattern.

What happens if the drain succeeds but the deploy fails afterwards?

The host stays drained, and nothing will put it back unless you wrote something that will.

This is the capacity-loss failure in its most common form. The drain worked, the balancer correctly stopped sending traffic, and then the deployment failed for an unrelated reason — a package repository timeout, a full disk, a failed health check. The failure policy stops the play. The return-to-service task is further down the task list and is never reached.

The host is now healthy, running the old version, and receiving no traffic. Every check that runs against the host reports it as fine, because it is fine. Only the load balancer knows, and nobody looks at the load balancer after a failed deploy — they look at the Ansible log, which ends with a failure about a package.

The structural answer is the block/always from the previous lesson, and it needs to wrap the drain as well as the deploy: the moment the drain succeeds, you have incurred an obligation to return the host, and that obligation has to be discharged on the failure path too.

Knowledge check

Why this matters

The drain is the step where a rolling deployment takes on its debt.

Everything before it is reversible and cheap. The moment the load balancer stops sending traffic to a host, that host is out of service and the fleet is running with reduced capacity — and it stays that way until something explicitly puts it back. Ansible will not do it for you, no timeout will do it, and no other part of your infrastructure knows it is supposed to happen.

That is why the verification matters twice over. Verifying the drain protects the users whose requests were in flight. But it also forces you to notice that draining is an action with an obligation attached, and the obligation has to be discharged on every path out of the play, including the ones where everything went wrong.

A rolling deployment that drains without verifying is restarting services under live traffic. One that drains without guaranteeing the return is losing capacity every time it fails. Most implementations that get the first right still get the second wrong, and the second is the one nobody notices until peak.