The complete rolling playbook · Ansible

Everything in the last two parts, in one file.

The comments are the lesson. Each one names the specific failure that block prevents, because a rolling playbook without that context looks like ceremony — and the first person to find it slow will delete the parts whose purpose is not written down.

Play one: preconditions, unbatched

Read-only / Safeplay 1 of 3 — refuse to start if the fleet is not ready— Read-only throughout: ping connects and returns, assert evaluates. Unbatched so it sees the whole fleet, which a batched play cannot.

---
# =============================================================
# Play 1 - PRECONDITIONS
# Unbatched on purpose: every check here is fleet-wide, and
# inside the rolling play it would only ever see one batch.
# =============================================================
- name: Pre-flight checks before rolling anything
hosts: appservers
gather_facts: false
any_errors_fatal: true

tasks:
  # PREVENTS: rolling an incompatible release.
  # During the rollout the previous version runs alongside this
  # one. Nothing here can verify that is safe - this records
  # that a human decided it, and puts the decision in Git.
  - name: Refuse a release that has not been assessed for version skew
    ansible.builtin.assert:
      that:
        - skew_assessed | default(false) | bool
        - skew_compatible | default(false) | bool
      fail_msg: >-
        Release {{ release_version }} is not declared skew-compatible.
        Use a maintenance window, or restructure with expand-and-contract.
    run_once: true
    delegate_to: localhost

  # PREVENTS: a rollout that proceeds through a network partition.
  # Verified on 2.21.3: neither max_fail_percentage nor
  # any_errors_fatal reacts to an unreachable host, so this is the
  # only place connectivity gets gated.
  - name: Contact every host
    ansible.builtin.ping:

  - name: Refuse to roll if any host did not answer
    ansible.builtin.assert:
      that: missing | length == 0
      fail_msg: >-
        Did not answer: {{ missing | join(", ") }}.
        A rollout will proceed past unreachable hosts and leave them
        on the old version.
    vars:
      missing: '{{ ansible_play_hosts_all | difference(ansible_play_hosts) }}'
    run_once: true

  # PREVENTS: a run that touches far more than the change ticket said.
  # An unexpected host count means the inventory changed, not the play.
  - name: Refuse a target set larger than the change was reviewed for
    ansible.builtin.assert:
      that:
        - ansible_play_hosts_all | length > 0
        - ansible_play_hosts_all | length <= max_hosts | int
      fail_msg: >-
        Targeting {{ ansible_play_hosts_all | length }} hosts;
        this change was reviewed for at most {{ max_hosts }}.
    run_once: true

  # PREVENTS: draining a batch from a fleet with no headroom.
  - name: Refuse to roll if losing a batch would drop below minimum capacity
    ansible.builtin.assert:
      that: >-
        (ansible_play_hosts_all | length) - (max_batch_size | int)
        >= (min_healthy_hosts | int)
      fail_msg: >-
        Fleet of {{ ansible_play_hosts_all | length }} cannot lose
        {{ max_batch_size }} hosts and stay above {{ min_healthy_hosts }}.
    run_once: true

Play two: the rollout

Service impact possibleplay 2 of 3 — the rolling loop— This drains production hosts from the load balancer and deploys new code. Every block carries the failure it prevents. Run it with --limit and read the effective host list first.

# =============================================================
# Play 2 - THE ROLLOUT
# =============================================================
- name: Roll {{ release_version }} across the fleet
hosts: appservers
become: true

# PREVENTS: a fleet-wide simultaneous change.
# One host proves the release works at all; five prove it works on
# more than a lucky host; 25% at a time exposes what only appears
# with volume. The absence of this line is the default, and the
# default is "every targeted host at once".
serial:
  - 1
  - 5
  - 25%

# PREVENTS: a broken release reaching the whole fleet.
# 0 aborts on the first failure in any batch, because any non-zero
# failure rate exceeds zero. Evaluated per batch - verified on
# 2.21.3 - and the threshold must be EXCEEDED, not equalled, which
# is why 0 rather than a small number.
max_fail_percentage: 0

tasks:
  # PREVENTS: a rollout that cannot be stopped without Ctrl-C.
  # A batched play re-runs its task list per batch, so a check
  # placed first runs before every batch. Stopping here is clean:
  # nothing is drained and no host is halfway through anything.
  - name: Look for an operator stop request
    ansible.builtin.stat:
      path: /var/run/ansible/stop-rollout
    register: stop_request
    delegate_to: localhost
    run_once: true

  - name: Halt at this batch boundary if asked to
    ansible.builtin.debug:
      msg: >-
        STOP REQUESTED before batch [{{ ansible_play_batch | join(', ') }}].
        Hosts not yet deployed remain on {{ previous_version | default('the previous release') }}.
    run_once: true
    when: stop_request.stat.exists

  - name: End the play on request
    ansible.builtin.meta: end_play
    when: stop_request.stat.exists

  # PREVENTS: a drained host that is never returned to service.
  # Verified in Part XXXI: when a failure policy triggers, surviving
  # hosts in the batch run NO further tasks. A return-to-service
  # step written as an ordinary final task is exactly such an
  # unreached task. always is dispatched however the block ended.
  - name: Deploy this host, returning it to service whatever happens
    block:

      # --- 1. Drain -------------------------------------------
      - name: Remove this host from the load balancer pool
        ansible.builtin.uri:
          url: '{{ lb_api }}/pools/{{ app_pool }}/members/{{ inventory_hostname }}'
          method: PATCH
          body_format: json
          body:
            state: draining
          headers:
            Authorization: 'Bearer {{ lb_api_token }}'
          status_code: [200, 204]
        delegate_to: localhost
        changed_when: true

      # PREVENTS: restarting a service mid-request.
      # "sleep 10" asserts nothing and cannot fail. This can, and a
      # host that will not drain is a host you should not restart.
      # exclude_hosts keeps monitoring pollers from making the
      # drain never complete.
      - name: Wait for in-flight requests to finish
        ansible.builtin.wait_for:
          host: '{{ ansible_host | default(inventory_hostname) }}'
          port: '{{ app_port }}'
          state: drained
          timeout: 90
          sleep: 2
          exclude_hosts: '{{ drain_exclude_hosts }}'

      # --- 2. Deploy ------------------------------------------
      - name: Install the release
        ansible.builtin.unarchive:
          src: 'app-{{ release_version }}.tar.gz'
          dest: /opt/app
          owner: app
          group: app
          mode: '0755'
        notify: Restart app

      - name: Record the deployed version for the census playbook
        ansible.builtin.copy:
          content: '{{ release_version }}'
          dest: /etc/app/deployed-version
          owner: root
          group: root
          mode: '0644'

      # PREVENTS: health-checking the process that has not restarted.
      # The implicit end-of-batch flush protects the NEXT batch and
      # is too late for this one to verify itself.
      - name: Apply the restart now, not at the end of the batch
        ansible.builtin.meta: flush_handlers

      # --- 3. Verify ------------------------------------------
      # PREVENTS: a slow starter failing the correctness check for
      # the wrong reason. A failure here says "never started"; a
      # failure below says "started and is wrong".
      - name: Wait for the application to bind its port
        ansible.builtin.wait_for:
          host: '{{ ansible_host | default(inventory_hostname) }}'
          port: '{{ app_port }}'
          state: started
          timeout: 60

      # PREVENTS: the whole reason this part exists - template
      # succeeded, restart succeeded, service is broken. Asserts the
      # VERSION, not just a 200, so a cached response or a service
      # still running the old code cannot satisfy it. Every default()
      # is chosen to FAIL the condition when the field is absent.
      - name: Assert the service is serving the release we just deployed
        ansible.builtin.uri:
          url: 'http://{{ ansible_host | default(inventory_hostname) }}:{{ app_port }}/healthz'
          return_content: true
          status_code: 200
          timeout: 10
        register: health
        retries: 12
        delay: 5
        until:
          - health.status | default(0) == 200
          - health.json.version | default('') == release_version
          - health.json.database | default('') == 'ok'
        changed_when: false

    always:
      # PREVENTS: the silent capacity loss. Healthy hosts go back;
      # unhealthy ones are announced, because the always block is
      # the last point in the run where anything knows this host
      # was drained.
      - name: Return a verified host to the load balancer
        ansible.builtin.uri:
          url: '{{ lb_api }}/pools/{{ app_pool }}/members/{{ inventory_hostname }}'
          method: PATCH
          body_format: json
          body:
            state: active
          headers:
            Authorization: 'Bearer {{ lb_api_token }}'
          status_code: [200, 204]
        delegate_to: localhost
        changed_when: true
        when: health.json.version | default('') == release_version

      # PREVENTS: sending traffic to a host the balancer has not yet
      # health-checked, while the next batch is already being drained.
      - name: Wait for the balancer to mark this backend active and healthy
        ansible.builtin.uri:
          url: '{{ lb_api }}/pools/{{ app_pool }}/members/{{ inventory_hostname }}'
          method: GET
          headers:
            Authorization: 'Bearer {{ lb_api_token }}'
          return_content: true
        delegate_to: localhost
        register: member
        retries: 20
        delay: 3
        until:
          - member.json.state | default('') == 'active'
          - member.json.health | default('') == 'healthy'
        changed_when: false
        when: health.json.version | default('') == release_version

      - name: Announce a host left out of service
        ansible.builtin.debug:
          msg: >-
            CAPACITY WARNING: {{ inventory_hostname }} failed its health
            check and remains OUT OF SERVICE in pool {{ app_pool }}.
            Investigate before the next traffic peak.
        when: health.json.version | default('') != release_version

  # PREVENTS: a slow-burning failure reaching the whole fleet.
  # The health check answers "is it working now". A memory leak, a
  # filling connection pool or a timer-driven job that fails on its
  # first fire all need elapsed time. Longest after the canary.
  - name: Soak before the next batch
    ansible.builtin.pause:
      seconds: >-
        {{ canary_soak | default(300)
           if ansible_play_hosts_all[0] in ansible_play_batch
           else batch_soak | default(60) }}
    run_once: true
    when: not (skip_soak | default(false) | bool)

handlers:
  # Change-gated: fires only if the unarchive actually changed
  # something, so a re-run against an already-current fleet does
  # not restart every service for nothing. state: restarted, not
  # reloaded - a reload does not replace a running binary.
  - name: Restart app
    ansible.builtin.systemd_service:
      name: app
      state: restarted
      daemon_reload: true

Play three: what must happen either way

Configuration changeplay 3 of 3 — the report— Read-mostly. Runs after the rollout to state what the fleet actually looks like. Deliberately NOT the place for anything that must run regardless of outcome.

# =============================================================
# Play 3 - REPORT
# NOTE: any_errors_fatal in play 2 would prevent this play from
# running at all - verified on 2.21.3, a subsequent play does not
# run. Anything that MUST happen regardless of outcome belongs in
# the always block of play 2, not here.
# =============================================================
- name: Report the resulting fleet state
hosts: appservers
gather_facts: false

tasks:
  - name: Read the deployed-version marker
    ansible.builtin.slurp:
      src: /etc/app/deployed-version
    register: marker
    failed_when: false

  - name: Summarise the fleet
    ansible.builtin.debug:
      msg: >-
        {{ inventory_hostname }}:
        {{ (marker.content | default('') | b64decode | trim) or 'ABSENT' }}

Running it

Read-only / Safethe pre-execution sequence— Read-only. Establish the effective host list and the task list before anything runs. --check is genuinely limited here and the next callout says why.

ansible-playbook deploy.yml --limit appservers --list-hosts

ansible-playbook deploy.yml --limit appservers --list-tasks

ansible-playbook deploy.yml --limit appservers --check --diff

Service impact possiblethe real run— Deploys to production. Run inside tmux on the controller with log_path configured, so an interrupted run leaves a record and a dropped session does not stop it.

tmux new -s deploy-{{ release_version }}

ANSIBLE_LOG_PATH=/var/log/ansible/deploy-$(date +%F-%H%M).log \
ansible-playbook deploy.yml --limit appservers

Reviewing a rolling playbook

The checklist form, for reading somebody else’s:

Question	If the answer is no
Is there a `serial`?	fleet-wide simultaneous change
Is there a failure policy beside it?	staging without stopping — the outage arrives one tidy batch at a time
Is the health check asserting the version, or just a `200`?	a cached page or an unrestarted service passes
Is there a `flush_handlers` before the health check?	the check tests the previous release
Is return-to-service in an `always`?	a failed batch is left drained
Does `always` branch on health?	broken hosts get real traffic
Are fleet-wide preconditions in their own unbatched play?	they check one batch against itself
Is anything that must always run placed in a later play?	`any_errors_fatal` will skip it
Is reachability gated before the rollout?	the run proceeds through a partition
Has skew been assessed?	the rollout may be staging an outage

Under the hood

Read the file as three layers and the design becomes legible.

Layer one is the loop: drain, deploy, restart, verify, return. This is the part that looks like a deployment, and on a good night it is the only part that executes anything interesting.

Layer two is the failure containment: serial bounds how much can break at once, max_fail_percentage: 0 stops the loop, and block/always guarantees the obligation created by the drain is discharged on the failure path. None of this runs on a good night.

Layer three is the preconditions: the skew assertion, the reachability gate, the host-count guard, the capacity check. These run first and their entire purpose is to refuse.

The layers are ordered by how early they can fail, and that ordering is the design. A precondition failure costs nothing — nothing has been touched. A loop failure costs one batch. There is no layer that costs the fleet, and constructing that property is what the two parts have been for.

The one thing no layer covers is worth naming again: a skew failure produces no failing task on any host in the current batch, so layers one and two are structurally blind to it and layer three can only record that somebody was asked. Detection there comes from outside Ansible entirely — which is why the soak exists, and why the soak is most valuable with somebody watching an error-rate dashboard during it.

What happens if somebody removes the serial keyword?

Every safety property in the file collapses at once, and nothing else changes visibly.

max_fail_percentage: 0 still applies — but to a single batch containing the entire fleet, and it is evaluated after the failing task has run everywhere. So the health check fails on all 200 hosts, having already drained and restarted all 200.

The always block still runs, so hosts are returned to service. The meta: end_play stop file still works, checked once. The soak runs once, at the end, protecting nothing.

And meta: end_batch, if the play used it, would silently change meaning — the module documentation states that with serial set to 0 or undefined it “behaves the same as end_play”.

The playbook would still look careful in review. It has health checks, blocks, guards and a soak. It would take the entire fleet down in one pass.

This is the argument for the review checklist above rather than for trusting that a file with a lot of safety machinery in it is safe. The keywords interact, and removing one line changes what the others mean.

Knowledge check

Why this matters

This file is long, and every line of it exists because of a specific failure that has happened to somebody.

That is worth stating plainly, because the pressure on a playbook like this is always toward deletion. It is slow — the soak alone can add twenty minutes. It is verbose. Most of it never executes, because most deployments succeed. Somebody under time pressure will look at the wait_for on the drain and see ninety seconds of waiting per batch that has never once caught anything.

They will be right about the observation and wrong about the conclusion. Safety machinery that has never fired looks like waste and is indistinguishable from waste until the night it fires.

Which is why the comments matter more than the tasks. A block labelled PREVENTS: a drained host that is never returned to service survives review. An unlabelled always block gets refactored away by somebody who cannot see what it is for, and the capacity loss it was preventing arrives six weeks later looking like an unrelated incident.

Write down what each piece prevents. It is the only defence the pattern has against its own success.