The shape of a rolling deployment

A rolling deployment is not a feature. There is no rolling: true keyword, no module that does it, nothing in Ansible that knows the pattern by name.

It is a shape you assemble, from serial, delegate_to, handlers, wait_for, uri and a block structure — and it exists because of one observation: a change that succeeds is not the same as a change that worked.

template returned changed. systemd_service returned changed. Both tasks are green. The service is broken, has been since the restart, and the play is about to do the same thing to the next fifty hosts.

Preventing exactly that is what this part is for.

The loop

flowchart LR
  subgraph BATCH["for each serial batch"]
    direction TB
    A["1. Drain<br/>remove from load balancer"]
    B["2. Verify drained<br/>connections actually gone"]
    C["3. Deploy<br/>write the new version"]
    D["4. Restart or reload<br/>make it live"]
    E["5. Health-check<br/>assert it works"]
    F["6. Return to service<br/>re-enable in load balancer"]
    G["7. Verify traffic<br/>and soak"]
    A --> B --> C --> D --> E --> F --> G
  end
  START(["batch N"]) --> A
  G --> NEXT(["batch N+1"])
  E -.->|"health check fails"| STOP["STOP the rollout.<br/>Return this batch to service<br/>or record it as drained."]

Six steps in the curriculum’s naming, seven boxes here because “verify drained” and “verify traffic” are the two that implementations most often skip and both deserve to be visible.

The dashed line is the point of the whole diagram. A health check that fails must stop the rollout and deal with the batch it is holding — because those hosts are drained, and nothing else is going to put them back.

Every step owes a verification

This is the rule that separates a rolling deployment from a staged outage, and it applies to every step without exception.

Read the right-hand column as the real work. The left-hand column is what everybody writes.

Ansible reports on the action, never on the outcome. changed: true from a template task means the file on disk now differs from what was there before. It does not mean the file is valid, that the service can parse it, that the service reloaded it, or that anything is working. The module has no opinion about any of that, and it should not — its contract is about the file.

The gap between “the task succeeded” and “the thing works” is where production incidents live, and on a fleet it is multiplied by the number of hosts you proceed to.

Which construct implements which step

The pattern is assembled from parts you already have.

Three of those are worth flagging now because they are the ones that go wrong.

Handlers flush per batch, which is what makes the restart land inside the loop rather than at the end of the play. Verified in Part XXXI: serial: 2 over six hosts produced three RUNNING HANDLER banners. Without that property the pattern would not work at all.

delegate_to runs the task against a different host — the load balancer — while the play is iterating over application servers. The drain and return-to-service steps are the only two in the loop that do not touch the host being deployed.

always is what stops a failure from stranding a drained host. If the health check fails, the play must still decide what happens to the hosts it has drained. Part XXXI showed that both failure keywords stop the play without running the remaining tasks — so the return-to-service step cannot be an ordinary task at the end of the list.

The skeleton

Not the finished playbook — that is the last lesson of this part — but the shape, so the rest of the lessons have somewhere to attach.

- name: Roll the application release
hosts: appservers
become: true
serial:
  - 1
  - 5
  - 25%
max_fail_percentage: 0

tasks:
  - name: Deploy this batch, returning it to service whatever happens
    block:
      # 1 + 2. Drain, and confirm the drain completed
      - name: Remove this host from the load balancer
        ansible.builtin.command: >-
          /usr/local/bin/lb-ctl disable {{ inventory_hostname }}
        delegate_to: '{{ load_balancer_host }}'
        changed_when: true

      - name: Wait for in-flight connections to finish
        ansible.builtin.wait_for:
          host: '{{ ansible_host | default(inventory_hostname) }}'
          port: 8080
          state: drained
          timeout: 90

      # 3. Deploy
      - name: Install the release
        ansible.builtin.unarchive:
          src: 'app-{{ release_version }}.tar.gz'
          dest: /opt/app
          owner: app
          group: app
        notify: Restart app

      # 4. Restart now, not at the end of the batch
      - name: Apply the restart before checking health
        ansible.builtin.meta: flush_handlers

      # 5. Health-check: the step that makes the rest worth doing
      - name: Assert the service is serving the new version correctly
        ansible.builtin.uri:
          url: 'http://{{ inventory_hostname }}:8080/healthz'
          return_content: true
          status_code: 200
        register: health
        retries: 12
        delay: 5
        until: health.status == 200 and release_version in health.content

    always:
      # 6. Return to service - runs whether the block succeeded or failed
      - name: Put this host back in the load balancer
        ansible.builtin.command: >-
          /usr/local/bin/lb-ctl enable {{ inventory_hostname }}
        delegate_to: '{{ load_balancer_host }}'
        changed_when: true
        when: health is defined and health.status | default(0) == 200

      - name: Record a host that was drained and not returned
        ansible.builtin.debug:
          msg: >-
            {{ inventory_hostname }} FAILED its health check and remains
            OUT OF SERVICE. Capacity is reduced. Investigate before the
            next peak.
        when: health is not defined or health.status | default(0) != 200

handlers:
  - name: Restart app
    ansible.builtin.systemd_service:
      name: app
      state: restarted

When this is the wrong pattern

Worth saying at the start of the part rather than the end.

When the versions cannot coexist. A rolling deployment guarantees that old and new run simultaneously. If a schema change, a message format or an API contract makes that impossible, rolling is not a gentler option — it is a guaranteed outage in slow motion. That is the subject of the version-skew lesson, and it is a precondition to check before designing any of this.

When there is no capacity headroom. If the service needs every instance to carry peak traffic, draining even one host causes a degradation. Rolling deployments are paid for in spare capacity, and a fleet running at 100% cannot afford one.

When the change is not per-host. A database migration, a DNS change, a certificate on a shared load balancer — these happen once, not once per host. Wrapping them in a rolling play makes them run once per batch, which is the run_once trap in a different costume.

When the platform already does it. A container orchestrator with a rolling update strategy and readiness probes implements this pattern natively and better, because it owns the load balancer. Reimplementing it in Ansible on top of that is duplicated logic that will disagree with the platform at the worst moment. The Docker course covers the orchestrator-native equivalents.

Knowledge check