Orchestrating the rolling loop - automation that stops on evidence

Doing rolling maintenance by hand works and does not scale. Two nodes is a pleasant afternoon; forty is a week of windows, and somewhere in that week somebody skips the observation period because the window is running late — which is the exact failure the workflow lesson warns about.

Automating it is the right answer and it is also how the failure gets faster. A script that patches forty nodes in sequence with no gate between them does not remove the risk of a bad change; it removes the human who would have noticed after the first one.

The whole design problem is therefore: what stops the loop?

The shape

Ansible expresses the loop directly. serial splits the play into batches and runs the entire play — including handlers — once per batch, which is what makes a per-batch gate possible.

# rolling-patch.yml
- name: Rolling patch of the web tier
  hosts: webservers
  # Canary first, then widen. Never start at the ceiling.
  serial:
    - 1
    - 2
    - "25%"
  # Any host failing in a batch aborts the play before the next one
  max_fail_percentage: 0
  order: sorted

  pre_tasks:
    - name: Record the kernel we are rolling back to
      ansible.builtin.command: uname -r
      changed_when: false
      register: pre_kernel

    - name: Signal the load balancer to drain this node
      ansible.builtin.file:
        path: /etc/myapp/draining
        state: touch
        mode: '0644'
      become: true

    - name: Wait out the load balancer detection window
      ansible.builtin.wait_for:
        timeout: 20

    - name: Confirm no established connections remain
      ansible.builtin.shell: >
        set -o pipefail;
        ss -Htn state established '( sport = :8080 )' | wc -l
      args:
        executable: /bin/bash
      register: conns
      changed_when: false
      retries: 12
      delay: 10
      until: conns.stdout | int == 0

  tasks:
    - name: Apply the change
      ansible.builtin.apt:
        upgrade: dist
        update_cache: true
      become: true

    - name: Does this host need a reboot?
      ansible.builtin.stat:
        path: /var/run/reboot-required
      register: reboot_flag

    - name: Reboot and wait for the host to answer again
      ansible.builtin.reboot:
        reboot_timeout: 900
        post_reboot_delay: 15
      become: true
      when: reboot_flag.stat.exists

  post_tasks:
    - name: No failed units
      ansible.builtin.command: systemctl is-system-running
      register: sysstate
      changed_when: false
      failed_when: sysstate.stdout not in ['running', 'starting']

    - name: The application answers on this node, not through the VIP
      ansible.builtin.uri:
        url: "http://{{ ansible_default_ipv4.address }}:8080/healthz"
        status_code: 200
      register: health
      retries: 30
      delay: 10
      until: health.status == 200

    - name: Return the node to the pool
      ansible.builtin.file:
        path: /etc/myapp/draining
        state: absent
      become: true

    - name: Observation period - let the node carry real traffic
      ansible.builtin.wait_for:
        timeout: 600

    - name: Assert the node logged no server errors while serving
      ansible.builtin.shell: >
        set -o pipefail;
        journalctl -u myapp --since '10 min ago' --no-pager
        | grep -c ' 5[0-9][0-9] ' || true
      args:
        executable: /bin/bash
      become: true
      register: errs
      changed_when: false
      failed_when: errs.stdout | int > 0

The last two tasks are the only place a fixed wait belongs. The observation period is genuinely a duration — the node has to serve real traffic for a while — but it is followed by an assertion, so the wait is setting up evidence rather than standing in for it.

Read that play as four gates, not as a list of tasks. The drain gate blocks until connections have actually fallen. The reboot gate blocks until the host answers again. The health gate blocks until the application responds on that host. The observation gate holds for a fixed period and then asserts that the node served that traffic without errors. Any one of them failing stops the batch, and max_fail_percentage: 0 stops the play.

until and retries are the whole mechanism

The difference between automation that gates on evidence and automation that gates on a timer is one keyword.

# A TIMER. This is not a gate. It waits and then proceeds
# regardless of what it finds.
- name: Wait for the service to settle
  ansible.builtin.pause:
    minutes: 5

# A GATE. It re-evaluates a condition and fails if the condition
# is never met within retries x delay.
- name: Wait for the service to be healthy
  ansible.builtin.uri:
    url: "http://{{ ansible_default_ipv4.address }}:8080/healthz"
    status_code: 200
  register: health
  retries: 30
  delay: 10
  until: health.status == 200

Both take five minutes on a healthy node. On an unhealthy node, the first one takes five minutes and then patches the next node anyway. The second one fails the host after five minutes and, with max_fail_percentage: 0, stops the play.

retries multiplied by delay is your patience budget, and it should be generous. A gate that gives up after 30 seconds converts every slow start into an aborted maintenance, and a team that has seen that twice starts running the playbook with the gate removed.

Choosing the abort condition

Ansible’s default is generous in a way that is wrong for maintenance. A host that fails a task is removed from the play, and the play carries on with the survivors. On a rolling patch that means node 3 breaks, gets quietly dropped, and nodes 4 through 40 are patched with the same broken change.

Two directives fix it, and they are not the same.

With serial: 1 the two are nearly equivalent, because a batch is one host. With a larger batch they differ: max_fail_percentage lets the rest of the batch finish and then stops, while any_errors_fatal halts the moment the first host fails.

Neither of them helps if a task cannot fail. ignore_errors: true and a command task without failed_when both produce gates that always pass, and both are usually added during a frustrating debugging session and never removed.

Rerunning safely

Rolling maintenance automation gets rerun constantly — after an abort, after a fix, after adding hosts. Three properties make that safe.

Idempotent tasks. Covered in the configuration-management part, and it is what lets you rerun the whole play against a fleet that is half patched. The already-patched hosts report ok and take seconds.

A drain step that tolerates an already-drained node. file: state=touch on a file that exists is fine; command: touch wrapped in a creates guard is fine; a task that fails when the flag is already there turns your recovery rerun into another abort.

--limit for surgical reruns. After an abort you usually want to work on one host, not resume the fleet:

# Just the node that failed
ansible-playbook -i /etc/ansible/prod rolling-patch.yml --limit web03

# Resume the campaign, excluding the one you are still debugging
ansible-playbook -i /etc/ansible/prod rolling-patch.yml --limit 'webservers:!web03'

Knowledge check