Fleet patch strategy - waves, canaries, and rollback · Linux

Patching a fleet is not “apt upgrade on every host”. It is a disciplined wave-based rollout with health validation and rollback capability.

The wave strategy

Wave 1: dev / staging (immediate)
Wave 2: canary (5-10% of production)
Wave 3: production (50%)
Wave 4: production (remaining)

Each wave has a hold period (15-60 minutes) to validate health before proceeding to the next wave. That hold has to be implemented by something - a pause step, a scheduler, a human gate. No tool gives it to you for free.

Apply via configuration management

Use Ansible, Puppet, or similar to apply patches:

# Ansible playbook
- name: Apply security patches
  hosts: all
  serial: "5%"
  max_fail_percentage: 0        # any host failure aborts the whole rollout
  tasks:
    - name: Security-only upgrade
      ansible.builtin.apt:
        upgrade: safe           # never removes packages; dist can
        update_cache: yes
        cache_valid_time: 3600
      environment:
        DEBIAN_FRONTEND: noninteractive

    - name: Detect pending reboot (Debian/Ubuntu)
      ansible.builtin.stat:
        path: /var/run/reboot-required
      register: reboot_required

    - name: Reboot if the kernel or libc changed
      ansible.builtin.reboot:
        reboot_timeout: 600
      when: reboot_required.stat.exists

    - name: Verify service health from outside the host
      ansible.builtin.uri:
        url: "https://{{ inventory_hostname }}/health"
        status_code: 200
      register: health
      until: health.status == 200
      retries: 10
      delay: 15
      delegate_to: localhost

    - name: Soak before the next batch
      ansible.builtin.pause:
        minutes: 15

On RHEL, replace the stat check with a needs-restarting -r command task: exit code 1 means a reboot is required.

Four ways the obvious playbook is wrong

The naive version of this playbook is the one most fleets actually run. Each defect is quiet, and together they produce a fleet that reports compliant while still running the vulnerable code.

when: ansible_reboot_required | default(false). There is no such fact. setup does not gather it, so the filter falls back to false on every host, forever, and the reboot task never runs. New kernels are installed and never booted; the old libc and libssl stay mapped in every long-lived process. Check a real file with stat, or run needs-restarting -r, and act on that.

upgrade: dist under the name “Apply security patches”. dist is apt-get dist-upgrade: every package, and it is permitted to remove installed packages to resolve the upgrade. A task labelled security-only that may uninstall your application’s dependency is a change nobody reviewed. Use upgrade: safe.

Assuming serial provides a hold period. It does not. serial sets the batch size; batches run back to back with no delay at all. If the design calls for a 15-minute soak, the playbook must contain an explicit pause (or an external orchestrator must drive the waves). Prose promising a hold period does not create one.

No max_fail_percentage. By default a host that fails a task is simply dropped from the play and the rollout continues. A health-check failure on the canary batch removes the canaries and proceeds to the rest of the fleet - the precise outcome the wave strategy exists to prevent. max_fail_percentage: 0 makes any failure abort the play.

serial: "5%" rolls out to 5% of hosts at a time. It provides batching only - the delay between batches is the explicit pause task, and the abort gate is max_fail_percentage.

Health validation

After each wave, check:

Service is responding (HTTP health endpoint), probed from outside the host.
The running kernel matches the newest installed kernel: uname -r against /boot/vmlinuz-*.
No pending reboot: /var/run/reboot-required absent, or needs-restarting -r exits 0.
No process still mapping a replaced library: needrestart -b (Debian/Ubuntu) or needs-restarting -s (RHEL) returns an empty service list.
No failed units: systemctl list-units --state=failed.
No new errors in journald / auditd.
CPU, memory, network within normal range.
Synthetic monitoring passes.

The first four are what separate installed from remediated. A vulnerability scanner reads package versions and will call a patched-but-not-rebooted host compliant. It is not. linux-checklist-post-patching-validation carries the full list with commands.

Automate the validation:

# After a patch wave, check the error rate.
# --data-urlencode does the escaping: a bare
# ...?query=rate(errors[5m]) is a shell syntax error, because ( ) and [ ]
# are shell metacharacters, and the URL would be malformed even if it ran.
error_rate=$(curl -sG http://prometheus:9090/api/v1/query \
    --data-urlencode 'query=sum(rate(http_requests_total{code=~"5.."}[5m]))' \
    | jq -r '.data.result[0].value[1] // "0"')

# bc, not [[ ]] - bash cannot compare floating-point numbers
if (( $(echo "$error_rate > 0.01" | bc -l) )); then
    echo "Error rate too high: $error_rate - halting the rollout" >&2
    exit 1
fi

Rollback

Have a rollback plan for every patch:

Packages: sudo dnf history undo <id> or sudo apt-get install <package>=<old-version>. Use undo, not rollback: dnf history undo <id> reverses that one transaction, while dnf history rollback <id> reverses every transaction performed after <id>, sweeping up unrelated security updates applied since. linux-rollback-strategies covers the distinction.
Containers: redeploy the previous image.
Configuration: revert the configuration management state.

Test rollback in staging: deploy the patch, deploy the rollback, verify both work.

Maintenance windows

For non-emergency patches, schedule maintenance windows:

Off-peak hours (typically nights or weekends).
Pre-announced to stakeholders.
Limited duration (2-4 hours).
Rollback ready.

The window is a contract: the change starts at T+0, rolls forward at T+30 minutes, completes by T+window-end. If something goes wrong, rollback by T+window-end.