Runbook: Perform a rolling reboot

A rolling reboot is the change with no rollback. Once a host is going down it is going down, and the only available control is deciding whether the next one does.

That reframes the whole procedure. Every gate here exists to protect the hosts that have not been rebooted yet, and the single most important rule is the one that sounds like bookkeeping: do not drain the next batch until the last batch is back in rotation and carrying traffic.

When to use this runbook

• A kernel, glibc or systemd update requires a reboot (see also the rolling kernel update runbook, which adds the fallback-kernel steps).
• Clearing a condition that only a reboot resolves.
• Proving that hosts still boot - a reboot drill.
• Applying a hypervisor or firmware change that requires a guest restart.

Blast radius

One batch at a time, and the batch size is a capacity decision. If the service needs N hosts and you have N+2, the batch size is 2 and no argument about round numbers changes that.

State it explicitly before starting:

group          web
total hosts    40
required       32
batch size     4      (leaves 36 in service)
batches        10

Step 1: Will it boot into the state it is running?

This is the pre-check that catches the expensive failure, and it is the one most often skipped because it feels unrelated to rebooting.

# Every fstab entry mounts - if this fails now it will fail at boot
ansible web -b -m command -a 'findmnt --verify --verbose' -o

# Services that are running but not enabled will not come back.
# systemctl compares the two lists for you:
ansible web -b -m command \
-a 'systemctl list-unit-files --type=service --state=disabled --no-pager --plain' -o
ansible web -b -m command \
-a 'systemctl list-units --type=service --state=running --no-pager --plain' -o

# Anything already broken
ansible web -b -m command -a 'systemctl --failed --no-pager --plain' -o

Step 2: Capture the baseline

ansible web -b -m command -a 'uptime -s' -o | tee baseline-boottime.txt
ansible web -b -m command -a 'uname -r' -o | tee baseline-kernel.txt
ansible web -b -m command \
-a 'systemctl list-units --type=service --state=running --no-pager --plain' \
> baseline-services.txt
ansible web -b -m command -a 'findmnt -rn -o TARGET' > baseline-mounts.txt

uptime -s prints the boot timestamp. That is the value Step 6 compares against to prove the host actually rebooted, and it is far more reliable than “it responded to ping again”.

Step 3: The play

- name: Rolling reboot
hosts: web
become: true
serial: 4
max_fail_percentage: 0
tasks:
  - name: Record the boot time before we touch anything
    ansible.builtin.command: uptime -s
    register: boot_before
    changed_when: false

  - name: Drain from the load balancer
    ansible.builtin.uri:
      url: "https://lb.example.com/api/pool/web/{{ inventory_hostname }}/drain"
      method: POST
      status_code: [200, 204]
    delegate_to: localhost

  - name: Allow in-flight connections to finish
    ansible.builtin.wait_for:
      timeout: 30
    delegate_to: localhost

  - name: Reboot and wait for the host to come back
    ansible.builtin.reboot:
      msg: "Rolling reboot - scheduled maintenance"
      pre_reboot_delay: 5
      post_reboot_delay: 30
      reboot_timeout: 900
      test_command: systemctl is-system-running --wait

  - name: Confirm the host actually rebooted
    ansible.builtin.command: uptime -s
    register: boot_after
    changed_when: false
    failed_when: boot_after.stdout == boot_before.stdout

  - name: Every fstab entry is mounted
    ansible.builtin.command: findmnt --verify
    changed_when: false

  - name: Service answers a real request
    ansible.builtin.uri:
      url: "http://{{ ansible_host }}:8080/healthz"
      status_code: 200
      return_content: true
    register: health
    retries: 12
    delay: 10
    until: health.status == 200 and 'ok' in health.content

  - name: Return to the load balancer pool
    ansible.builtin.uri:
      url: "https://lb.example.com/api/pool/web/{{ inventory_hostname }}/enable"
      method: POST
      status_code: [200, 204]
    delegate_to: localhost

  - name: Confirm it is receiving traffic before the next batch drains
    ansible.builtin.uri:
      url: "https://lb.example.com/api/pool/web/{{ inventory_hostname }}"
      return_content: true
    register: pool
    retries: 6
    delay: 10
    until: "'active' in pool.content"
    delegate_to: localhost

Read that task list as a contract. Every task between the reboot and the return to the pool is a gate the host must pass while it is out of rotation and harmless. max_fail_percentage: 0 means any host failing any of them ends the play, and the next batch never drains.

The last task is the one people leave out and the one that makes serial safe. Without it, serial proceeds as soon as the play tasks finish, which can be before the load balancer has actually started sending traffic to the returned hosts. Batch after batch, the pool shrinks slightly faster than it recovers, and at batch seven you are below capacity with no single failure to point at.

Step 4: The reboot module’s options, and what they actually do

ansible-doc ansible.builtin.reboot | head -60

Verified parameter names on 2.21.3: boot_time_command, connect_timeout, msg, post_reboot_delay, pre_reboot_delay, reboot_command, reboot_timeout, search_paths, test_command.

The two that matter most:

• reboot_timeout is how long the module waits for the host to come back before failing. The default is generous for a VM and short for a physical host with a long POST and a RAID controller that takes four minutes to initialise. Set it from the observed boot time of this hardware, not from the default.
• test_command is what the module runs to decide the host is usable. The default proves the connection works. systemctl is-system-running --wait waits for systemd to finish starting units, which is a meaningfully stronger claim - it is the difference between “sshd is up” and “the machine has finished booting”.

Step 5: Watch a batch by hand for the first one

ansible-playbook -i inventories/production rolling-reboot.yml \
--limit web --list-hosts

ansible-playbook -i inventories/production rolling-reboot.yml \
--limit 'web01.example.com:web02.example.com:web03.example.com:web04.example.com' \
--diff | tee "reboot-batch1.log"

Run the first batch as its own invocation. It costs one extra command and it means the first time you learn that reboot_timeout is too short for this hardware, it happens to four hosts with you watching rather than to forty unattended.

Step 6: Verify, per batch and at the end

ansible web -b -m command -a 'uptime -s' -o | tee after-boottime.txt
diff baseline-boottime.txt after-boottime.txt | head -50

Every host must show a different boot time. A host with an unchanged boot time did not reboot, whatever the run log said, and it is still running the old kernel or the old configuration while the change record says otherwise.

ansible web -b -m command \
-a 'systemctl list-units --type=service --state=running --no-pager --plain' \
> after-services.txt
diff baseline-services.txt after-services.txt

ansible web -b -m command -a 'findmnt -rn -o TARGET' > after-mounts.txt
diff baseline-mounts.txt after-mounts.txt

ansible web -b -m command -a 'systemctl --failed --no-pager --plain' -o

Missing services and missing mounts are the two findings that the health check will not catch, because a web tier can serve /healthz perfectly while the NFS share holding user uploads never came back.

Step 7: Close out

curl -sS https://lb.example.com/api/pool/web | python3 -m json.tool | grep -c '"active"'
ansible web --list-hosts | grep -c 'example.com'

Those counts must match. A host left drained is online, patched, idle and invisible - it reports healthy, it is in the inventory, and it is carrying nothing. Find it now; the alternative is discovering it during the next capacity incident.

Rollback

There is none. A reboot is not reversible, and this section is about what to do instead.

# Which hosts rebooted?
ansible web -b -m command -a 'uptime -s' -o | tee state-after-abort.txt

# Which are drained?
curl -sS https://lb.example.com/api/pool/web | python3 -m json.tool | grep -B2 drain

A stopped rolling reboot leaves the fleet in three states: rebooted and in service, rebooted and drained, not rebooted. Write that list down and close it out deliberately. It is the state that gets forgotten, and a host that stayed drained after an aborted window is capacity you are paying for and not receiving.

Common patterns

Escalation

Escalate when:

• A host does not return in the expected time. That needs console access, not a second reboot.
• Two hosts fail identically. The fleet shares the fault.
• A host in scope affects quorum.
• Capacity cannot be restored between batches.