Capstone 4: Survive it

The last of the four capstone labs. It runs against the estate built and operated in capstone lab 1, lab 2 and lab 3.

Every previous lab changed the estate while it was working. This one takes things away — a configuration, a kernel, a password, and finally the controller itself — and asks whether you can put them back.

Objective

By the end you will have rolled back a configuration change from a written plan, patched and rebooted the whole fleet without an unscheduled outage, upgraded and rolled back a kernel, rotated a secret that two systems have to agree on, and rebuilt the controller from nothing but the repository and an escrowed password — then proved from a fresh controller that the estate is exactly as the repository says it should be.

Architecture

The same five production hosts, now examined for what they cannot survive.

   TIER          HOSTS   CAN IT BE ROLLED?
   ─────────────────────────────────────────────────────────────
   appservers      3     Yes. serial: 1, drain, health, return.
   loadbalancers   1     NO. One host is 100% of ingress. A reboot
                         here is an outage, and it needs a window.
   databases       1     NO. And worse: a reboot takes the whole
                         application down, because every app host
                         depends on it.

   controller      1     Not part of the estate. Rebuilt in Task 8
                         from git + escrowed vault passwords + a key.

Two of three tiers cannot be rolled. That is the most important sentence in this lab. A rolling reboot is a technique for tiers with redundancy, and applying it to a singleton produces a procedure that looks safe and is not.

Requirements

• Capstone labs 1–3 complete, the estate healthy, playbooks/health.yml passing.
• ansible-core 2.21.x, community.general, ansible.posix.
• Six VMs with systemd as PID 1, their own kernel and their own bootloader. B-nested only, and this lab is why the whole capstone declares it: Tasks 4 and 5 reboot machines and replace kernels. A container has no bootloader, shares the host kernel, and its boot_time_command never changes — every reboot assertion in this lab would pass without a reboot having happened, which is the worst possible outcome for a procedure whose purpose is to prove a host came back.
• Out-of-band access to every node: hypervisor console or serial. This is not optional. Task 5 is capable of producing a machine that does not boot, and no amount of Ansible reaches a GRUB prompt.
• At least 500 MB free in /boot on every node, or wherever your kernel images live. Task 5 asserts this and refuses hosts that fail.
• A VM snapshot of every node, taken now, before Task 1.
• The escrow envelope from lab 1: both vault passwords and the controller’s SSH private key, stored somewhere that is not the controller. Task 8 destroys the controller; if the only copy of those is on it, Task 8 ends the capstone permanently.
• Roughly 4.5 hours.

Scenario

The estate has been in production for a quarter. There is a kernel CVE with a fixed package available, the security team has asked for the database password to be rotated, and last week somebody changed a proxy timeout by hand on lb01 at two in the morning and nobody can remember what it was before.

And on Thursday the controller VM is going to be rebuilt by the virtualisation team, whether or not you are ready.

Tasks

Task 1: Capture everything this lab can destroy

cd "$HOME/estate"
mkdir -p reports/pre-survive

# playbooks/capture-boot.yml
- name: Record boot, kernel and package state before any of it changes
  hosts: estate
  become: true
  gather_facts: true

  tasks:
    - name: Read the installed kernel packages
      ansible.builtin.shell: |
        set -o pipefail
        dpkg-query -W -f='${Package}\n' 'linux-image-*' 2>/dev/null | sort
      args:
        executable: /bin/bash
      register: kernels
      changed_when: false
      failed_when: false

    - name: Read free space in /boot
      ansible.builtin.shell: |
        set -o pipefail
        df -Pk /boot | awk 'NR==2 {print $4}'
      args:
        executable: /bin/bash
      register: bootfree
      changed_when: false

    - name: Read the bootloader default
      ansible.builtin.shell: |
        set -o pipefail
        grep -E '^GRUB_DEFAULT=' /etc/default/grub || echo 'GRUB_DEFAULT=0'
      args:
        executable: /bin/bash
      register: grubdefault
      changed_when: false

    - name: Is a reboot already pending
      ansible.builtin.stat:
        path: /var/run/reboot-required
      register: rr

    - name: Preserve the current proxy configuration
      ansible.builtin.fetch:
        src: /etc/haproxy/haproxy.cfg
        dest: "{{ playbook_dir }}/../reports/pre-survive/"
        flat: false
      when: "'loadbalancers' in group_names"

    - name: Write the capture
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          captured: {{ ansible_date_time.iso8601 }}
          running_kernel: {{ ansible_facts.kernel }}
          installed_kernels: {{ kernels.stdout_lines | to_json }}
          boot_free_kb: {{ bootfree.stdout | trim }}
          grub_default: {{ grubdefault.stdout | trim }}
          reboot_pending: {{ rr.stat.exists }}
          uptime_seconds: {{ ansible_facts.uptime_seconds }}
        dest: "{{ playbook_dir }}/../reports/pre-survive/{{ inventory_hostname }}.yml"
        mode: '0644'
      delegate_to: localhost
      become: false

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/capture-boot.yml --limit estate

Now copy the whole capture off the controller, along with the escrow envelope. Task 8 destroys the controller and everything on it.

$ # Substitute your own off-controller destination:
ESCROW=/mnt/escrow

install -d -m 0700 "$ESCROW"
cp -a "$HOME/estate/reports" "$ESCROW/reports-pre-survive"
cp -a "$HOME/.estate-vault" "$ESCROW/vault-passwords"
cp -a "$HOME/.ssh/id_ed25519" "$ESCROW/controller-key"
chmod -R go-rwx "$ESCROW"
ls -lR "$ESCROW" | head -20

Task 2: Configuration rollback, from the plan

Somebody changed a proxy timeout by hand. Reproduce that, and then roll it back the way the plan says — which is not by restoring the file.

First, write the plan. Before the change.

docs/change-lb-timeout.md

CHANGE: haproxy timeout server 30s -> 60s on lb01
WHY:    a long-running report endpoint is being cut off at 30s

WHICH HOSTS   lb01. One host. 100% of ingress.
BLAST RADIUS  A reload, not a restart. Established connections survive.
              A configuration error would stop the proxy entirely, so
              the template's `validate: haproxy -c -f %s` is the control.

VALIDATION    - haproxy -c -f passes (enforced by the template task)
              - systemctl is-active haproxy == active
              - playbooks/health.yml passes
              - `show info` reports the process was reloaded, not restarted

ROLLBACK
  Mechanism:  revert the value in group_vars and re-run the role.
              NOT `cp haproxy.cfg.<timestamp> haproxy.cfg`.
  Command:    ansible-playbook -i inventories/production/hosts.yml \
                playbooks/site.yml --limit loadbalancers
              after reverting lb_server_timeout in group_vars.
  Time:       under 30s. One host, one template, one reload.
  Tested:     Task 2 of this lab.
  Emergency:  the timestamped backup beside the file is the path of last
              resort, for when git is unavailable. It leaves the repo
              and the host disagreeing, so the next ordinary run undoes
              it — which is why it is not the documented mechanism.

Parameterise the timeout so it is a variable rather than a literal:

# roles/estate_lb/defaults/main.yml — add
lb_server_timeout: 30s

{# roles/estate_lb/templates/haproxy.cfg.j2 — replace the literal #}
    timeout server  {{ lb_server_timeout }}

Make the change:

# inventories/production/group_vars/loadbalancers.yml — add
lb_server_timeout: 60s

$ cd "$HOME/estate"
ansible-playbook -i inventories/production/hosts.yml \
playbooks/site.yml --limit loadbalancers --check --diff \
| tee reports/lb-timeout-check.txt

TASK [estate_lb : Render the proxy configuration, validated before it is activated]
--- before: /etc/haproxy/haproxy.cfg
+++ after: /home/operator/.ansible/tmp/.../haproxy.cfg.j2
@@ -14,7 +14,7 @@
   timeout connect 5s
   timeout client  30s
-    timeout server  30s
+    timeout server  60s

changed: [lb01]

That diff is the deliverable for this task. A change whose --diff output you have read is a change you can describe; one you have not is a change you are hoping about.

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/site.yml --limit loadbalancers \
| tee reports/lb-timeout-apply.txt

ansible -i inventories/production/hosts.yml loadbalancers -b -m shell \
-a 'grep "timeout server" /etc/haproxy/haproxy.cfg; systemctl is-active haproxy'

Now roll it back, timing it, and using only what the plan says:

$ cd "$HOME/estate"

# The rollback is a repository change, not a file restore.
sed -i 's/^lb_server_timeout: 60s$/lb_server_timeout: 30s/' \
inventories/production/group_vars/loadbalancers.yml

time ansible-playbook -i inventories/production/hosts.yml \
playbooks/site.yml --limit loadbalancers \
| tee reports/lb-timeout-rollback.txt

ansible -i inventories/production/hosts.yml loadbalancers -b -m shell \
-a 'grep "timeout server" /etc/haproxy/haproxy.cfg'

Record the elapsed time in the change document. A rollback whose duration is unknown cannot be weighed against the alternative during an incident, and “we can roll back” means something different at 30 seconds than at 30 minutes.

Task 3: Patch the fleet

Patching is not a schedule, it is a decision made from evidence. Report first.

# playbooks/patch-report.yml
- name: What would patching change
  hosts: estate
  become: true
  gather_facts: true

  tasks:
    - name: Refresh the package index
      ansible.builtin.apt:
        update_cache: true
        cache_valid_time: 3600
      changed_when: false

    - name: List upgradable packages
      ansible.builtin.shell: |
        set -o pipefail
        apt-get -s -o Debug::NoLocking=1 upgrade \
          | awk '/^Inst /{print $2, $3, $4}'
      args:
        executable: /bin/bash
      register: upgradable
      changed_when: false

    - name: Is a reboot already required
      ansible.builtin.stat:
        path: /var/run/reboot-required
      register: rr

    - name: Which services would need restarting
      ansible.builtin.command: needrestart -b
      register: nr
      changed_when: false
      failed_when: false

    - name: Write the per-host patch report
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          groups: {{ group_names | to_json }}
          at: {{ ansible_date_time.iso8601 }}
          upgradable_count: {{ upgradable.stdout_lines | length }}
          kernel_upgrade_pending: >-
            {{ upgradable.stdout_lines
               | select('search', '^linux-image') | list | length > 0 }}
          reboot_already_required: {{ rr.stat.exists }}
          upgradable: |
            {{ upgradable.stdout | indent(12) }}
          needrestart: |
            {{ nr.stdout | default('needrestart not installed') | indent(12) }}
        dest: "{{ playbook_dir }}/../reports/patch-{{ inventory_hostname }}.yml"
        mode: '0644'
      delegate_to: localhost
      become: false

$ cd "$HOME/estate"
ansible-playbook -i inventories/production/hosts.yml \
playbooks/patch-report.yml --limit estate

grep -H -E 'upgradable_count|kernel_upgrade_pending' reports/patch-*.yml

reports/patch-app01.yml:upgradable_count: 14
reports/patch-app01.yml:kernel_upgrade_pending: True
reports/patch-app02.yml:upgradable_count: 14
reports/patch-app02.yml:kernel_upgrade_pending: True
reports/patch-app03.yml:upgradable_count: 14
reports/patch-app03.yml:kernel_upgrade_pending: True
reports/patch-db01.yml:upgradable_count: 11
reports/patch-db01.yml:kernel_upgrade_pending: True
reports/patch-lb01.yml:upgradable_count: 12
reports/patch-lb01.yml:kernel_upgrade_pending: True

Now the patch playbook. estate_app hosts can be batched; the singletons cannot.

# playbooks/patch.yml
- name: Patch the application tier, one host at a time
  hosts: appservers
  become: true
  gather_facts: true
  serial: 1
  max_fail_percentage: 0

  vars:
    proxy_host: "{{ groups['loadbalancers'][0] }}"

  pre_tasks:
    - name: Guardrail
      ansible.builtin.import_tasks: guard.yml

  tasks:
    - name: Drain from the proxy
      community.general.haproxy:
        state: drain
        host: "{{ inventory_hostname }}"
        backend: "{{ lb_backend_name }}"
        socket: "{{ lb_admin_socket }}"
        wait: true
        wait_interval: 1
        wait_retries: 30
      delegate_to: "{{ proxy_host }}"

    - name: Upgrade every package
      ansible.builtin.apt:
        upgrade: safe
        update_cache: true
      register: patched

    - name: Record whether the host now needs a reboot
      ansible.builtin.stat:
        path: /var/run/reboot-required
      register: rr_after

    - name: List the services the upgrade left running old code
      ansible.builtin.shell: |
        set -o pipefail
        needrestart -b | grep '^NEEDRESTART-SVC' || true
      args:
        executable: /bin/bash
      register: nr_list
      changed_when: false
      failed_when: false

    - name: Restart those services, without rebooting
      ansible.builtin.command: needrestart -r a
      when: nr_list.stdout | trim | length > 0
      changed_when: true
      failed_when: false

    - name: Health gate before returning to service
      ansible.builtin.uri:
        url: "http://{{ ansible_host }}:{{ app_port }}{{ app_health_path }}"
        status_code: 200
        timeout: 5
      register: health
      retries: 12
      delay: 5
      until: health.status == 200
      delegate_to: localhost
      become: false

    - name: Return to service
      community.general.haproxy:
        state: enabled
        host: "{{ inventory_hostname }}"
        backend: "{{ lb_backend_name }}"
        socket: "{{ lb_admin_socket }}"
        wait: true
        wait_interval: 1
        wait_retries: 30
      delegate_to: "{{ proxy_host }}"

    - name: Record the outcome
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          at: {{ lookup('pipe', 'date -Is') }}
          changed: {{ patched.changed }}
          reboot_required_after: {{ rr_after.stat.exists }}
        dest: "{{ playbook_dir }}/../reports/patched-{{ inventory_hostname }}.yml"
        mode: '0644'
      delegate_to: localhost
      become: false

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/patch.yml --limit appservers \
| tee reports/patch-run-appservers.txt

Now the singletons, and the honest answer about them:

$ ansible -i inventories/production/hosts.yml loadbalancers -b \
-m apt -a 'upgrade=safe update_cache=yes' \
| tee reports/patch-run-lb01.txt

Task 4: Rolling reboot

Reboot only the hosts that need it, and prove each one came back before touching the next.

# playbooks/reboot.yml
- name: Rolling reboot of the application tier
  hosts: appservers
  become: true
  gather_facts: true
  serial: 1
  max_fail_percentage: 0

  vars:
    proxy_host: "{{ groups['loadbalancers'][0] }}"

  pre_tasks:
    - name: Guardrail
      ansible.builtin.import_tasks: guard.yml

  tasks:
    - name: Does this host actually need a reboot
      ansible.builtin.stat:
        path: /var/run/reboot-required
      register: rr

    - name: Record the kernel and boot id before
      ansible.builtin.set_fact:
        kernel_before: "{{ ansible_facts.kernel }}"

    - name: Skip hosts that do not need it
      ansible.builtin.debug:
        msg: "{{ inventory_hostname }}: no reboot required, skipping"
      when: not rr.stat.exists

    - name: Drain from the proxy
      community.general.haproxy:
        state: drain
        host: "{{ inventory_hostname }}"
        backend: "{{ lb_backend_name }}"
        socket: "{{ lb_admin_socket }}"
        wait: true
        wait_interval: 1
        wait_retries: 30
      delegate_to: "{{ proxy_host }}"
      when: rr.stat.exists

    - name: Reboot and wait for the host to become usable
      ansible.builtin.reboot:
        msg: "Rolling reboot, {{ inventory_hostname }}"
        pre_reboot_delay: 5
        post_reboot_delay: 20
        reboot_timeout: 900
        connect_timeout: 20
        boot_time_command: "cat /proc/sys/kernel/random/boot_id"
        test_command: "systemctl is-system-running --wait"
      when: rr.stat.exists

    - name: Re-gather facts, because a reboot does not refresh them
      ansible.builtin.setup:
        gather_subset: min
      when: rr.stat.exists

    - name: Confirm the reboot-required flag is gone
      ansible.builtin.stat:
        path: /var/run/reboot-required
      register: rr_post
      when: rr.stat.exists

    - name: Refuse to continue if the flag survived the reboot
      ansible.builtin.assert:
        that: not rr_post.stat.exists
        fail_msg: >-
          {{ inventory_hostname }} rebooted and /var/run/reboot-required
          still exists. Something re-created it during boot — usually an
          unattended-upgrades run that started before you got here.
      when: rr.stat.exists

    - name: Health gate
      ansible.builtin.uri:
        url: "http://{{ ansible_host }}:{{ app_port }}{{ app_health_path }}"
        status_code: 200
        timeout: 5
      register: health
      retries: 24
      delay: 5
      until: health.status == 200
      delegate_to: localhost
      become: false

    - name: Return to service
      community.general.haproxy:
        state: enabled
        host: "{{ inventory_hostname }}"
        backend: "{{ lb_backend_name }}"
        socket: "{{ lb_admin_socket }}"
        wait: true
        wait_interval: 1
        wait_retries: 30
      delegate_to: "{{ proxy_host }}"
      when: rr.stat.exists

    - name: Record the out-of-service window
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          rebooted: {{ rr.stat.exists }}
          kernel_before: {{ kernel_before }}
          kernel_after: {{ ansible_facts.kernel }}
          uptime_after_seconds: {{ ansible_facts.uptime_seconds | default('n/a') }}
          at: {{ lookup('pipe', 'date -Is') }}
        dest: "{{ playbook_dir }}/../reports/reboot-{{ inventory_hostname }}.yml"
        mode: '0644'
      delegate_to: localhost
      become: false

Start the traffic generator from lab 3 in a second terminal before running this. The number you want is how many requests failed, and the answer should be zero.

$ cd "$HOME/estate"
ansible-playbook -i inventories/production/hosts.yml \
playbooks/reboot.yml --limit appservers \
| tee reports/reboot-run.txt

The singletons again. Say the quiet part in the record:

$ ansible -i inventories/production/hosts.yml loadbalancers -b -m reboot \
-a 'boot_time_command="cat /proc/sys/kernel/random/boot_id" test_command="systemctl is-system-running --wait" reboot_timeout=900'

$ ansible -i inventories/production/hosts.yml databases -b -m reboot \
-a 'boot_time_command="cat /proc/sys/kernel/random/boot_id" test_command="systemctl is-system-running --wait" reboot_timeout=900'

ansible-playbook -i inventories/production/hosts.yml \
playbooks/health.yml --limit estate

Watch the traffic generator during the database reboot. Every request fails, because the application’s health depends on the database and the database is a singleton. That is the architecture’s cost, measured. Put the number in the handover document.

Task 5: Rolling kernel upgrade, with a boot rollback

# playbooks/kernel.yml
- name: Rolling kernel upgrade
  hosts: appservers
  become: true
  gather_facts: true
  serial: 1
  max_fail_percentage: 0

  vars:
    proxy_host: "{{ groups['loadbalancers'][0] }}"
    boot_free_floor_kb: 512000

  pre_tasks:
    - name: Guardrail
      ansible.builtin.import_tasks: guard.yml

  tasks:
    - name: Read free space in /boot
      ansible.builtin.shell: |
        set -o pipefail
        df -Pk /boot | awk 'NR==2 {print $4}'
      args:
        executable: /bin/bash
      register: bootfree
      changed_when: false

    - name: Refuse a host that cannot hold another kernel
      ansible.builtin.assert:
        that: (bootfree.stdout | trim | int) > boot_free_floor_kb
        fail_msg: >-
          {{ inventory_hostname }}: only {{ bootfree.stdout | trim }} KB
          free in /boot. A kernel install here fails part-way, usually
          while generating the initramfs, and leaves an unbootable
          image. Clear old kernels before continuing.
        success_msg: "{{ inventory_hostname }}: {{ bootfree.stdout | trim }} KB free in /boot"

    - name: Record the running kernel
      ansible.builtin.set_fact:
        kernel_before: "{{ ansible_facts.kernel }}"

    - name: Upgrade the kernel package
      ansible.builtin.apt:
        name: linux-image-amd64
        state: latest
        update_cache: true
      register: kernel_pkg

    - name: Confirm a newer image is on disk before rebooting for it
      ansible.builtin.shell: |
        set -o pipefail
        ls -1 /boot/vmlinuz-* | sed 's|.*/vmlinuz-||' | sort -V | tail -1
      args:
        executable: /bin/bash
      register: newest_image
      changed_when: false

    - name: Refuse to reboot for a kernel that is already running
      ansible.builtin.assert:
        that: newest_image.stdout | trim != kernel_before
        fail_msg: >-
          {{ inventory_hostname }}: newest image on disk is
          {{ kernel_before }}, which is already running. Rebooting would
          take the host out of service for nothing.
        success_msg: >-
          {{ inventory_hostname }}: will boot {{ newest_image.stdout | trim }}
      when: kernel_pkg.changed

    - name: Drain from the proxy
      community.general.haproxy:
        state: drain
        host: "{{ inventory_hostname }}"
        backend: "{{ lb_backend_name }}"
        socket: "{{ lb_admin_socket }}"
        wait: true
        wait_interval: 1
        wait_retries: 30
      delegate_to: "{{ proxy_host }}"
      when: kernel_pkg.changed

    - name: Reboot into the new kernel
      ansible.builtin.reboot:
        msg: "Kernel upgrade, {{ inventory_hostname }}"
        pre_reboot_delay: 5
        post_reboot_delay: 20
        reboot_timeout: 900
        boot_time_command: "cat /proc/sys/kernel/random/boot_id"
        test_command: "systemctl is-system-running --wait"
      when: kernel_pkg.changed

    - name: Re-gather facts
      ansible.builtin.setup:
        gather_subset: min
      when: kernel_pkg.changed

    - name: Assert the host is on a different kernel
      ansible.builtin.assert:
        that: ansible_facts.kernel != kernel_before
        fail_msg: >-
          {{ inventory_hostname }} rebooted but is running
          {{ ansible_facts.kernel }}, the kernel it started on. The
          bootloader default did not change — usually because
          update-grub failed, which usually means /boot filled up.
        success_msg: "{{ inventory_hostname }}: {{ kernel_before }} -> {{ ansible_facts.kernel }}"
      when: kernel_pkg.changed

    - name: Health gate
      ansible.builtin.uri:
        url: "http://{{ ansible_host }}:{{ app_port }}{{ app_health_path }}"
        status_code: 200
        timeout: 5
      register: health
      retries: 24
      delay: 5
      until: health.status == 200
      delegate_to: localhost
      become: false

    - name: Return to service
      community.general.haproxy:
        state: enabled
        host: "{{ inventory_hostname }}"
        backend: "{{ lb_backend_name }}"
        socket: "{{ lb_admin_socket }}"
        wait: true
        wait_interval: 1
        wait_retries: 30
      delegate_to: "{{ proxy_host }}"
      when: kernel_pkg.changed

    - name: Record the transition
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          kernel_before: {{ kernel_before }}
          kernel_after: {{ ansible_facts.kernel }}
          at: {{ lookup('pipe', 'date -Is') }}
        dest: "{{ playbook_dir }}/../reports/kernel-{{ inventory_hostname }}.yml"
        mode: '0644'
      delegate_to: localhost
      become: false

$ cd "$HOME/estate"
ansible-playbook -i inventories/production/hosts.yml \
playbooks/kernel.yml --limit appservers \
| tee reports/kernel-run.txt

Now prove the rollback. One host, back to the kernel it was running, through Ansible rather than through a console — because a rollback that needs a console is a rollback you cannot do at scale.

# playbooks/kernel-rollback.yml
- name: Boot a named host into its previous kernel entry
  hosts: "{{ target }}"
  become: true
  gather_facts: true

  tasks:
    - name: Read the captured previous kernel
      ansible.builtin.set_fact:
        previous_kernel: "{{ (lookup('file', playbook_dir + '/../reports/pre-survive/' + inventory_hostname + '.yml') | from_yaml).running_kernel }}"

    - name: Confirm GRUB_DEFAULT is saved, which grub-reboot requires
      ansible.builtin.shell: |
        set -o pipefail
        grep -E '^GRUB_DEFAULT=' /etc/default/grub || echo 'GRUB_DEFAULT=0'
      args:
        executable: /bin/bash
      register: gd
      changed_when: false

    - name: Refuse if the bootloader will ignore a one-shot entry
      ansible.builtin.assert:
        that: "'saved' in gd.stdout"
        fail_msg: >-
          {{ inventory_hostname }} has {{ gd.stdout | trim }}. grub-reboot
          writes next_entry into grubenv and GRUB only honours it when
          GRUB_DEFAULT=saved. This rollback would silently boot the new
          kernel again.

    - name: Find the menu entry for the previous kernel
      ansible.builtin.shell: |
        set -o pipefail
        awk -F"'" '/^menuentry |^\s+menuentry /{print $2}' /boot/grub/grub.cfg \
          | grep -F "{{ previous_kernel }}" | head -1
      args:
        executable: /bin/bash
      register: entry
      changed_when: false

    - name: Refuse if no entry matches
      ansible.builtin.assert:
        that: entry.stdout | trim | length > 0
        fail_msg: >-
          No GRUB entry for {{ previous_kernel }} on {{ inventory_hostname }}.
          The package may have been autoremoved. There is no rollback
          target; recover from the snapshot instead.

    - name: Set the one-shot boot entry
      ansible.builtin.command: "grub-reboot '{{ entry.stdout | trim }}'"
      changed_when: true

    - name: Reboot into it
      ansible.builtin.reboot:
        msg: "Kernel rollback to {{ previous_kernel }}"
        boot_time_command: "cat /proc/sys/kernel/random/boot_id"
        test_command: "systemctl is-system-running --wait"
        reboot_timeout: 900

    - name: Re-gather facts
      ansible.builtin.setup:
        gather_subset: min

    - name: Confirm the rollback landed
      ansible.builtin.assert:
        that: ansible_facts.kernel == previous_kernel
        fail_msg: >-
          {{ inventory_hostname }} is running {{ ansible_facts.kernel }},
          not {{ previous_kernel }}. Check `grub-editenv list`.
        success_msg: "{{ inventory_hostname }} rolled back to {{ ansible_facts.kernel }}"

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/kernel-rollback.yml -e target=app03 \
| tee reports/kernel-rollback.txt

$ ansible -i inventories/production/hosts.yml estate -b \
-m command -a 'grub-editenv list'

Bring app03 forward again to complete the fleet:

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/reboot.yml --limit app03

Task 6: Rotate the database password

Two systems have to agree on this secret: PostgreSQL, which stores a hash of it, and the application, which presents it. Change either one alone and the application stops working.

Write the runbook before doing anything, because the order is the whole problem.

docs/runbook-rotate-db-password.md

SECRET: vault_db_password, production
HELD BY: PostgreSQL on db01 (as a hash), and the estate_app config on
         app01, app02, app03.

THE ORDERING PROBLEM
  PostgreSQL stores one password per role. There is no window in which
  both the old and the new password are accepted, so the change is
  atomic from the database's point of view and NOT atomic from the
  fleet's: between ALTER ROLE and the last app host restarting, every
  app host still presenting the old password is broken.

  Options, in order of preference:

  A. Two roles. Create estate_app_v2 with the new password, grant it
     the same privileges, move the fleet onto it host by host with the
     usual drain and health gate, then drop the old role. No window in
     which anything is broken. This is the correct answer and it costs
     an extra role.

  B. Maintenance window. ALTER ROLE, then push the new config to every
     host as fast as possible. Simple, and the estate is down for the
     length of it.

  C. ALTER ROLE and hope. Not an option. It is option B without the
     announcement.

  This estate uses A.

ORDER OF OPERATIONS
  1. Generate the new secret. Put it in the vault as vault_db_password_next.
  2. Create the v2 role in PostgreSQL with the new secret and identical
     grants. Both roles now work.
  3. Roll the app tier onto v2: serial 1, drain, config, restart,
     health gate, return. Same shape as a deploy.
  4. Verify no connection is using the old role:
     SELECT usename, count(*) FROM pg_stat_activity GROUP BY usename;
  5. Drop the old role.
  6. Rename in the vault: vault_db_password_next becomes
     vault_db_password. Remove the transitional variable.

ROLLBACK
  Before step 5, rollback is: roll the app tier back onto the old role.
  Both work, so it is a normal deploy.
  After step 5, rollback requires re-creating the old role, which means
  you need the old secret. DO NOT DELETE IT until step 6 is complete
  and the estate has been healthy for a full business day.

WHERE THE OLD SECRET WAS WRITTEN
  - the vault file (rotated)
  - PostgreSQL's role hash (replaced at step 5)
  - /etc/default/estate-app on three hosts (replaced at step 3)
  - possibly PostgreSQL's statement log, if log_statement was ever set
    to all. CHECK THIS. Rotation does not un-log a logged password.

Add the transitional variable:

$ cd "$HOME/estate"

# Generate the new secret first and paste it in; do not invent one.
openssl rand -base64 24

ansible-vault edit \
--vault-id production@"$HOME/.estate-vault/production" \
inventories/production/group_vars/all/vault.yml

# inside the vault file, add alongside the existing value
vault_db_password_next: REPLACE_ME_WITH_THE_GENERATED_VALUE

# playbooks/rotate-db-secret.yml
- name: Step 2 — create the successor role with identical grants
  hosts: databases
  become: true
  gather_facts: false

  tasks:
    - name: Create the v2 role
      ansible.builtin.command:
        argv:
          - psql
          - -v
          - ON_ERROR_STOP=1
          - -c
          - >-
            CREATE ROLE {{ db_user }}_v2 LOGIN PASSWORD
            '{{ vault_db_password_next }}'
      become_user: postgres
      register: created
      changed_when: created.rc == 0
      failed_when: created.rc != 0 and 'already exists' not in created.stderr
      no_log: true

    - name: Grant it what the original role has
      ansible.builtin.command:
        argv:
          - psql
          - -v
          - ON_ERROR_STOP=1
          - -d
          - "{{ db_name }}"
          - -c
          - >-
            GRANT ALL PRIVILEGES ON DATABASE {{ db_name }}
            TO {{ db_user }}_v2
      become_user: postgres
      changed_when: true

- name: Step 3 — move the fleet onto the successor role
  hosts: appservers
  become: true
  gather_facts: true
  serial: 1
  max_fail_percentage: 0

  vars:
    db_user: "{{ hostvars[groups['databases'][0]].db_user }}_v2"
    db_password: "{{ vault_db_password_next }}"
    proxy_host: "{{ groups['loadbalancers'][0] }}"

  pre_tasks:
    - name: Guardrail
      ansible.builtin.import_tasks: guard.yml

  tasks:
    - name: Drain
      community.general.haproxy:
        state: drain
        host: "{{ inventory_hostname }}"
        backend: "{{ lb_backend_name }}"
        socket: "{{ lb_admin_socket }}"
        wait: true
      delegate_to: "{{ proxy_host }}"

    - name: Write the new credentials
      ansible.builtin.copy:
        content: |
          APP_VERSION={{ app_version }}
          APP_DB_USER={{ db_user }}
          APP_DB_PASSWORD={{ db_password }}
        dest: /etc/default/estate-app
        owner: root
        group: estate
        mode: '0640'
      no_log: true
      notify: Restart estate-app

    - name: Apply now, so the gate below tests the new credentials
      ansible.builtin.meta: flush_handlers

    - name: Health gate
      ansible.builtin.uri:
        url: "http://{{ ansible_host }}:{{ app_port }}{{ app_health_path }}"
        status_code: 200
        timeout: 5
      register: health
      retries: 12
      delay: 5
      until: health.status == 200
      delegate_to: localhost
      become: false

    - name: Return to service
      community.general.haproxy:
        state: enabled
        host: "{{ inventory_hostname }}"
        backend: "{{ lb_backend_name }}"
        socket: "{{ lb_admin_socket }}"
        wait: true
      delegate_to: "{{ proxy_host }}"

  handlers:
    - name: Restart estate-app
      ansible.builtin.systemd_service:
        name: estate-app
        state: restarted

$ cd "$HOME/estate"
ansible-playbook -i inventories/production/hosts.yml \
playbooks/rotate-db-secret.yml --limit estate \
| tee reports/rotate-run.txt

Verify nothing is still using the old role before you drop it:

$ ansible -i inventories/production/hosts.yml databases -b \
--become-user postgres -m command \
-a "psql -tAc \"SELECT usename, count(*) FROM pg_stat_activity GROUP BY usename\""

estate_app_v2|3
postgres|1

$ ansible -i inventories/production/hosts.yml databases -b \
--become-user postgres -m command \
-a 'psql -c "DROP ROLE IF EXISTS estate_app"'

Finally, rotate the vault password itself. The passphrase protecting the secrets is a secret too, and it has the same lifecycle.

$ cd "$HOME/estate"
umask 077
openssl rand -base64 32 > "$HOME/.estate-vault/production.new"
chmod 0600 "$HOME/.estate-vault/production.new"

ansible-vault rekey \
--vault-id production@"$HOME/.estate-vault/production" \
--new-vault-id production@"$HOME/.estate-vault/production.new" \
inventories/production/group_vars/all/vault.yml

head -1 inventories/production/group_vars/all/vault.yml

Rekey successful
$ANSIBLE_VAULT;1.2;AES256;production

The header still says production: --new-vault-id keeps the label and changes only the password, which is what makes the rekey invisible to every playbook. Swap the files, re-run something read-only to confirm, and update the escrow envelope in the same breath — a rekey whose new password is only on the controller has just made your escrow useless.

$ # Substitute your own off-controller escrow path:
ESCROW=/mnt/escrow

ansible-vault view \
--vault-id production@"$HOME/.estate-vault/production.new" \
inventories/production/group_vars/all/vault.yml > /dev/null \
&& echo 'new password decrypts: OK'

mv "$HOME/.estate-vault/production.new" "$HOME/.estate-vault/production"
cp -a "$HOME/.estate-vault/production" "$ESCROW/vault-passwords/production"
chmod 0600 "$ESCROW/vault-passwords/production"

Task 7: The rehearsal before the controller is destroyed

Before Task 8 takes the controller away, prove you could rebuild it. The question is not “is the repository in git” — it is “is everything the controller does reproducible from things that are not on the controller”.

$ cd "$HOME/estate"

echo '--- uncommitted work ---'
git status --short

echo '--- what the config depends on outside the checkout ---'
ansible-config dump --only-changed | grep -E 'VAULT|LOG_PATH|COLLECTIONS'

echo '--- collections, and whether requirements.yml would reproduce them ---'
ansible-galaxy collection list 2>/dev/null | head -20
cat requirements.yml

echo '--- the key that reaches the fleet ---'
ssh-keygen -lf "$HOME/.ssh/id_ed25519.pub"

Anything in that output that is not reproducible from the repository plus the escrow envelope is a gap. Write each one into docs/controller-rebuild.md with how it is recovered.

Task 8: Lose the controller

$ # Confirm the escrow first. Do not proceed if either line fails.
ESCROW=/mnt/escrow
test -s "$ESCROW/vault-passwords/production" && echo 'vault escrow present'
test -s "$ESCROW/controller-key" && echo 'key escrow present'

shred -u "$HOME/.estate-vault/production" "$HOME/.estate-vault/staging"
rm -rf "$HOME/.estate-vault"
rm -f "$HOME/.ssh/id_ed25519" "$HOME/.ssh/id_ed25519.pub"
rm -rf "$HOME/estate"

Confirm the estate is still serving. It is, and that is the point: the controller is not in the request path.

$ # Substitute your own value before running:
VIP=192.0.2.11

curl -s "http://$VIP/" | head -3

Now rebuild. On a fresh machine, or on the same one, from nothing but the repository and the escrow envelope.

$ # Substitute your own values before running:
ESCROW=/mnt/escrow
REPO=git@git.example.com:platform/estate.git

# 1. A pinned Ansible, in its own environment.
python3 -m venv "$HOME/ansible-venv"
"$HOME/ansible-venv/bin/pip" install --upgrade pip
"$HOME/ansible-venv/bin/pip" install 'ansible-core==2.21.*'
"$HOME/ansible-venv/bin/ansible" --version | head -2

# 2. The repository.
git clone "$REPO" "$HOME/estate"
cd "$HOME/estate"

# 3. The collections, from the pinned requirements.
"$HOME/ansible-venv/bin/ansible-galaxy" collection install -r requirements.yml

# 4. The secrets, from escrow.
install -d -m 0700 "$HOME/.estate-vault"
cp "$ESCROW/vault-passwords/production" "$HOME/.estate-vault/production"
cp "$ESCROW/vault-passwords/staging" "$HOME/.estate-vault/staging"
chmod 0600 "$HOME/.estate-vault"/*

# 5. The key that reaches the fleet.
install -d -m 0700 "$HOME/.ssh"
cp "$ESCROW/controller-key" "$HOME/.ssh/id_ed25519"
chmod 0600 "$HOME/.ssh/id_ed25519"
ssh-keygen -y -f "$HOME/.ssh/id_ed25519" > "$HOME/.ssh/id_ed25519.pub"

The proof is a check-mode run against production reporting no drift. If the rebuild is complete, the new controller’s view of the estate and the estate’s actual state are identical.

$ cd "$HOME/estate"
"$HOME/ansible-venv/bin/ansible-playbook" \
-i inventories/production/hosts.yml \
playbooks/site.yml --limit estate --check --diff \
| tee reports/rebuild-proof.txt

grep -E 'changed=[1-9]' reports/rebuild-proof.txt \
|| echo 'REBUILD PROVEN: no drift between repository and estate'

Task 9: The handover

The last deliverable, and the one that outlives everything else.

docs/handover.md

WHAT THIS IS
  A five-host production estate: one proxy, three application hosts,
  one database, managed from a rebuildable controller.
  Repository: git@git.example.com:platform/estate.git

HOW TO CHANGE IT
  Always: staging first, canary second, batched fleet third.
  Deploy:   playbooks/deploy.yml -e app_version=X --limit <hosts>
  Patch:    playbooks/patch-report.yml, read it, then playbooks/patch.yml
  Reboot:   playbooks/reboot.yml — reboots only hosts that need it
  Rollback: revert in group_vars and re-run. NOT a file restore.

WHAT IT COSTS TO RUN
  Patching the proxy       ~4 min full ingress outage. No redundancy.
  Rebooting the database   ~90 s total application outage. No replica.
  Rolling the app tier     zero downtime, ~2 min per host.
  Rebuilding the controller ~20 min from escrow. Proven, see
                            reports/rebuild-proof.txt.

THE FIVE THINGS THAT WILL BREAK FIRST
  1. lb01. One proxy is 100% of ingress and it is patched in place.
     Fix: add lb02. The inventory is already shaped for it.
  2. db01. No replica, so every database event is an application
     outage, and the only rollback is a restore.
     Fix: a streaming replica, then this tier becomes rollable.
  3. The vault passwords. Escrowed to one location. If that location
     is lost at the same time as the controller, the estate is
     unmanageable until every secret is re-created by hand.
     Fix: a second escrow custodian.
  4. /boot capacity. The kernel play asserts 500 MB and refuses below
     it, which converts the failure into a refusal — but nothing
     currently removes old kernels, so hosts trend towards the floor.
     Fix: a scheduled autoremove, with the running kernel protected.
  5. The controller's SSH key. NOPASSWD:ALL on six hosts means one key
     is total fleet compromise, and it has never been rotated.
     Fix: a rotation procedure, and a narrower sudo grant derived from
     a month of sudo logs.

WHAT NOBODY HAS TESTED
  - A restore of db01 from backup. There is no backup. This is the
    largest gap in the estate and it is deliberate that it is named
    here rather than implied by its absence.
  - Behaviour under load. Every health check in this estate is a
    single request.

That “what nobody has tested” section is the most valuable part of the document. Every estate has one; most handover documents omit it, which is how the next person finds out the hard way.

Validation

$ cd "$HOME/estate"
A="$HOME/ansible-venv/bin/ansible"
AP="$HOME/ansible-venv/bin/ansible-playbook"

# 1. The rollback was executed from the plan, and the value is back.
$A -i inventories/production/hosts.yml loadbalancers -b -m shell \
-a 'grep "timeout server" /etc/haproxy/haproxy.cfg'

# 2. Every host was patched and the report says what changed.
grep -H 'reboot_required_after' reports/patched-*.yml

# 3. Every host rebooted onto a different kernel, and one was rolled back.
grep -H -E 'kernel_before|kernel_after' reports/kernel-*.yml
grep -E 'rolled back' reports/kernel-rollback.txt

# 4. No host has a pending one-shot boot entry.
$A -i inventories/production/hosts.yml estate -b \
-m command -a 'grub-editenv list' | grep -c next_entry || echo 'NO PENDING ENTRIES'

# 5. The old database role is gone and the new one is in use.
$A -i inventories/production/hosts.yml databases -b --become-user postgres \
-m command -a "psql -tAc \"SELECT rolname FROM pg_roles WHERE rolname LIKE 'estate_app%'\""

# 6. The rebuilt controller sees no drift.
grep -E 'changed=[1-9]' reports/rebuild-proof.txt || echo 'NO DRIFT'

# 7. The estate is healthy.
$AP -i inventories/production/hosts.yml playbooks/health.yml --limit estate

Every line must pass:

• timeout server 30s — the rollback returned the value, and reports/lb-timeout-rollback.txt records how long it took.
• Every patched-*.yml exists and states whether a reboot became required.
• Every kernel-*.yml shows kernel_before different from kernel_after, and app03 was rolled back and forward again.
• No host reports a next_entry in grub-editenv list.
• pg_roles contains estate_app_v2 and does not contain estate_app.
• The check-mode run from the rebuilt controller reports changed=0 on every host.
• health.yml passes for the whole estate.

Expected Outcome

estate/                         (on a controller that did not exist an hour ago)
├── docs/
│   ├── change-lb-timeout.md
│   ├── controller-rebuild.md
│   ├── handover.md
│   └── runbook-rotate-db-password.md
├── playbooks/
│   ├── capture-boot.yml
│   ├── kernel.yml
│   ├── kernel-rollback.yml
│   ├── patch.yml
│   ├── patch-report.yml
│   ├── reboot.yml
│   └── rotate-db-secret.yml
└── reports/
    ├── kernel-app0{1,2,3}.yml
    ├── kernel-rollback.txt
    ├── lb-timeout-{check,apply,rollback}.txt
    ├── patch-{app01,app02,app03,db01,lb01}.yml
    ├── patched-app0{1,2,3}.yml
    ├── pre-survive/*.yml
    ├── rebuild-proof.txt
    ├── reboot-app0{1,2,3}.yml
    └── rotate-run.txt

Six hosts on a new kernel, fully patched, serving through a proxy whose timeout was changed and rolled back, authenticating to the database with a password that was rotated without an outage — all managed from a controller that was destroyed and rebuilt, and which reports zero drift against the estate it inherited.

Troubleshooting

The reboot task hangs until reboot_timeout. The host rebooted and test_command never succeeded. Try test_command: whoami to distinguish “never came back” from “came back degraded”: if whoami works and systemctl is-system-running --wait does not, a unit is failing and the host is up but not healthy.

boot_time_command returns the same value after the reboot. The machine did not reboot. With /proc/sys/kernel/random/boot_id that is unambiguous.

The host comes back on the same kernel. The bootloader default did not change. update-grub runs from the kernel package’s postinst; if it failed, /boot/grub/grub.cfg has no entry for the new image. The usual cause is a full /boot, which is what the Task 5 assertion exists to prevent.

grub-reboot succeeds and the host boots the new kernel anyway. GRUB_DEFAULT is not saved, so GRUB ignores the saved next-entry. The rollback play asserts this before acting; if you bypassed the assert, grub-editenv list shows the entry was written and ignored.

apt fails with “Could not get lock”. unattended-upgrades holds the dpkg lock. Wait for it, or disable it on hosts where you own the patching. Never remove the lock file — that is how a package ends up half-configured.

The application cannot authenticate after the rotation. Check which role each host is presenting: grep APP_DB_USER /etc/default/estate-app. A host that failed its health gate mid-rotation kept the old role and is drained. That is the designed outcome; complete it with a narrowed --limit, do not drop the old role until every host has moved.

ansible-vault reports “Decryption failed” after the rekey. The new password file was moved into place before something else had finished using the old one, or the escrow copy is the old password. head -1 confirms the label; only the password changed.

The rebuilt controller reports drift on tasks that use command. Those tasks are skipped in check mode, so a changed result there is usually a changed_when that evaluates true regardless. Read the task.

The rebuilt controller cannot reach any host. The key’s permissions. chmod 0600 on the private key, and confirm the public key derived with ssh-keygen -y matches what authorized_keys on a node contains.

Cleanup

This is the end of the capstone. Cleanup here restores the estate to a defined state and preserves the deliverables.

Step 1. Remove anything transitional. A leftover one-shot boot entry or failure marker will surface weeks later looking like a new fault:

$ cd "$HOME/estate"
A="$HOME/ansible-venv/bin/ansible"

$A -i inventories/production/hosts.yml estate -b \
-m command -a 'grub-editenv - unset next_entry'

$A -i inventories/production/hosts.yml appservers -b \
-m file -a 'path=/etc/estate-app-fail state=absent'

$A -i inventories/production/hosts.yml estate -b \
-m command -a 'grub-editenv list'

Step 2. Confirm no host is left drained:

$ $HOME/ansible-venv/bin/ansible \
-i inventories/production/hosts.yml loadbalancers -b -m shell \
-a 'echo "show stat" | socat stdio /run/haproxy/admin.sock | cut -d, -f1,2,18 | grep estate_backend'

Step 3. Tidy the vault. The transitional variable is no longer used and leaving it means the next reader cannot tell which secret is live:

$ $HOME/ansible-venv/bin/ansible-vault edit \
--vault-id production@"$HOME/.estate-vault/production" \
inventories/production/group_vars/all/vault.yml

Step 4. Preserve the deliverables outside the working tree, because this is the artefact the capstone produced:

$ mkdir -p "$HOME/estate-deliverables/capstone-4"
cp -a "$HOME/estate/docs" "$HOME/estate/reports" \
    "$HOME/estate-deliverables/capstone-4/"

ls "$HOME/estate-deliverables"/*/docs/*.md

Step 5. Decide whether to keep the estate. Keeping it is a legitimate choice — it is a working reference environment. If you are tearing it down, do it in dependency order: proxy, then application, then database, using the Cleanup sections from capstone labs 2 and 1 in that order.

Step 6. Final state check, whichever you chose:

$ $HOME/ansible-venv/bin/ansible-playbook \
-i inventories/production/hosts.yml \
playbooks/health.yml --limit estate \
|| echo 'estate is not serving — confirm this is intentional'

Step 7. If you are finished entirely, destroy the escrow envelope and the snapshots last, and only when you are certain you no longer need the evidence.

What You Learned

• needrestart -b reports and never restarts, so -r a -b is a task that succeeds while doing nothing. Report with one invocation, act with another.
• A rollback is a repository change, not a file restore. Copying the timestamped backup works until the next ordinary run re-renders the template, at a time nobody chose.
• Patching is a decision made from a simulation. apt-get -s with Debug::NoLocking=1 tells you what would change and what would be removed, and upgrade: full without reading it is how a dependency disappears.
• You cannot roll a fleet of one. serial: 1 against a singleton is a single unbatched run, and the correct control is a maintenance window — an organisational thing no keyword provides.
• boot_time_command proves a reboot happened; test_command proves the host is usable. They answer different questions and both defaults are weaker than they look.
• Facts do not refresh across a reboot, so without an explicit setup the assertion comparing kernels compares a value with itself.
• grub-reboot is one-shot, which stops a failed rollback compounding and means a host you believe is pinned is pinned to nothing.
• A rotation has an ordering problem whenever two systems must agree. Two roles with overlapping validity removes the window; a maintenance window accepts it; changing one side and hoping is neither.
• Rotation is incomplete until you know where the old secret was written — including a backup taken before the change, which un-rotates it on restore.
• The controller is not in the request path. Losing it costs you the ability to change the estate, not the estate. That makes it a recovery problem, not an availability problem, and a twenty-minute rebuild from escrow is worth more than a highly available controller.
• --check proves that everything the repository manages matches the repository. It does not prove the estate is correct, and knowing the difference is the last thing this capstone teaches.