Capstone 3: Operate it

Capstone lab 3 of 4. It operates the estate built in capstone lab 1 and deployed in capstone lab 2. The service must be running and healthy before you start.

Objective

By the end you will have taken one version bump from staging to production through a canary and a batched rollout, absorbed a deliberate mid-fleet failure without improvising, and produced a written assessment and an audit record that together answer the question every incident review asks: which hosts changed, and to what?

The command sequences in this lab are the easy part. The deliverable is the judgement: at every stage, what will change, on which hosts, how is it validated, and how is it undone.

Architecture

The same estate, now being changed while it is serving.

            client traffic
                  │
             ┌────▼────┐
             │  lb01   │   drain/ready via admin socket
             └────┬────┘
      ┌───────────┼───────────┐
      ▼           ▼           ▼
   app01       app02       app03
  ──────────────────────────────────
   batch 1     batch 2     batch 2      serial: [1, 2]
   canary      ← failure injected here

The rollout is serial: [1, 2]: one host, then the remaining two. The first batch is the canary. If it fails, the play stops and two hosts have never been touched.

Capacity arithmetic, which belongs in the change record and not in your head: three hosts serving, one draining, leaves 67 per cent of capacity. Two draining leaves 33 per cent. That is why the second batch is two and not three, and it is a decision about your traffic, not a default.

Requirements

Capstone labs 1 and 2 complete. playbooks/health.yml must currently pass against production.
ansible-core 2.21.x and community.general for the haproxy module.
Six VMs with systemd as PID 1. B-nested only. This lab measures real service interruption across real restarts; a simulated restart measures nothing.
A second terminal on the controller, for the traffic generator in Task 5. The measurement is the point of that task.
Roughly 4 hours. Do not start Task 7 with less than 90 minutes left — the assessment is the part that takes thought, and rushing it defeats the exercise.

Verification status: the execution semantics here were tested, the service steps were not

This lab has not been run end to end on an estate, and the split matters more here than elsewhere, so it is worth being precise.

Executed against ansible-core 2.21.3, and the claims made from them are measurements rather than readings of the documentation:

serial: [1, 2] re-enters the play once per batch, and a failing canary batch stops it with the later hosts never touched.
max_fail_percentage aborts only when the failed percentage exceeds the value. One host of five failing continued at 20 and aborted at 0. That is the arithmetic the Task 3 callout rests on.
run_once: true under serial: 2 executed three times across five hosts — once per batch.
Handlers flushed at the end of each batch, three times, not once.
A # line in a --limit @file produced Could not match supplied host pattern, as Task 8 describes.

Not executed: the drain and return-to-service against a real HAProxy, the traffic generator and its response-code counts, the service restarts, and the injected failure on app02. Every recap and client-log figure in this lab is labelled illustrative.

So the shape of the rollout is verified and its effects on a running service are not. Your own numbers from Task 5 are the real ones.

Scenario

Version 1.1.0 of the application is ready. The change is small — the health endpoint gains a field — and the previous three deployments of this application were done by running site.yml against appservers with no batching, which worked twice and caused a nine-minute outage the third time.

Your job is to deploy 1.1.0 without an outage, and to be able to prove afterwards exactly what happened.

Tasks

Task 1: The second-run test

Before changing anything, establish that the current playbooks are honest. Run site.yml against production twice with no modifications between the runs.

Service impact possiblecontroller— First run against the whole estate. Handlers may restart services if anything has drifted.

$ cd "$HOME/estate"
ansible-playbook -i inventories/production/hosts.yml \
playbooks/site.yml --limit estate \
| tee reports/idempotency-run1.txt

Service impact possiblecontroller— Second run, immediately after, with nothing changed in between. Must report changed=0.

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/site.yml --limit estate \
| tee reports/idempotency-run2.txt

grep -E 'changed=[1-9]' reports/idempotency-run2.txt \
|| echo 'IDEMPOTENT: second run changed nothing'

PLAY RECAP *********************************************************************
app01  : ok=14  changed=0  unreachable=0  failed=0  skipped=1
app02  : ok=14  changed=0  unreachable=0  failed=0  skipped=1
app03  : ok=14  changed=0  unreachable=0  failed=0  skipped=1
db01   : ok=11  changed=0  unreachable=0  failed=0  skipped=0
lb01   : ok=7   changed=0  unreachable=0  failed=0  skipped=0

IDEMPOTENT: second run changed nothing

Illustrative output

The second run carries a SERVICE-IMPACT badge and not READ-ONLY because you do not yet know it will change nothing. That is the whole point of running it.

If any host reports a change, find the task and fix it before going further. The three that commonly do, and what each one really is:

Task reports changed every run	What it actually is	Fix
A `command` with no `changed_when`	Ansible cannot know, so it assumes yes	Add `changed_when:` comparing before and after, or `changed_when: false` for a read
A `template` whose output contains a timestamp	The file genuinely differs each render	Remove the timestamp, or move it to a file that is not managed
A `copy` with `content:` ending without a newline	The file on disk gains one; the comparison never matches	End the content with `\n`

Task 2: The change record and the rollback plan, written first

The rollback is planned before the change, not discovered during it. Write docs/change-2026-08-12-app-110.md now:

CHANGE: estate-app 1.0.0 -> 1.1.0

WHAT CHANGES
  inventories/production/group_vars/all/main.yml  app_version: 1.1.0
  On each app host: /opt/estate-app/app.py re-rendered, service restarted.
  Nothing on db01. Nothing on lb01 except backend state during drain.

WHICH HOSTS
  app01, app02, app03.  Three of five. 100% of the application tier.
  db01 and lb01 are NOT in scope and must not appear in any --limit.

BLAST RADIUS (from docs/blast-radius.md)
  appservers = 3 hosts. Losing one = 33% capacity. Losing two = 67%.
  Batching: serial [1, 2]. Never more than one host out of service at a
  time in batch 1; never more than two in batch 2.

VALIDATION AT EACH STAGE
  Per host, before return to service:
    - systemd reports estate-app active
    - GET /health on the host returns 200
    - the JSON version field equals 1.1.0
  After the last host:
    - playbooks/health.yml passes for the whole estate
    - all three backends report UP on the proxy

ABORT CRITERIA — decided now, not during the incident
  Abort if: the canary host fails its health gate;
            the proxy reports two or more backends DOWN at once;
            any host takes longer than 120s to return 200.

ROLLBACK
  Mechanism: re-run the deploy with app_version 1.0.0.
  Command:   HOSTS=appservers        # narrow this to the affected hosts
             ansible-playbook -i inventories/production/hosts.yml \
               playbooks/deploy.yml -e app_version=1.0.0 --limit "$HOSTS"
  Time:      ~40s per host, same drain and health gate.
  Tested:    yes — see reports/rollback-drill.txt (Task 8).
  NOT covered by this rollback: any database schema change. There is
  none in 1.1.0. If there were, the rollback would not be symmetric and
  this section would say so.

WHO KNOWS
  Change ticket CHG-0000 raised. Proxy logs retained for the window.

Task 3: The deploy playbook

site.yml builds the estate. Deployment is a narrower operation and deserves its own entry point, because the safety controls differ.

# playbooks/deploy.yml
- name: Rolling application deployment
  hosts: appservers
  become: true
  gather_facts: true

  serial:
    - 1
    - 2
  max_fail_percentage: 0

  vars:
    drain_settle_seconds: 5
    health_retries: 12
    health_delay: 5
    proxy_host: "{{ groups['loadbalancers'][0] }}"

  pre_tasks:
    - name: Guardrail
      ansible.builtin.import_tasks: guard.yml

    - name: Record the version this host is running before the change
      ansible.builtin.uri:
        url: "http://{{ ansible_host }}:{{ app_port }}{{ app_health_path }}"
        status_code: [200, 503]
        return_content: true
        timeout: 5
      register: before
      delegate_to: localhost
      become: false

    - name: Remember it
      ansible.builtin.set_fact:
        version_before: "{{ (before.content | from_json).version }}"

  tasks:
    - name: Drain this host from the proxy
      community.general.haproxy:
        state: drain
        host: "{{ inventory_hostname }}"
        backend: "{{ lb_backend_name }}"
        socket: "{{ lb_admin_socket }}"
        wait: true
        wait_interval: 1
        wait_retries: 30
      delegate_to: "{{ proxy_host }}"
      become: true

    - name: Let in-flight requests finish
      ansible.builtin.wait_for:
        timeout: "{{ drain_settle_seconds }}"
      delegate_to: localhost
      become: false

    - name: Apply the application role at the new version
      ansible.builtin.include_role:
        name: estate_app

    - name: Wait for the port to accept connections again
      ansible.builtin.wait_for:
        host: "{{ ansible_host }}"
        port: "{{ app_port }}"
        state: started
        timeout: 60
      delegate_to: localhost
      become: false

    - name: Health gate on the host itself, before it sees traffic
      ansible.builtin.uri:
        url: "http://{{ ansible_host }}:{{ app_port }}{{ app_health_path }}"
        status_code: 200
        return_content: true
        timeout: 5
      register: after
      retries: "{{ health_retries }}"
      delay: "{{ health_delay }}"
      until: after.status == 200
      delegate_to: localhost
      become: false

    - name: Confirm it is serving the version we deployed
      ansible.builtin.assert:
        that: (after.content | from_json).version == app_version
        fail_msg: >-
          {{ inventory_hostname }} answers 200 but reports version
          {{ (after.content | from_json).version }}, not {{ app_version }}.
          The file was written and the process was not replaced. Do not
          return this host to service.
        success_msg: >-
          {{ inventory_hostname }}: {{ version_before }} -> {{ app_version }}

    - name: Return the host to service
      community.general.haproxy:
        state: enabled
        host: "{{ inventory_hostname }}"
        backend: "{{ lb_backend_name }}"
        socket: "{{ lb_admin_socket }}"
        wait: true
        wait_interval: 1
        wait_retries: 30
      delegate_to: "{{ proxy_host }}"
      become: true

    - name: Write the per-host audit record
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          at: {{ lookup('pipe', 'date -Is') }}
          version_before: {{ version_before }}
          version_after: {{ (after.content | from_json).version }}
          batch: {{ ansible_play_batch | join(',') }}
          run_by: {{ lookup('env', 'USER') }}
        dest: "{{ playbook_dir }}/../reports/deploy-{{ inventory_hostname }}.yml"
        mode: '0644'
      delegate_to: localhost
      become: false

Two keywords do the safety work, and both were checked against ansible-core 2.21.3 rather than assumed.

serial: [1, 2] splits the play into batches of one host and then two. The play is re-entered per batch — you will see PLAY [Rolling application deployment] printed three times for five hosts, or twice for three.

max_fail_percentage: 0 aborts the play when more than zero per cent of a batch fails. The comparison is strictly greater-than, which is worth proving to yourself:

Read-only / Safecontroller — arithmetic worth knowing before you rely on it— Verified against ansible-core 2.21.3 on a five-host play.

$ # 1 host of 5 fails, max_fail_percentage: 20  ->  20 is not > 20  ->  CONTINUES
# 1 host of 5 fails, max_fail_percentage: 0   ->  20 is  > 0   ->  NO MORE HOSTS LEFT
# 1 host of 3 fails, max_fail_percentage: 20  ->  33 is  > 20  ->  aborts

echo 'max_fail_percentage aborts when the failed percentage EXCEEDS it.'

Task 4: Staging first, and what staging cannot prove

Bump the version in staging and run the deploy.

Service impact possiblecontroller— Deploys 1.1.0 to stg01, including the real drain and health gate.

$ cd "$HOME/estate"
ansible-playbook -i inventories/staging/hosts.yml \
playbooks/deploy.yml -e app_version=1.1.0 \
| tee reports/staging-deploy-110.txt

It passes. Now write down what it did not tell you — this is the second deliverable and it is the one most teams skip.

docs/promotion-record-110.md

STAGING RUN: reports/staging-deploy-110.txt — PASSED

WHAT STAGING PROVED
  - The template renders with the new version and compiles.
  - The unit restarts and the service comes back.
  - /health returns 200 and reports 1.1.0.
  - The drain and enable calls against the admin socket work.

WHAT STAGING COULD NOT PROVE
  - Batching. staging has ONE host in appservers, so serial [1, 2]
    produced a single batch of one. The second batch was never
    exercised, so a bug that only appears with two hosts in a batch —
    a shared resource, a lock, a race on the proxy socket — is
    undetected.
  - Capacity during drain. Draining the only host removed 100% of
    staging capacity; nothing measured what draining one of three does.
  - Cross-host version skew. In production, app01 runs 1.1.0 while
    app02 and app03 still run 1.0.0 for the length of the rollout. On
    staging that state cannot exist. If 1.1.0 changed a shared data
    format, staging would never see the incompatibility.
  - Real proxy behaviour. staging's proxy backend has one server, so
    "the proxy has other healthy backends to route to" was never true.

CONCLUSION
  Staging authorises the change mechanically. It does not authorise the
  rollout shape. The canary in production is what tests the shape, and
  it is not optional because staging passed.

Task 5: Measure the interruption, then canary

Start a traffic generator in a second terminal and leave it running for the whole rollout. What you are measuring is whether drain actually works.

Read-only / Safecontroller, second terminal— Polls the proxy continuously and records the status code of every response.

$ # Substitute your own values before running:
VIP=192.0.2.11
OUT="$HOME/estate/reports/client-rollout.txt"

: > "$OUT"
while true; do
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 2 "http://$VIP/")
printf '%s %s\n' "$(date +%s.%N)" "$code" >> "$OUT"
sleep 0.05
done

Now bump the version in the inventory and run the canary — one host, explicitly, before the fleet.

# inventories/production/group_vars/all/main.yml
app_version: '1.1.0'

Service impact possiblecontroller— Deploys 1.1.0 to app01 only. One host of three; the other two are untouched and still serving 1.0.0.

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/deploy.yml --limit app01 \
| tee reports/canary-app01.txt

Stop and evaluate against the abort criteria you wrote in Task 2 before running anything else. The canary is only a canary if there is a pause after it.

Read-only / Safecontroller— The canary evidence: version per host, and the proxy's own view of each backend.

$ cd "$HOME/estate"

for h in 192.0.2.21 192.0.2.22 192.0.2.23; do
printf '%s ' "$h"
curl -s --max-time 2 "http://$h:8080/health" | jq -r '.version + " " + .status'
done

ansible -i inventories/production/hosts.yml loadbalancers -b -m shell \
-a 'echo "show stat" | socat stdio /run/haproxy/admin.sock | cut -d, -f1,2,18 | grep estate_backend'

192.0.2.21 1.1.0 ok
192.0.2.22 1.0.0 ok
192.0.2.23 1.0.0 ok
estate_backend,app01,UP
estate_backend,app02,UP
estate_backend,app03,UP

Illustrative output

One host on the new version, all three backends UP, and the client log should show no 5xx at all. Check that:

Read-only / Safecontroller— Counts response codes seen by the client during the canary.

$ awk '{print $2}' "$HOME/estate/reports/client-rollout.txt" | sort | uniq -c

   1843 200

Illustrative output

Every 200 means the drain did its job: the proxy stopped sending requests to app01 before the service was restarted, and the other two carried the load. Any 503 means the drain did not settle before the restart — increase drain_settle_seconds and investigate before continuing.

Task 6: Roll the rest

Service impact possiblecontroller— Deploys 1.1.0 to app02 and app03 as one batch. Two of three hosts drain and restart; capacity falls to 33% briefly.

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/deploy.yml --limit 'appservers:!app01' \
| tee reports/rollout-remaining.txt

Two of three draining at once is a capacity decision, and it is arguable

serial: [1, 2] on a three-host tier puts two hosts out of service simultaneously in the second batch. One host carries all the traffic for the length of that batch.

Whether that is acceptable is arithmetic about your load, not a default. If one host cannot carry the peak, the correct shape is serial: 1 all the way through — slower, and with the tier never below two thirds.

The reason this lab uses [1, 2] is to make the trade-off visible: the canary is deliberately small and the follow-on batch is deliberately as large as the tier can stand. On a tier of twenty, serial: [1, 5, 10] is the same idea at a different scale. On a tier of three, [1, 2] and [1, 1, 1] are both defensible and you should be able to say which one your traffic requires.

Confirm the fleet and the client log:

Read-only / Safecontroller— Final state after the rollout: every host on the new version, every backend UP, no 5xx seen by the client.

$ cd "$HOME/estate"

ansible-playbook -i inventories/production/hosts.yml \
playbooks/health.yml --limit estate

awk '{print $2}' reports/client-rollout.txt | sort | uniq -c
cat reports/deploy-app0*.yml

   4102 200

host: app01
at: 2026-08-12T10:14:07+00:00
version_before: 1.0.0
version_after: 1.1.0
batch: app01
host: app02
...

Illustrative output

Now stop the traffic generator with Ctrl-C in the second terminal.

Task 7: The partial fleet failure

This is the part of the capstone that is about judgement rather than commands. You will break one host, run a rollout that fails midway, and then write an assessment before touching anything.

First, revert the fleet to 1.0.0 so there is a real change to make. This is also your rollback drill.

Service impact possiblecontroller— Rolls all three application hosts back to 1.0.0 using the documented rollback command. This is the drill.

$ cd "$HOME/estate"
ansible-playbook -i inventories/production/hosts.yml \
playbooks/deploy.yml -e app_version=1.0.0 --limit appservers \
| tee reports/rollback-drill.txt

That file is the evidence for the “Tested: yes” line in your change record. A rollback plan whose command has never been run is a plan whose first execution is during an incident.

Now inject the fault. app02 gets the marker file that makes /health return 503 while the process is otherwise perfectly healthy.

Service impact possiblecontroller— Creates the failure marker on app02. Its /health endpoint begins returning 503 immediately; the proxy marks it DOWN within fall x inter.

$ ansible -i inventories/production/hosts.yml app02 -b \
-m file -a 'path=/etc/estate-app-fail state=touch mode=0644'

Run the rollout to 1.1.0 across the whole tier and watch it stop.

Service impact possiblecontroller— Rolls 1.1.0 across three hosts. Expected to fail during the second batch, on app02.

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/deploy.yml --limit appservers \
| tee reports/partial-failure.txt

echo "exit status: ${PIPESTATUS[0]}"

PLAY [Rolling application deployment] ******************************************
TASK [Health gate on the host itself, before it sees traffic] ******************
ok: [app01]
...
PLAY [Rolling application deployment] ******************************************
TASK [Health gate on the host itself, before it sees traffic] ******************
FAILED - RETRYING: [app02] (12 retries left).
...
fatal: [app02]: FAILED! => {"attempts": 12, "status": 503}

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
app01  : ok=12  changed=3  unreachable=0  failed=0
app02  : ok=8   changed=3  unreachable=0  failed=1
app03  : ok=4   changed=0  unreachable=0  failed=0

exit status: 2

Illustrative output

Stop here. Before running anything else, gather evidence and write the assessment. These commands are read-only.

Read-only / Safecontroller— Establishes the true state of the fleet after the aborted rollout. Changes nothing.

$ cd "$HOME/estate"

echo '--- what each host is actually serving ---'
for h in 192.0.2.21 192.0.2.22 192.0.2.23; do
printf '%s ' "$h"
curl -s --max-time 2 "http://$h:8080/health" \
  | jq -r '.version + " " + .status' || echo 'no response'
done

echo '--- what the proxy believes ---'
ansible -i inventories/production/hosts.yml loadbalancers -b -m shell \
-a 'echo "show stat" | socat stdio /run/haproxy/admin.sock | cut -d, -f1,2,18 | grep estate_backend'

echo '--- what the run recorded ---'
cat reports/deploy-app0*.yml

echo '--- what is serving through the front door ---'
for i in 1 2 3 4 5 6; do
curl -s --max-time 2 http://192.0.2.11/ | jq -r '.host + " " + .version'
done

--- what each host is actually serving ---
192.0.2.21 1.1.0 ok
192.0.2.22 1.1.0 unfit
192.0.2.23 1.0.0 ok
--- what the proxy believes ---
estate_backend,app01,UP
estate_backend,app02,MAINT
estate_backend,app03,UP
--- what is serving through the front door ---
app01 1.1.0
app03 1.0.0
app01 1.1.0
app03 1.0.0
app01 1.1.0
app03 1.0.0

Illustrative output

Now write docs/incident-partial-rollout.md. Answer these six questions in writing, with evidence, before you take any action:

docs/incident-partial-rollout.md

1. WHICH HOSTS CHANGED?
   app01: changed, 1.0.0 -> 1.1.0, healthy, in service.
   app02: changed, 1.0.0 -> 1.1.0, code deployed and running, but
          /health reports 503 so it was NEVER returned to service.
   app03: NOT changed. Still 1.0.0. Still in service.
   Evidence: reports/deploy-app01.yml exists; there is no
   reports/deploy-app02.yml, because that task is after the health gate.

2. WHAT IS SERVING TRAFFIC RIGHT NOW?
   Two of three hosts: app01 on 1.1.0 and app03 on 1.0.0.
   Capacity is 67%. The service is up and is serving two versions.

3. IS app02 DANGEROUS WHERE IT IS?
   No. It is in MAINT on the proxy, so it receives no traffic. The
   drain step ran before the deploy and the enable step never ran.
   This is the designed outcome: a host that fails its gate stays out.

4. WHY DID IT FAIL?
   /health returns 503. The process is running and answering, so this
   is not a crash. The endpoint returns 503 when the database is
   unreachable OR when /etc/estate-app-fail exists. app01 and app03
   reach the same database, so the database is not the cause.
   Conclusion: the marker file. Verify with `stat` before acting.

5. ROLL FORWARD OR ROLL BACK?
   Roll forward. Justification: 1.1.0 is proven healthy on app01 under
   real traffic; the failure is host-specific and diagnosed; app03 is
   untouched and can be deployed after app02 is fixed. Rolling back
   app01 would mean two changes instead of one and would leave the
   original defect on app02 undiagnosed.
   The decision would be the opposite if app01 had ALSO failed: two of
   three failing is a property of the release, not of a host.

6. WHAT IS THE NEXT COMMAND, AND WHAT IS ITS BLAST RADIUS?
   Fix app02's fault, re-run the gate against app02 alone, and only
   then deploy app02 and app03 with --limit naming exactly those two.
   Blast radius: 2 hosts. app01 must not be in the limit; it is
   already correct and re-running it would drain a healthy host for
   no reason.

What could go wrong?

What happens if max_fail_percentage were left at its default?

The default is 100, meaning the play only aborts when every host in the batch fails. app02 would have failed and app03 would have been deployed anyway, in the same batch — so instead of one broken host and one untouched host, you would have one broken host, one host deployed without anybody noticing the batch had already gone wrong, and a rollout that reported failure only at the end.

What happens if the health gate came after the return-to-service?

app02 would be back in the proxy backend serving 503 to real users while the play was still deciding whether it was healthy. The order — drain, change, validate, then enable — is the whole safety property, and any reordering that returns a host to service before it is proven is a bug regardless of how green the recap looks.

What happens if you re-run the same command to “just try again”?

--limit appservers re-runs app01 too: it drains a healthy host, re-applies an identical role, and returns it. Idempotent, so nothing changes — but it takes a healthy host out of service during an incident for no reason, and it doubles the window in which capacity is reduced. Narrow the limit to the hosts that need work. This is why the limit file in Task 8 exists.

What happens if app02 had been UNREACHABLE rather than failing?

Different exit code — 4 rather than 2 — and a different recap column. unreachable means Ansible never ran a task there, so the host is in whatever state it was in before, including still drained if the drain succeeded and the connection died afterwards. A drained host that nobody re-enabled is invisible in the recap and reduces capacity silently until somebody reads the proxy stats. Check show stat after every aborted rollout for exactly this reason.

Under the hood

Three behaviours of serial that shape what you just saw, all verified against ansible-core 2.21.3 rather than inferred.

The play is re-entered per batch. serial does not batch tasks within one play run; it runs the whole play once per batch. That is why PLAY [...] appears three times for serial: 2 over five hosts, and why ansible_play_batch contains only the current batch’s hosts.

Handlers flush at the end of each batch, not at the end of the play. With serial: 2 over five hosts, a notified handler runs three times — once for each batch — which is exactly what you want for a rolling restart and exactly wrong for a “notify the ticketing system when the deployment finishes” handler.

run_once means once per batch. This one surprises people. A task with run_once: true under serial: 2 over five hosts executes three times, on the first host of each batch. A “post to chat that the deployment has started” task written with run_once posts three times; a “take a database snapshot before the rollout” task takes three snapshots.

The fix is not a cleverer run_once. It is a separate play, before the rolling play, with hosts: appservers[0] or hosts: localhost, which runs exactly once because it is a different play with no serial on it.

Task 8: Recover, deliberately

Only now, with the assessment written, fix it.

Read-only / Safecontroller— Confirms the diagnosis before acting on it.

$ ansible -i inventories/production/hosts.yml app02 -b \
-m stat -a 'path=/etc/estate-app-fail' | grep -E '"exists"'

Configuration changecontroller— Removes the injected fault on app02 only. The service does not need restarting; the marker is read per request.

$ ansible -i inventories/production/hosts.yml app02 -b \
-m file -a 'path=/etc/estate-app-fail state=absent'

Build the resume limit from the assessment, not from memory. A limit file records the decision alongside the command.

Read-only / Safecontroller— Writes the resume limit and shows exactly which hosts it selects, before running anything against them.

$ cd "$HOME/estate"

{
echo "app02"
echo "app03"
} > reports/resume.limit

ansible-playbook -i inventories/production/hosts.yml \
playbooks/deploy.yml --limit @reports/resume.limit --list-hosts

playbook: playbooks/deploy.yml

play #1 (appservers): Rolling application deployment	TAGS: []
  pattern: ['appservers']
  hosts (2):
    app02
    app03

Illustrative output

A limit file has no comment syntax

The obvious instinct is to put a # resume after app02 failure line at the top. Ansible does not treat # as a comment there — it parses the line as a host pattern, fails to match it, and emits

[WARNING]: Could not match supplied host pattern, ignoring: # resume

It is only a warning, so the run proceeds and the file works. But a warning you have taught yourself to ignore is a warning that will not save you the day the pattern that failed to match was a real hostname you typo’d. Keep limit files bare, and put the reasoning in the change record where it belongs.

Service impact possiblecontroller— Completes the rollout on app02 and app03 only. app01 is deliberately excluded and stays in service throughout.

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/deploy.yml --limit @reports/resume.limit \
| tee reports/resume-rollout.txt

Task 9: The audit record

The question an auditor, an incident review or your future self will ask is not “did it work”. It is “which hosts changed, when, by whom, from what to what”. Assemble the answer.

Read-only / Safecontroller— Assembles the audit record from the per-host artefacts and the run log.

$ cd "$HOME/estate"

{
echo "AUDIT RECORD — estate-app 1.0.0 -> 1.1.0"
echo "assembled: $(date -Is)"
echo
echo "== per-host records (present only for hosts returned to service) =="
for f in reports/deploy-*.yml; do echo "--- $f"; cat "$f"; done
echo
echo "== runs, from the controller log =="
grep -E 'PLAY RECAP|^(app|db|lb)[0-9]+ +:' reports/ansible.log | tail -40
echo
echo "== final observed state =="
for h in 192.0.2.21 192.0.2.22 192.0.2.23; do
  printf '%s ' "$h"
  curl -s --max-time 2 "http://$h:8080/health" | jq -c .
done
} > reports/audit-record-110.txt

wc -l reports/audit-record-110.txt

The controller log is evidence, and it is not tamper-proof

log_path in ansible.cfg writes every run to reports/ansible.log. That gives you a timeline, but it is a file on the controller, owned by the person who ran the play, and editable by them. It answers “what happened” for an honest operator; it does not answer “what happened” for an audit that has to assume the operator is the subject.

Real change attribution needs the run to happen somewhere the operator does not control: a CI pipeline that owns the vault password and writes to a log the operator cannot edit, or a callback plugin shipping results to an external store such as ARA. Lab 4 makes the same point from the other direction, when the controller is lost.

Know which one you have. “We have logs” and “we have evidence” are different claims.

Validation

Read-only / Safecontroller— Acceptance checks for this lab. Read-only.

$ cd "$HOME/estate"

# 1. Idempotency proof exists and is clean.
grep -E 'changed=[1-9]' reports/idempotency-run2.txt || echo 'IDEMPOTENT'

# 2. The canary was a separate run against one host.
grep -c 'app01' reports/canary-app01.txt
grep -E 'app0[23]' reports/canary-app01.txt || echo 'CANARY TOUCHED ONE HOST'

# 3. No 5xx reached the client during the clean rollout.
awk '$2 ~ /^5/ {n++} END {print (n ? n : 0), "5xx responses"}' \
reports/client-rollout.txt

# 4. The partial failure stopped the play and left app03 untouched.
grep -E 'NO MORE HOSTS LEFT' reports/partial-failure.txt
grep -E '^app03 +: .*changed=0' reports/partial-failure.txt

# 5. The assessment was written, and lists all three hosts.
grep -cE 'app0[123]' docs/incident-partial-rollout.md

# 6. Every host is on the new version and in service.
ansible-playbook -i inventories/production/hosts.yml \
playbooks/health.yml --limit estate

# 7. The rollback command has actually been run.
grep -E 'version_after: 1.0.0|app_version=1.0.0' reports/rollback-drill.txt \
| head -3

Every line must pass:

The second site.yml run reports changed=0 on all five hosts.
The canary run names app01 and does not touch app02 or app03.
Zero 5xx responses in the client log across the clean rollout — the drain worked.
partial-failure.txt contains NO MORE HOSTS LEFT and app03 with changed=0.
docs/incident-partial-rollout.md exists, was written before recovery, and answers all six questions.
health.yml passes and all three hosts report 1.1.0.
rollback-drill.txt shows the rollback to 1.0.0 actually executed.

Expected Outcome

estate/
├── docs/
│   ├── change-2026-08-12-app-110.md
│   ├── incident-partial-rollout.md
│   └── promotion-record-110.md
├── playbooks/deploy.yml
└── reports/
    ├── audit-record-110.txt
    ├── canary-app01.txt
    ├── client-rollout.txt
    ├── deploy-app0{1,2,3}.yml
    ├── idempotency-run{1,2}.txt
    ├── partial-failure.txt
    ├── resume.limit
    ├── resume-rollout.txt
    ├── rollback-drill.txt
    ├── rollout-remaining.txt
    └── staging-deploy-110.txt

Three application hosts on 1.1.0, all three backends UP, zero 5xx served during the clean rollout, one deliberate failure absorbed without an outage, and a written record that answers which hosts changed and to what.

Troubleshooting

The haproxy module reports “Could not connect to socket”. The task must run on the proxy host with become: true — the socket is mode 660 owned by the haproxy group. Confirm delegate_to names the proxy and that become: true is on the task, not only on the play.

Drain succeeds but the client still sees 503s. drain_settle_seconds is too short: requests already in flight when the drain landed were still being served when the restart happened. Increase it, and note that the right value is a function of your slowest request, not a constant.

The health gate passes but the version assertion fails. The template was rendered and the service was not restarted. Look for a handler that did not run — a task failing between the template and the end of the play leaves handlers unflushed unless force_handlers is set. This is precisely the failure the assertion exists to catch, and it is why the gate checks the version rather than only the status code.

ansible_limit is undefined and the guardrail fires when you did pass --limit. The variable is set from the command line only. A limit set through ANSIBLE_LIMIT in the environment also populates it; a hosts: pattern in the play does not.

The rollout is slow and one host takes 60 seconds to return. wait_for on the port succeeds as soon as something is listening, but the until loop on /health retries health_retries times at health_delay intervals — 12 × 5 = 60 seconds of tolerance. That is the budget you set; if the service genuinely needs it, the abort criterion of 120 seconds in the change record is what decides whether it is acceptable.

show stat reports a host in MAINT after a successful run. The enable task did not run, which means the play aborted after the drain. Re-enable explicitly rather than re-running the whole deploy: ansible -i ... loadbalancers -b -m shell -a 'echo "set server estate_backend/app02 state ready" | socat stdio /run/haproxy/admin.sock'.

Two hosts drain at once and capacity collapses. serial: [1, 2] on a three-host tier is doing what you told it to. If one host cannot carry the load, the shape is wrong for your traffic, not the tool.

Cleanup

Lab 4 continues from here and needs the estate running. Do not run Cleanup if you are continuing to lab 4.

Step 1. Remove any injected fault, unconditionally, on every host. A forgotten marker file makes a host fail its gate weeks later during an unrelated change, and nobody will connect the two:

Configuration changecontroller— Removes the failure marker from every application host.

$ cd "$HOME/estate"
ansible -i inventories/production/hosts.yml appservers -b \
-m file -a 'path=/etc/estate-app-fail state=absent'

ansible -i inventories/staging/hosts.yml appservers -b \
-m file -a 'path=/etc/estate-app-fail state=absent'

Step 2. Confirm no host is left drained. This is the silent one:

Read-only / Safecontroller— Lists every backend and its state. Anything not UP is a host still out of service.

$ ansible -i inventories/production/hosts.yml loadbalancers -b -m shell \
-a 'echo "show stat" | socat stdio /run/haproxy/admin.sock | cut -d, -f1,2,18 | grep estate_backend'

Any host reporting MAINT or DRAIN must be returned to service before you walk away:

Service impact possiblecontroller— Returns a named host to the proxy backend. Substitute the host that is drained.

$ # Substitute your own values before running:
HOST=app02
BACKEND=estate_backend

ansible -i inventories/production/hosts.yml loadbalancers -b -m shell \
-a "echo 'set server $BACKEND/$HOST state ready' | socat stdio /run/haproxy/admin.sock"

Step 3. Decide what version the estate should be left on, and say so. The honest default is to leave it on 1.1.0 — that is a normal, supported state and it is what a real deployment would leave behind. If you want 1.0.0, use the documented rollback rather than editing files by hand:

Service impact possiblecontroller— Optional. Returns all three hosts to 1.0.0 through the same drain and health gate.

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/deploy.yml -e app_version=1.0.0 --limit appservers

Step 4. Stop the traffic generator if it is still running, and keep the evidence:

Read-only / Safecontroller— Preserves the deliverables outside the working tree.

$ mkdir -p "$HOME/estate-deliverables/capstone-3"
cp -a "$HOME/estate/docs" "$HOME/estate/reports" \
    "$HOME/estate-deliverables/capstone-3/"

Step 5. Confirm the estate is healthy before you leave it:

Read-only / Safecontroller— Final health gate. Must pass.

$ ansible-playbook -i inventories/production/hosts.yml \
playbooks/health.yml --limit estate

What You Learned

The second run is the safety test. An estate that reports changes on every run cannot answer “has anything drifted”, and its phantom changes fire handlers that restart production.
max_fail_percentage aborts when the failure rate exceeds the value. One host of five failing is 20 per cent, which does not abort at 20. If you mean “stop on the first failure”, write 0.
run_once under serial runs once per batch, so a notification task fires three times over five hosts at serial: 2. Put genuinely once-per-run work in a separate play.
Handlers flush per batch, which is right for a rolling restart and wrong for an end-of-deployment notification.
Staging with one host per tier cannot prove batching, capacity or version skew. Writing down what it could not prove is what makes the canary non-optional.
Drain, change, validate, then enable. Any reordering that returns a host to service before it is proven is a bug however green the recap.
The absence of an audit record is evidence. Writing it after the return-to-service makes ls reports/ answer “which hosts finished”.
A limit file has no comments, and the warning it emits for a # line is one you must not learn to ignore.

Why this matters

The difference between an operator and somebody who runs playbooks shows up in the ninety seconds after a rollout fails halfway.

The instinct is to fix it: re-run the command, roll everything back, restart the failed service. Every one of those is a change made with an incomplete picture, and each one destroys evidence about what the previous change did. The discipline is to establish the state first — which hosts changed, what is serving, is the failed host dangerous where it is, roll forward or back and why — and to write it down, because writing it down is what forces the questions to be answered rather than felt.

The batching, the drain, the health gate and the audit record all exist to make that ninety seconds survivable. They are worth building before the incident, because during it there is no time to build anything.

Objectives

Prerequisites

Objective

Architecture

Requirements

Scenario

Tasks

Task 1: The second-run test

Task 2: The change record and the rollback plan, written first

Task 3: The deploy playbook

Task 4: Staging first, and what staging cannot prove

Task 5: Measure the interruption, then canary

Task 6: Roll the rest

Task 7: The partial fleet failure

Task 8: Recover, deliberately

Task 9: The audit record

Validation

Expected Outcome

Troubleshooting

Cleanup

What You Learned

Deliverables

Verification status