Runbook: Recover from a partial fleet failure

A rollout that stops halfway leaves a fleet in several states at once, and the instinct - re-run it and see how far it gets - is the one action that makes the situation harder to reason about. The fleet is not broken; it is unknown, and the work is turning unknown into known before doing anything else.

The single most important fact in this runbook: hosts that were never attempted do not appear in the recap at all. Not as skipped, not as unreachable. They are simply absent, and a recovery driven by reading the recap will not know they exist.

When to use this runbook

• A rolling change stopped on max_fail_percentage or any_errors_fatal.
• A run was interrupted - cancelled, controller lost, session dropped.
• A batch failed and the operator stopped rather than continuing.
• Any run where the question “what state is the fleet in?” does not have an immediate answer.

Blast radius

The whole scope of the original run, in an unknown mixture of states. Everything in Steps 1 to 5 is read-only; the decision in Step 7 is where you start changing things again.

Step 1: Stop everything

# Is the run still going?
pgrep -af 'ansible-playbook' || echo 'no ansible-playbook running'

# Scheduled invocations
systemctl list-timers --all --no-pager | grep -i ansible
crontab -l 2>/dev/null | grep -i ansible
sudo crontab -l -u ansible 2>/dev/null

# CI pipelines
echo 'Check the pipeline UI and pause the job that runs this playbook'

A recovery conducted while a scheduled converge is running against the same hosts is not a recovery. Confirm quiet before assessing, and tell the team the rollout is stopped so nobody helpfully re-runs it.

Step 2: Reconstruct the intended scope

# The same command, with --list-hosts, gives the original scope
ansible-playbook -i inventories/production deploy.yml \
--limit web -e app_version=2.4.0 --list-hosts \
| grep -oE '[a-z0-9.-]+\.example\.com' | sort > intended.txt
wc -l intended.txt

This is the denominator. Every host in this file must end up in exactly one of the five states in Step 3, and the counts must sum to this total.

Step 3: Find the invisible hosts

LOG=deploy-2026-08-11.log

# Hosts that appear in the recap - i.e. hosts the run actually attempted
awk '/PLAY RECAP/,0' "$LOG" | grep -oE '^[a-z0-9.-]+\.example\.com' \
| sort -u > attempted.txt

# Hosts in scope that the run never reached
comm -23 intended.txt attempted.txt | tee never-attempted.txt
wc -l never-attempted.txt

Step 4: Classify every host by measurement

Five states. Every host in intended.txt goes into exactly one.

# What version is each host actually running?
ansible web -m uri -o \
-a 'url=http://{{ ansible_host }}:8080/version return_content=true' \
| tee measured-versions.txt

# What package versions are installed?
ansible web -m package_facts -o >/dev/null
ansible-playbook -i inventories/production report-versions.yml | tee measured-packages.txt

# What is on disk?
ansible web -b -m stat -a 'path=/etc/app/app.conf' -o | tee measured-config.txt

Measurement, not the log. The log records what Ansible attempted and what it believed; the host records what is true. Those diverge in exactly the situation you are in - a run that stopped partway, possibly with handlers pending, possibly with a task that succeeded and a task that did not.

Step 5: Assess service health separately

Change state and service health are different questions and both need answering.

# A real request per host, not a port check
for h in $(cat intended.txt); do
code=$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 "http://$h:8080/healthz" || echo 000)
printf '%-30s %s\n' "$h" "$code"
done | tee measured-health.txt

# Anything left drained from the load balancer
curl -sS https://lb.example.com/api/pool/web | python3 -m json.tool | grep -c drain

# Anything in a failed state
ansible web -b -m command -a 'systemctl --failed --no-pager --plain' -o

A host in State 4 can still be unhealthy, and a host in State 1 is usually perfectly healthy on the old version. Draining is the item most often overlooked: a rollout that stopped mid-batch leaves hosts out of rotation, and those are capacity you are not receiving.

Step 6: Communicate before deciding

Produce a short, numeric assessment and send it before you act:

Rollout of app 2.4.0 to group web stopped at 14:22.

Scope           40 hosts
Fully applied   12  (on 2.4.0, healthy)
Partly applied   1  (web13 - package updated, service not restarted)
Attempted, no change  2  (web14, web15 - failed the pre-check)
Not attempted   24  (still on 2.3.7, healthy, unaffected)
Unreachable      1  (web27 - state unknown)

Drained and not returned: 3 hosts (web13, web14, web15)
Service currently healthy on 37 of 40; capacity at 92%.

Recommendation: hold. Root cause of the web14/web15 failure is not yet
known and applies to the 24 untouched hosts equally.

The point is not the format. It is that “we are 30% through and something failed” is not a basis for a decision and this is. The recipient can act on numbers; they cannot act on an impression.

Step 7: Forward, backward, or hold

One decision, for the whole affected set, made with the service owner.

Step 8: Execute as a rolling change

Whichever direction, it is a new rolling change against a measured host list - with batching, health checks and a stop rule of its own.

# Forward: the hosts not yet on the new version
grep -v '2.4.0' measured-versions.txt \
| grep -oE '[a-z0-9.-]+\.example\.com' | sort -u > to-forward.txt

# Backward: the hosts that DID get the new version
grep '2.4.0' measured-versions.txt \
| grep -oE '[a-z0-9.-]+\.example\.com' | sort -u > to-rollback.txt

wc -l to-forward.txt to-rollback.txt

LIMIT=$(paste -sd, to-rollback.txt)

ansible-playbook -i inventories/production deploy.yml \
--limit "$LIMIT" -e app_version=2.3.7 --list-hosts

ansible-playbook -i inventories/production deploy.yml \
--limit "$LIMIT" -e app_version=2.3.7 --diff | tee "recovery.log"

Note the target list is built from measurement, not from the original log. The log says what was attempted; the measurement says what is true, and after a partial failure those are different documents.

Restore capacity first. Before investigating anything, return every healthy drained host to rotation - the service does not benefit from your investigation and it does benefit from its capacity back.

Step 9: Verify convergence

ansible web -m uri -o \
-a 'url=http://{{ ansible_host }}:8080/version return_content=true' \
| grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | sort | uniq -c

# Every host in scope accounted for
ansible web --list-hosts | grep -c 'example.com'

# A converge run is now clean
ansible-playbook -i inventories/production deploy.yml --limit web --diff \
| grep -E 'changed=[1-9]'

# Nothing left drained
curl -sS https://lb.example.com/api/pool/web | grep -c drain

The uniq -c should show a single version with a count equal to the host count. Two lines means the fleet is still split, and that is the condition you set out to remove.

Step 10: Write the timeline

• When the rollout started, and when it stopped.
• The host counts in each of the five states at assessment time.
• The decision, who made it, and on what evidence.
• When capacity was restored.
• How long the fleet was split, and across which versions.

The duration of the split is the number that gets asked for later, when someone is investigating an intermittent fault that occurred during it.

Common patterns

Escalation

Escalate when:

• The forward/backward choice needs making. That is the service owner’s.
• Any host reached an irreversible step and the change is bad.
• The state of any host cannot be determined.
• The recovery attempt fails the same way on its first host.
• The failure was in the rollout mechanism rather than the change.