Partial deployment and resumability — when a deploy gets four of five changes done

A deploy fails after applying four of five changes. The first four are in place. The fifth is missing. The pipeline is red. The engineer presses retry. The pipeline runs from the top. The first four changes apply again. Some succeed; some fail because they were already applied. The deploy is now in a state that no single command can describe - four changes applied once, one change applied zero times, and several changes in a duplicate-key error. The pipeline is not recoverable by retry; it is recoverable only by knowing exactly which sub-changes are complete.

The two failure modes of an interrupted deploy

An interrupted deploy produces one of two states:

• Resumable. A record exists of which sub-changes completed before the failure. The retry continues from the checkpoint; it does not reapply the completed changes.
• Non-resumable. No record exists, or the operation does not tolerate a partial-state retry. The retry would produce a worse state than the failure.

flowchart LR
    A["Deploy 5 sub-changes"] --> B["Failure after sub-change 4"]
    B --> C{"Is the operation resumable?"}
    C -- "Yes, with checkpoint" --> D["Retry from sub-change 5"]
    C -- "No" --> E["Complete or revert"]
    D --> F["All 5 sub-changes complete"]
    E --> G["Revert the 4 completed sub-changes"]
    E --> H["Then re-deploy all 5"]

Most infrastructure operations are not resumable by default. Terraform’s resumability lives in the state file: after a failed apply, a fresh terraform plan against the current state, reviewed and applied, picks up only what is missing (-target is documented only as an exceptional escape hatch for error recovery, not the routine partial-recovery path). Ansible’s recovery is a re-run of the idempotent playbook, or --limit with an explicit list of the failed hosts, but the operator must read the run output to know which hosts failed. Helm’s --wait flag only waits for resource readiness - it does not resume a failed upgrade; the recovery is helm rollback or a corrected helm upgrade, and the operator must know what helm status reports. Kubernetes’ kubectl apply is the most unfriendly: there is no built-in checkpoint, and a partial apply is a partial apply the operator must reconcile manually.

Tracking completed sub-changes

The mechanical fix for resumability is a checkpoint file that the deploy writes after each sub-change:

#!/usr/bin/env bash
set -eu
CHECKPOINT="${CHECKPOINT:-/var/run/deploy.state}"

for sub_change in 1 2 3 4 5; do
    if grep -q "^done:$sub_change$" "$CHECKPOINT" 2>/dev/null; then
        echo "skip $sub_change (already complete)"
        continue
    fi
    ./apply-sub-change-$sub_change.sh
    echo "done:$sub_change" >> "$CHECKPOINT"
done

The first run applies all five sub-changes because none are in the checkpoint. The retry reads the checkpoint, skips the completed four, and applies only the missing fifth. The mechanism is independent of the operation; applying the same discipline to Terraform, Ansible, and Helm is a matter of writing the checkpoint after each sub-change completes.

The discipline has a cost: the checkpoint itself is a piece of state the operator must clean up. A retry that succeeds leaves the checkpoint on disk; a deploy that fails-and-reverts must also delete the checkpoint so the next deploy starts from scratch:

rm -f "$CHECKPOINT"

The “complete or revert” alternative

A deploy that cannot be made resumable falls back on “complete or revert” - the discipline that says the deploy either finishes or rolls back to a known state, never leaving a partial state behind. The two paths:

deploy-or-revert() {
    if ./deploy.sh; then
        echo "deploy complete"
        return 0
    fi
    echo "deploy failed, reverting"
    ./revert.sh
    echo "revert complete"
    return 1
}

The function makes a single decision: either the deploy finishes, or the revert runs. There is no third outcome in which the deploy partially succeeds and the production system carries a partial state into the next retries. The discipline is in the function’s shape: it returns success or failure, and the pipeline does not attempt a retry of the same function on failure because the revert has already cleaned up.

Resumability in the common deploy tools

The deploy tools in this course each have their own partial-deployment failure mode:

• Terraform. Each resource in the plan is one sub-change. A plan-then-apply that fails mid-apply leaves some resources created and some not. The recovery is terraform plan (read-only, safe to repeat) to see the current state, then terraform apply (will skip resources already in state, apply only the missing ones). The state file is the checkpoint.
• Ansible. Each play is one sub-change. A playbook that fails midway leaves some hosts configured and some not. The recovery is to re-run the idempotent playbook - the configured hosts converge with no changes - or --limit with an explicit list of the failed hosts. Retry files (--limit @playbook.retry) exist only if retry_files_enabled is explicitly turned on; since Ansible 2.8 it defaults to off.
• Helm. Each manifest in the chart is one sub-change. An upgrade that fails midway leaves some manifests applied and some not. The recovery is helm rollback to the previous revision, or helm upgrade again to re-apply the missing pieces. Helm tracks state in Kubernetes secrets, so the upgrade of a partially- applied chart is more idempotent than a non-Helm equivalent.
• kubectl. Each manifest is one sub-change. A kubectl apply -f manifest.yaml that fails midway leaves some resources created and some not. The recovery is kubectl diff -f manifest.yaml (safe to repeat) to see the current state, then kubectl apply again to apply the missing pieces.

Each tool has its own partial-failure mode, and each tool has its own recovery path. The discipline is to learn the recovery path for each tool the pipeline uses; the resume-from-top retry is rarely correct.

Production discipline

1. Know which deploy tools are resumable and which are not. The runbook lists both cases.
2. For resumable tools, the deploy-and-retry is the correct path. Terraform, Ansible, Helm, and kubectl each have their own partial-recovery commands.
3. For non-resumable tools, complete-or-revert is the discipline. The retry from the top is the option the runbook removes.
4. Track completed sub-changes explicitly when the tool does not provide its own checkpoint.
5. Treat a partial deploy as a state the next operator will see. Document what is in place; document what is missing; document the path to the next good state.

Cross-course references

• This course, Part LXI-05 (Idempotency) covers the property a resumable operation must have.
• This course, Part LIX (Rollback) covers the rollback paths for partial deployments.
• Terraform for Production Sysadmins - Part IX (State) covers the state-file-as-checkpoint pattern.

Quiz