The change rollback plan — what we do when the change goes wrong

Every change goes well until it does not. The rollback plan is the part of the change record that decides whether the wrong-time-to-be-wrong is a 5-minute incident or a 5-hour incident. A pull request without a rollback plan is a pull request whose on-call will reverse in panic, with no documented procedure and no audit trail. A pull request with a rollback plan is a pull request whose on-call executes a rehearsed procedure and produces a documented recovery.

Three rollback patterns

The three patterns cover most production changes. Each has a different blast radius, a different speed, and a different audit property.

flowchart LR
    C["Change goes wrong"] --> P1["Revert PR: durable, auditable"]
    C --> P2["Blue/green: instant, infrastructure"]
    C --> P3["Feature flag: instant, code path"]
    P1 --> AUDIT["Audit trail intact"]
    P2 --> AUDIT
    P3 --> AUDIT

• Revert PR. A new pull request that undoes the change. The revert goes through the same review and CI as the original change; the merge produces an audit-grade record. The revert takes minutes, not seconds. Best for changes that can wait.
• Blue/green. Two production environments; the change is deployed to one and traffic is switched at the load balancer. The rollback is a switch back. Best for stateless services behind a load balancer.
• Feature flag. The change is deployed but disabled; a flag controls rollout. The rollback is flipping the flag. Best for application changes that are isolated to a code path.

The three patterns are not mutually exclusive. A change that flips a feature flag behind a blue/green deploy is a change that can be rolled back three ways. The plan in the PR says which is the primary path and which is the fallback.

Writing the rollback plan in the PR

The rollback plan is the sixth section of the change record. The author writes the plan at submission time; the approver verifies the plan at review time.

TITLE="Bump checkout HPA max replicas to 50"
BODY=$(cat <<'EOF'
## What
Increases max replicas from 20 to 50.

## Why
Peak traffic on 2026-08-21; current max saturates.

## When
Window: 2026-08-22 02:00-04:00 UTC.

## Who
Author: alice; Approver: bob.

## Where
Cluster: payments-prod; namespace: checkout.

## How
Apply HPA change via Argo CD sync.

## Rollback
Primary: revert PR. `gh pr revert` against this PR;
Argo CD will sync the previous HPA config. Estimated
time to restore: 5 minutes.

Fallback: `kubectl scale hpa checkout -n checkout
--maxreplicas=20` (in-place only; commit the revert
PR immediately after).

Trigger: 5xx rate > 0.5% sustained 2 minutes, or p99
latency > 800ms sustained 5 minutes.

Refs: #1234
EOF
)
gh pr create \
  --title "$TITLE" \
  --body "$BODY" \
  --base main \
  --head feat/checkout-hpa \
  --reviewer bob \
  --label "change,production,rollback-defined"

The plan has three parts: the primary rollback path (a revert PR), the fallback (an in-place scale, with a follow-up revert PR), and the trigger (the metric threshold at which the on-call executes). Each part is required; a plan without a trigger is a plan the on-call will execute on instinct.

Pre-mortem as the discipline

The pre-mortem is the discipline that surfaces the rollback plan before the change is applied. The author imagines the change has gone wrong, walks through the recovery, and writes the steps in the PR.

flowchart LR
    A["Author imagines failure"] --> B["Walks the recovery"]
    B --> C["Writes steps in PR"]
    C --> D["Approver verifies at review"]
    D --> E["On-call executes at 02:00"]

A pre-mortem surfaces the rollback steps that the on-call will need but the author has not thought of: the role assumption, the cluster context, the kubectl namespace, the verification command. A pre-mortem without a written plan is brainstorming; a pre-mortem with a written plan is the change record.

Production discipline

1. The rollback plan is a required section of the change record. A pull request without a rollback plan does not merge.
2. The primary rollback is a revert PR. The fallback can be faster, but the revert PR must follow.
3. The trigger is a metric, not a feeling. A rollback threshold in the PR body is executable; a “use judgement” instruction is not.

Cross-course references

• This course, Part LXIV (Auditability) - the audit chain the revert PR preserves.
• This course, Part LI (GitOps) - the GitOps reconciliation that requires the repository to stay in sync with the cluster.
• Kubernetes for Production Sysadmins - Parts XXVII-XXIX (DeploymentLifecycle) cover blue/green and rollback patterns for Kubernetes deployments.

Quiz