Skip to main content
RunBook Academy

← All runbooks in Git, CI/CD & GitOps

critical riskservice affecting~45 min

Runbook: Revert a Production Change

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Confirm the change ticket exists with the SHA(s) to revert: git log --oneline -n 50 origin/main and identify the deploy SHA that produced the regression
  • · Confirm the production cluster is the source of truth for "what is running": kubectl get deploy,sts,ds -A -o jsonpath="{.items[*].metadata.annotations.deploy\\.kubernetes\\.io\\/sha}" | tr "," "\\n" | sort -u (or the equivalent for Terraform/Ansible)
  • · Capture a state snapshot for the affected resources: terraform state pull > /tmp/state-before-revert.json and date — the snapshot is diagnostic and disaster-recovery evidence only; it is never pushed back to the backend. State restore is reserved for proven state corruption (see git-cicd-gitops-rb-26-recover-infra-pipeline)
  • · Confirm CI is green on main: gh run list --workflow deploy-prod --branch main --limit 5 --json conclusion,headSha,status or equivalent (a failed CI on main means the revert PR cannot be merged cleanly)
  • · Confirm the change is revertible: a regular forward commit is always revertible; a merge commit needs git revert -m 1; a squash-merged commit reverts as a single commit. Identify the kind: git log --format="%H %P %s" -n 1 <SHA> — one parent = forward commit, two = merge commit
  • · Capture the live state of the affected service: curl -fsS https://<service>/version and kubectl get -o yaml deploy/<svc> saved to /tmp/<svc>-pre-revert.yaml

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Create a revert branch from the deployed SHA: git fetch origin main && git checkout -b revert/"$CHANGE_TICKET" origin/main
  2. 2If the offending change was a forward commit: git revert <SHA> --no-edit. If it was a merge commit: git revert -m 1 <SHA> --no-edit. If it was a squash-merge: git revert <SHA> --no-edit (the squash makes it a single commit on main)
  3. 3Resolve any conflicts the revert surfaces: conflicts on a revert mean the post-change commits touched the same lines, and the resolution is the union of both intentions, not "pick the newer side"
  4. 4Validate the resulting tree with the IaC tool: terraform plan -lock=false (expect the inverse of the offending plan), ansible-playbook --check (expect idempotent), or kubectl diff -k overlays/prod/ (expect the inverse of the offending kustomize output)
  5. 5Push the branch: git push -u origin revert/"$CHANGE_TICKET"
  6. 6Open the PR: gh pr create --base main --head revert/"$CHANGE_TICKET" --title "revert: <SHA> -- <reason>" --body "Reverts <SHA> which caused <symptom>. See <incident-link> for the timeline." (or the equivalent for GitLab/Bitbucket)
  7. 7Wait for CI to pass on the PR — the deploy pipeline must run against a staging cluster first and the plan must match the inverse of the offending plan
  8. 8Get the required approval per the change authority for the revert. Document the approver in the PR conversation. Do not self-approve
  9. 9Merge the PR with a non-rewriting merge (Merge pull request or Squash and merge are both acceptable; Rebase and merge rewrites the SHA and breaks the audit trail for what was reverted)
  10. 10Trigger the production deploy from the merged SHA: gh workflow run deploy-prod.yml --ref main (or argocd app sync <app> for Argo CD, flux reconcile kustomization <name> for Flux). Do not deploy from the revert branch directly — main is the source of truth
  11. 11Watch the deploy converge: kubectl rollout status deploy/<svc> -n <ns> --timeout=10m or argocd app wait <app> --health or flux get kustomization <name>
  12. 12Verify the live system matches the pre-change state: kubectl get -o yaml deploy/<svc> matches /tmp/<svc>-pre-revert.yaml with only the reverted lines changed, curl -fsS https://<service>/version returns the pre-change version string, and the production dashboard (error rate, p95 latency, request volume) returns to the pre-change baseline within SLO
  13. 13Capture the timeline in the change ticket: deploy SHA → incident time → revert SHA → re-deploy SHA. Attach CI run URLs, dashboard screenshots, and the reviewer who approved the revert

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓The revert commit is in git log origin/main between the offending SHA and HEAD
  • ✓git diff <OFFENDING_SHA> HEAD -- <file> is empty for every file changed by the offending SHA (the revert fully inverts it)
  • ✓For Terraform: terraform plan -lock=false reports No changes. Your infrastructure matches the configuration." for the affected resources
  • ✓For Kubernetes: kubectl get deploy/<svc> -n <ns> reports Ready replicas equal to desired, and kubectl rollout history deploy/<svc> -n <ns> shows the revert as a new revision
  • ✓For Ansible: the affected hosts report changed=0 on a re-run: ansible-playbook -i inventories/prod/hosts.yml site.yml --check reports the expected no-change state
  • ✓Production dashboard p95 latency and error rate match the pre-change baseline within SLO for at least 10 minutes after the revert deploy
  • ✓The change ticket has the full SHA trail: offending → revert → redeploy, with CI URLs and approval

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the revert fails to converge: pause the deploy (kubectl rollout pause deploy/<svc> -n <ns>) and inspect the new ReplicaSet before deciding to roll forward or back
  • ↶If the revert PR has conflicts that cannot be resolved safely, abandon it and write a forward-fix PR instead: do not force-resolve a revert conflict, the conflict means the post-change work depends on the change you are reverting
  • ↶If the revert deploys but production does not recover: the regression is not the original change — escalate to the service owner with the revert evidence (CI green, deploy SHA matches, but metrics do not recover) and start a parallel investigation
  • ↶If the revert was merged with Rebase and merge, the SHA differs from git revert's expected output: the audit trail is still intact because the PR description names the offending SHA — document the rebased SHA in the change ticket
  • ↶If the live state was modified out-of-band between the change and the revert (drift): the revert will not bring the system back to the pre-change state — capture the drift in the change ticket and roll forward manually instead

6 · Escalation

When the runbook isn't enough, contact:

  • · The offending change introduced a new resource that downstream services now depend on: a literal revert will break downstream. Open a follow-up PR that removes the resource gracefully (drain, redirect, delete) before reverting
  • · The offending change is a Git history rewrite (force-push) that the revert cannot reach: see git-cicd-gitops-rb-06-respond-to-force-push-incident first, then write the revert against the reconstructed SHA
  • · The revert needs to deploy a database migration backwards: a code revert is not enough — write the inverse migration and run it with the same deploy pipeline. Do not run it out-of-band
  • · Multiple reverts in a short window: escalate to platform ownership, the system is in a flapping state and the next change is not safe

Reverting a production change is a code change in reverse. The runbook is not “roll back the deploy” — that is a deployment operation. The runbook is “write the inverse commit, get it through the same change authority, deploy through the same pipeline, validate against the same SLOs”. The deploy pipeline is the safety net; bypassing it to “just make prod work again” is how the next incident inherits the same uncertainty.

The decision tree is short: is the offending change a forward commit, a merge commit, or a squash-merge? A forward commit reverts with git revert <SHA>. A merge commit reverts with git revert -m 1 <SHA> (specifying which parent is the mainline). A squash-merge reverts with git revert <SHA> because squash collapses the PR into one commit.

1. Identify the offending change and the live state

Read-only / Safe
$ CHANGE_TICKET="INC-2026-0842"
OFFENDING_SHA="a1b2c3d4e5f6"
git fetch origin main
git log --oneline origin/main -n 20
echo '--- offending commit type ---'
PARENTS=$(git log -1 --format='%P' "$OFFENDING_SHA")
echo "parents: $PARENTS"
PARENT_COUNT=$(echo "$PARENTS" | wc -w)
echo "parent count: $PARENT_COUNT (1=forward, 2=merge, 3+=octopus)"
echo '--- pre-change version on the live system ---'
curl -fsS https://api.example.com/version
kubectl get deploy api -n prod -o jsonpath='{.spec.template.spec.containers[0].image}' > /tmp/pre-revert-image.txt
cat /tmp/pre-revert-image.txt

Save the live state. The terraform state pull snapshot from the pre-checks is diagnostic and forensic material plus a disaster-recovery backup — it is never pushed back to the backend. If the revert misbehaves, the recovery is another forward change (revert the revert, or a forward fix) planned against the current state. A state restore (terraform state push) is reserved for proven state corruption or loss — see git-cicd-gitops-rb-26-recover-infra-pipeline for that procedure.

2. Revert the offending commit

Read-only / Safe
$ git checkout -b "revert/$CHANGE_TICKET" origin/main
if [ "$PARENT_COUNT" -eq 1 ]; then
git revert "$OFFENDING_SHA" --no-edit
elif [ "$PARENT_COUNT" -eq 2 ]; then
git revert -m 1 "$OFFENDING_SHA" --no-edit
else
echo "octopus merge - pick the mainline parent and use -m"
git revert -m 1 "$OFFENDING_SHA" --no-edit
fi
echo '--- the revert commit ---'
git log -2 --format='%H %P%n%s%n%n%b' | head -40

For merge commits, -m 1 says “mainline is the first parent” (conventionally origin/main). If the offending PR was merged from a feature branch and you want the revert to look at the feature branch’s parent, you do not have a literal revert available — write a forward fix instead.

3. Validate the resulting tree

Read-only / Safe
$ echo '--- for Terraform-affected services ---'
terraform plan -lock=false -out=/tmp/revert-plan.tfplan
terraform show -no-color /tmp/revert-plan.tfplan | head -80

echo '--- for Kustomize/Helm services ---'
kubectl kustomize overlays/prod/ > /tmp/revert-manifests.yaml
kubectl diff -f /tmp/revert-manifests.yaml | head -80

echo '--- for Ansible playbooks ---'
ansible-playbook --syntax-check -i inventories/prod/hosts.yml site.yml
ansible-playbook --check -i inventories/prod/hosts.yml --diff site.yml | head -80

echo '--- does the revert fully invert the original change? ---'
git diff "$OFFENDING_SHA" HEAD --stat | tail -40

If terraform plan shows resource drift the revert did not intend, the offending change interacted with state that has since moved. The revert is still correct against the committed code; the live system will need a manual reconcile. Capture this in the change ticket.

4. Push the branch and open the PR

Read-only / Safe
$ git push -u origin "revert/$CHANGE_TICKET"
gh pr create \
--base main \
--head "revert/$CHANGE_TICKET" \
--title "revert: $OFFENDING_SHA -- <one-line reason>" \
--body "Reverts $OFFENDING_SHA which caused REPLACE_WITH_SYMPTOM. See REPLACE_WITH_INCIDENT_LINK for the timeline. Approval: REPLACE_WITH_APPROVER." \
--label revert --label production
gh pr checks "$(gh pr list --head "revert/$CHANGE_TICKET" --json number -q '.[0].number')" --watch

5. Merge and trigger the production deploy

Read-only / Safe
$ gh pr merge --auto --squash "revert/$CHANGE_TICKET"
# Or for non-squash: gh pr merge REPLACE_WITH_NUMBER --merge
# Avoid --rebase: rewrites the SHA and breaks the audit trail.

# Deploy from main, not from the revert branch
gh workflow run deploy-prod.yml --ref main

# Watch the deploy
kubectl rollout status deploy/api -n prod --timeout=10m
argocd app wait api --health --timeout 600
# or: flux get kustomization api --watch

The deploy must run from main (or the equivalent production branch). Deploying from the revert branch skips the audit trail that ties the deployed SHA to the PR that introduced it.

6. Validate under real traffic

Read-only / Safe
$ curl -fsS https://api.example.com/version
kubectl get deploy api -n prod -o jsonpath='{.spec.template.spec.containers[0].image}' > /tmp/post-revert-image.txt
diff /tmp/pre-revert-image.txt /tmp/post-revert-image.txt && echo 'image reverted' || echo 'image changed - investigate'
kubectl get pods -n prod -l app=api -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[0].ready,RESTARTS:.status.containerStatuses[0].restartCount
echo '--- production dashboard ---'
# Open the dashboard URL for the service
echo 'https://grafana.example.com/d/api-prod/api-overview'

The dashboard check is the validation, not the deploy output. A green deploy with red metrics means the service is running but the regression was not in the change you reverted.

Verification

git log origin/main shows the revert SHA between the offending SHA and HEAD. git diff &lt;OFFENDING_SHA&gt; HEAD --stat is empty for every file the offending change touched. terraform plan (or kubectl rollout history, or the Ansible --check output) reports the inverse of the original plan. Production dashboards match the pre-change baseline within SLO for at least 10 minutes. The change ticket has the full SHA trail, the CI run URLs, and the reviewer who approved the revert.

Rollback

If the revert itself fails to deploy, pause the rollout and inspect the new ReplicaSet before deciding direction — the revert may be correct against the code but wrong against the live state. If the revert deploys but the regression does not recover, the regression is not in the change you reverted; escalate to the service owner with the revert evidence and start a parallel investigation. If the live state had drifted between the change and the revert (someone ran a hotfix out-of-band), the revert against the committed code will not restore the system to its actual pre-change state — capture the drift and roll forward manually, do not chain reverts on top of drift.

References

  1. git-revert(1)
  2. GitHub Docs — Reverting a pull request
  3. Argo CD — Application sync
  4. Flux CD — Reconcile
  5. Terraform — State management and rollbacks