Skip to main content
RunBook Academy

← All runbooks in Git, CI/CD & GitOps

high riskservice affecting~60 min

Runbook: Roll Back a Deployment (Container, Kubernetes, Terraform)

1 · Prerequisites

Confirm every item is in place before any state change.

  • git-cicd-gitops-rb-05-revert-production-change
  • git-cicd-gitops-rb-15-validate-production-artifact
  • kubectl configured against the affected cluster context (kubectl config current-context)
  • Read access to the artifact registry and the GitOps repo for the previous known-good revision
  • For Terraform: terraform CLI 1.5+ and access to the state backend (S3/GCS/Azure Blob/Consul)

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Identify the rollback scope: container image rollback (registry tag), Kubernetes workload rollback (Deployment/StatefulSet), or Terraform rollback (configuration revert). The scope determines the procedure; do not mix them
  • · Capture the current state of the deployment BEFORE rolling back: for K8s, kubectl get deploy,rs,pods -n <ns> -o yaml > /tmp/pre-rollback.yaml; for Terraform, terraform show -json > /tmp/pre-rollback.tf.json. The capture proves what was running and enables forensic comparison
  • · Identify the last known-good revision. For containers: the previous image digest in the registry (crane ls <registry>/<image> | sort -V or registry UI). For K8s: the previous ReplicaSet revision (kubectl rollout history deploy/<name> -n <ns>). For Terraform: the last known-good commit hash on the live branch of the infrastructure repo (git log --oneline)
  • · Confirm the rollback target revision is still available. For containers: the previous image tag/digest must still exist in the registry. For K8s: the previous ReplicaSet must still be present (it is preserved by default for 10 revisions). For Terraform: the known-good commit must still be reachable in the infrastructure repo history
  • · Notify stakeholders via the on-call channel: "Initiating rollback of <service> from <current> to <target>. ETA: <minutes>. Bridge: <url>". A rollback is a change event; silence is a coordination failure
  • · For Terraform rollbacks: snapshot the current state file before any change: terraform state pull > /tmp/state-$(date -u +%s).tfstate.backup. The snapshot is a disaster-recovery backup in case the state file itself is later corrupted or lost; it is not part of the rollback procedure

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1STEP 1 - Container rollback (no orchestrator). Pull the previous known-good image tag, stop the running container, and start the new one. docker ps -q --filter ancestor=<registry>/<image>:<current> | xargs -r docker stop && docker run -d --name <service> --restart unless-stopped <registry>/<image>:<previous-known-good>. Verify with docker ps --filter name=<service> --format "{.Image} {.Status}"
  2. 2STEP 2 - Kubernetes rollback (kubectl imperative). kubectl rollout undo deploy/<name> -n <ns> --to-revision=<N> (omit --to-revision to roll back by one). Kubernetes scales the previous ReplicaSet up and the current one down. Confirm with kubectl rollout status deploy/<name> -n <ns> --timeout=300s
  3. 3STEP 3 - Kubernetes rollback (GitOps / Argo CD). Do NOT run kubectl rollout undo against a GitOps-managed cluster — the controller will re-sync and revert the rollback. Instead, revert the Git commit that introduced the bad change, push to the GitOps repo, and let Argo CD reconcile. git revert <bad-sha> → push → argocd app sync <app> --revision <good-sha> for an immediate sync. Argo CD-managed clusters: Git is the source of truth, not the cluster
  4. 4STEP 4 - Kubernetes rollback (Helm). helm rollback <release> <revision> -n <ns> --wait --timeout 300s. Confirm with helm history <release> -n <ns> and kubectl get deploy -n <ns>
  5. 5STEP 5 - Kubernetes StatefulSet / DaemonSet rollback. StatefulSets and DaemonSets support kubectl rollout undo just like Deployments: kubectl rollout undo sts/<name> -n <ns> --to-revision=<N> (same for ds/<name>). Alternatively, patch the container image to the previous tag (kubectl set image sts/<name> <container>=<registry>/<image>:<previous> -n <ns>). Note that kubectl rollout restart re-rolls the CURRENT spec and never selects a prior revision — it is not a rollback tool. Confirm with kubectl rollout status sts/<name> -n <ns>
  6. 6STEP 6 - Terraform rollback (configuration revert). Revert the configuration to the last known-good intent: git revert <bad-commit> on the infrastructure repo, merged through the normal review pipeline (or break-glass approval during an incident). Then run terraform plan against the CURRENT protected state, review the FULL plan — it must show only the intended reversal — and terraform apply. Restoring a state backup is NOT a rollback: state restore is a separate disaster-recovery procedure for proven state corruption or loss (see git-cicd-gitops-rb-26-recover-infra-pipeline) with locking, lineage/serial verification, backups, and post-restore reconciliation
  7. 7STEP 7 - Terraform rollback (forward fix). For a small, well-understood change, an alternative to the revert is a forward fix: edit the module/HCL to correct the defect, commit, and run the pipeline. Both paths are auditable in Git; prefer the plain git revert (STEP 6) when returning to the exact prior intent is the safest change, and a forward fix when the prior intent also needs correction
  8. 8STEP 8 - Verify the rollback took effect. For K8s: kubectl get deploy,rs,pods -n <ns> -L app.kubernetes.io/version,image — the version/image labels must show the previous revision. For Terraform: after the rollback apply, terraform plan should report "No changes". For containers: docker ps --filter name=<service> shows the previous image tag
  9. 9STEP 9 - Verify the application is healthy. Hit the health/readiness endpoint: curl -fsS https://<service>/healthz | jq .status and the smoke-test endpoint: curl -fsS https://<service>/readyz | jq .checks. Watch the error rate in the observability stack for 5 minutes; the rollback is not "done" until error rate returns to baseline
  10. 10STEP 10 - Record the rollback. Open an incident ticket with: the rolled-back service, the current and target revisions, the reason, the executor, the time, the verification evidence (logs, dashboards), and a follow-up action to fix the original change. The record is the audit trail

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓For Kubernetes rollbacks: kubectl get deploy <name> -n <ns> -o jsonpath="{.metadata.generation} {.status.observedGeneration} {.spec.template.spec.containers[0].image}" shows the target image and observedGeneration equals metadata.generation
  • ✓kubectl rollout status deploy/<name> -n <ns> reports "deployment rolled back successfully"
  • ✓All pods report Ready (kubectl get pods -n <ns> -l app=<name>)
  • ✓The application health/readiness endpoint returns {"status":"ok"} and the error rate in the observability stack has returned to pre-incident baseline
  • ✓For container rollbacks: docker ps --filter name=<service> --format "{.Image}" shows the previous tag
  • ✓For Terraform rollbacks: terraform plan reports "No changes". Infrastructure drift that existed before the rollback is still present; drift that the rollback introduced is a procedure bug

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the Kubernetes kubectl rollout undo fails or hangs: check the ReplicaSet history with kubectl rollout history deploy/<name> -n <ns>; if the target revision is missing, the rollback is impossible via undo. Fall back to editing the image field directly with kubectl set image deploy/<name> <container>=<image>:<previous> and a subsequent rollout
  • ↶If the Argo CD GitOps-managed rollback is reverted by the controller within minutes: the bad Git commit has not been reverted in the GitOps repo. Revert it (git revert <bad-sha>), push, and let the controller sync. Imperative rollbacks against GitOps clusters are temporary
  • ↶If the container rollback starts but the application immediately fails: the previous image is also broken (the bug predates the change you are trying to roll back from). Roll forward to the previous-previous known-good; if none exists, escalate per the runbook
  • ↶If the Terraform rollback terraform apply fails midway: do NOT retry blindly. The state still reflects reality for the operations that completed; re-run terraform plan, review what remains, and apply the remainder — or engage the platform team if the remaining plan is not the expected tail of the reversal
  • ↶If the rollback succeeded but the observability stack still shows elevated errors: the application is healthy but the previous revision also has a latent bug. The rollback is complete; the incident continues. Open a follow-up
  • ↶If the rollback took effect but the change ticket / Git history still shows the bad commit: the audit trail is broken. Open a follow-up to record what was deployed vs. what Git says; this is required for compliance

6 · Escalation

When the runbook isn't enough, contact:

  • · The previous known-good revision is not available (image deleted from registry, ReplicaSet pruned, infrastructure repo history rewritten so the known-good commit is unreachable): the rollback is impossible from history. Engage the platform team and the application owner to decide on a forward fix; if the service is down, escalate per the incident response process
  • · The rollback succeeds but the new revision is also broken: the incident is now a multi-regression. Engage the application owner and consider a freeze on further deployments to the service until the root cause is understood
  • · The Argo CD GitOps controller is ignoring the rollback because the cluster has drifted from Git (an emergency manual change was applied): see git-cicd-gitops-rb-19-reconcile-emergency-manual-change. The controller is correct; the manual change must be reconciled before the rollback can stick
  • · A Terraform rollback plan shows divergence greater than expected (>10 resources): the state does not match the live infrastructure. Do not apply, and do not attempt a state restore — state recovery is a separate disaster procedure (git-cicd-gitops-rb-26-recover-infra-pipeline). Engage the platform team
  • · A rollback causes a data-loss event (database migration was one-way): the rollback is technically correct but semantically wrong. Escalate to the data owner; do not attempt to "fix" by rolling forward without a data-recovery plan

A rollback reverts a deployment to a previously known-good revision. The revision is in the registry (container), in the ReplicaSet history (Kubernetes), or in the infrastructure repo’s Git history (Terraform). The rollback is a change event: capture the current state first, notify stakeholders, perform the revert, verify the application is healthy, and record the result.

1. Identify the rollback scope and capture pre-rollback state

Read-only / Safe
$ SERVICE="checkout-api"
NAMESPACE="prod"
CTX="prod-use1"
kubectl config use-context "$CTX"
echo "--- pre-rollback Deployment, ReplicaSets, Pods ---"
kubectl get deploy,rs,pods -n "$NAMESPACE" -l app="$SERVICE" -o yaml > /tmp/pre-rollback.yaml
kubectl get deploy "$SERVICE" -n "$NAMESPACE" -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.image}{"\n"}{end}'
echo "--- rollout history ---"
kubectl rollout history deploy/"$SERVICE" -n "$NAMESPACE"

The capture proves what was running. The rollout history shows the available revisions. The current image is the one being rolled back from.

2. Kubernetes rollback via kubectl rollout undo

Read-only / Safe
$ SERVICE="checkout-api"
NAMESPACE="prod"
TARGET_REVISION=3
kubectl rollout undo deploy/"$SERVICE" -n "$NAMESPACE" --to-revision="$TARGET_REVISION"
kubectl rollout status deploy/"$SERVICE" -n "$NAMESPACE" --timeout=300s
echo "--- post-rollback image ---"
kubectl get deploy "$SERVICE" -n "$NAMESPACE" -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

kubectl rollout undo is the standard Kubernetes rollback path. It is the correct procedure for clusters NOT managed by Argo CD or Flux. On GitOps-managed clusters, this is temporary — the controller will revert it.

3. GitOps rollback (Argo CD / Flux) via Git revert

Read-only / Safe
$ APP="checkout-api"
GITOPS_REPO="git@github.com:myorg/gitops-prod.git"
BAD_SHA="abc123def456"
git -C gitops-prod checkout main && git -C gitops-prod pull
git -C gitops-prod revert --no-edit "$BAD_SHA"
git -C gitops-prod push
echo "--- trigger immediate sync ---"
argocd app sync "$APP" --revision HEAD
argocd app wait "$APP" --health --timeout 300

The Git revert is the source-of-truth rollback. The argocd app sync triggers an immediate reconciliation; the argocd app wait blocks until the application is Healthy.

4. Helm rollback

Read-only / Safe
$ RELEASE="checkout-api"
NAMESPACE="prod"
helm history "$RELEASE" -n "$NAMESPACE"
helm rollback "$RELEASE" 3 -n "$NAMESPACE" --wait --timeout 300s
echo "--- post-rollback release ---"
helm list -n "$NAMESPACE" -f "$RELEASE"

Helm keeps a release history. helm rollback <release> <revision> restores a prior release in the same namespace. The --wait flag blocks until the rollout completes.

5. Terraform rollback (configuration revert)

Read-only / Safe
$ TF_DIR="/infra/prod/checkout-api"
BAD_SHA="abc123def456"
cd "$TF_DIR"
echo "--- safety snapshot of current state (disaster-recovery backup only) ---"
terraform state pull > "/tmp/state-$(date -u +%s).tfstate.backup"
ls -l /tmp/state-*.tfstate.backup | tail -1
echo "--- revert the configuration to the known-good intent ---"
git checkout main && git pull
git revert --no-edit "$BAD_SHA"
git push origin main   # or via PR / break-glass approval per repo policy
echo "--- plan against the CURRENT protected state ---"
terraform plan -out=/tmp/rollback.tfplan
echo "--- review the FULL plan BEFORE applying ---"
terraform show /tmp/rollback.tfplan
echo "--- apply only if the plan shows just the intended reversal ---"
terraform apply /tmp/rollback.tfplan

A Terraform rollback is a configuration revert planned and applied against the CURRENT protected state — never a state-file restore. The plan must show only the intended reversal; anything else means state and reality have diverged, so stop and investigate before applying. Restoring a state backup is a separate disaster-recovery procedure for proven state corruption or loss (see git-cicd-gitops-rb-26-recover-infra-pipeline) with locking, lineage/serial verification, backups, and post-restore reconciliation.

6. Verify the rollback took effect

Read-only / Safe
$ SERVICE="checkout-api"
NAMESPACE="prod"
echo "--- K8s deployment ---"
kubectl get deploy "$SERVICE" -n "$NAMESPACE" -o jsonpath='{.metadata.generation} {.status.observedGeneration} {.spec.template.spec.containers[0].image}{"\n"}'
echo "--- pods ---"
kubectl get pods -n "$NAMESPACE" -l app="$SERVICE" -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}'
echo "--- health check ---"
curl -fsS "https://$SERVICE.prod.internal/healthz" | jq -r '.status'
echo "--- error rate (5m) ---"
echo '(check observability dashboard for error rate returning to baseline)'

The deployment is rolled back only when the image matches the target revision, all pods are Ready, and the health endpoint reports ok. The error rate baseline check is the final confirmation.

7. Record the rollback

Read-only / Safe
$ SERVICE="checkout-api"
PREVIOUS_REVISION=3
CURRENT_REVISION=5
REASON="error rate spike from v4.2.1"
INCIDENT_ID="INC-1234"
gh issue create --repo myorg/myorg --title "rollback: $SERVICE rev $CURRENT_REVISION -> $PREVIOUS_REVISION" \
--body "Incident: $INCIDENT_ID. Reason: $REASON. Executor: $USER. Time: $(date -u +%FT%TZ). Verification: healthz ok, error rate at baseline. Follow-up: investigate root cause and ship fix as rev $((CURRENT_REVISION + 1))." \
--label rollback --label production --label incident

The record is the audit trail. Without it, the next operator does not know why a service is one revision behind, and the bug may re-appear.

Verification

The deployment shows the target revision’s image. All pods are Ready. The health/readiness endpoint returns ok. The error rate in the observability stack returns to pre-incident baseline within 5 minutes. For Helm: helm history shows the new revision. For Terraform: terraform plan reports “No changes”. The rollback is recorded in an incident ticket with the previous/target revisions, reason, executor, and follow-up.

Rollback

If kubectl rollout undo fails or the target revision is missing from the ReplicaSet history, fall back to kubectl set image to the previous tag. If the GitOps controller reverts the kubectl rollback within minutes, the bad Git commit has not been reverted — do that instead. If the container rollback starts but the application immediately fails, the previous image is also broken; roll forward to the previous-previous known-good. If the Terraform rollback apply fails midway, do NOT retry blindly — the state still reflects what completed; re-run terraform plan, review what remains, and apply the remainder or escalate to the platform team. If the rollback succeeded but errors remain elevated, the rollback is complete but the incident continues. If the audit trail is missing, open a follow-up to record what was deployed vs. what Git says.

References

  1. Kubernetes — Rolling Back a Deployment
  2. kubectl rollout undo
  3. Argo CD — Sync and Rollback
  4. Helm — helm rollback
  5. Terraform — State Rollback and Recovery
  6. Terraform — Recover from Failed State