When rollback is the right answer — config regressions, deployment bugs, image issues

Three categories of production change are rollback-safe by construction: config regressions, deployment bugs, and image issues. For these, the on-call engineer should reach for the rollback command; the forward-fix is the wrong reflex.

Config regressions

A config regression is a change to configuration that produces a behaviour the change did not intend. The regression is in the configuration; the data the system holds is unchanged. Examples: a ConfigMap value that broke feature-flag routing; a Terraform variable that shortened an idle timeout; an Ansible playbook that set a rejected sysctl; a Helm values file that set replicas to 0.

By the four properties: idempotent (yes), reversible (yes, via version control), data effects (no), dependency reach (narrow). The default response is rollback.

Deployment bugs

A deployment bug is a change to the workload manifest that produces a behaviour the change did not intend. The bug is in the pod template, the deployment strategy, or the resource limits. Examples: a readiness probe pointing at an unserved path; resource limits below the working set triggering OOM; an affinity rule excluding every node; a StatefulSet volume claim template change triggering an immutable-field error.

kubectl rollout history deployment/$NAME
kubectl rollout undo deployment/$NAME --to-revision=$N
kubectl rollout status deployment/$NAME

The pattern: inspect the history, identify the previous good revision, run the rollback, watch the rollout complete. The rollback restores the previous ReplicaSet; the cluster resources the Deployment depends on are unchanged by the bug.

Image issues

An image issue is a change to the container image that produces a behaviour the change did not intend. Examples: a broken migration runner baked into the image; a vulnerable base layer; a latest tag resolving to a development build; a missing CA certificate bundle. The image is part of the pod template; the rollback restores both.

kubectl rollout undo deployment/$NAME --to-revision=$N
argocd app rollback $APP_NAME

Caveat: an image tagged latest cannot be rolled back to a specific version. Pin by digest or by an immutable tag.

The operational pattern

The pattern for the three categories:

1. Inspect the history. kubectl rollout history, git log, the Terraform state diff. Identify the previous good revision.
2. Classify the change. Confirm the four properties match the category. If any property fails, fall back to Part LX-03.
3. Run the rollback. kubectl rollout undo, argocd app rollback. Watch the rollout complete.
4. Verify after. Confirm the application is healthy, the data tier is unchanged, and the surrounding cluster (ConfigMap, Secret, PVC, Service) is intact.

sequenceDiagram
    participant Op as On-call engineer
    participant K as kubectl / GitOps
    participant C as Controller
    participant W as Workload
    Op->>K: rollout history deployment/$NAME
    K-->>Op: revisions list
    Op->>Op: classify change
    Op->>K: rollout undo --to-revision=$N
    K->>C: patch Deployment
    C->>W: scale up rev $N
    C-->>Op: rollout complete
    Op->>Op: verify health

Production discipline

1. Inspect the history before running the rollback. Confirm the previous good revision matches the expected good state.
2. Classify against the four properties. Config, deployment, and image are rollback-safe; anything that writes data is not.
3. Watch the rollout, do not fire and forget. kubectl rollout status is the guard.
4. Verify the cluster after the rollback. The workload rolled back; the surrounding cluster is the operator’s responsibility.
5. Document the rollback in the incident record. The audit trail must show the revision, the rationale, and the verification.

Cross-course references

• This course, Part LIX-03 (Kubernetes rollback) and Part LIX-02 (Container rollback) cover the mechanisms.
• This course, Part LVIII-02 (Rolling update) covers the ReplicaSet model.

Quiz