Rollback across artifact boundaries — the five mechanisms and what each can undo

A “rollback” sounds like a single button. It is not. An infrastructure change touches five distinct artifacts: the container image, the Kubernetes Deployment, the Terraform state, the configuration management run, and the database schema. Each artifact has its own source of truth, its own history, and its own rollback mechanism. A rollback that restores one of them while leaving four others drifted is a partial reversion, and the drift between them is itself a failure mode. The discipline of Part LIX is to know which mechanism recovers from which failure, and to recognise the failures no mechanism can recover from.

The five rollback mechanisms

Each mechanism crosses a different artifact boundary. The mechanisms are not interchangeable: the failure you are recovering from dictates which boundary to undo.

flowchart LR
    A["Failure origin"] --> B{"Which artifact moved?"}
    B -- "Image digest" --> C["Image rollback"]
    B -- "Deployment spec" --> D["kubectl rollout undo"]
    B -- "Cloud resource" --> E["git revert + terraform apply"]
    B -- "Host config" --> F["Ansible next-run"]
    B -- "Schema" --> G["Forward-fix migration"]

• Application rollback reverts the container image digest pinned by the Deployment. The registry is the source of truth; the rollback promotes the previous digest that the registry still holds.
• Kubernetes rollback reverts the Deployment spec via ReplicaSet history. The controller remembers every ReplicaSet it has created; kubectl rollout undo re-applies the previous one.
• Terraform rollback reverts the cloud resources via state and applied configuration. A git revert of the bad commit plus a fresh reviewed plan and apply restores resources Terraform still owns; resources it no longer owns must be imported or recreated manually.
• Configuration management rollback does not need an explicit command: the next run restores the desired state declared in the playbook, because Ansible is idempotent.
• Database rollback is the hardest. Schema changes are forward-only by discipline; the rollback is a forward fix (a compensating migration), not a backward restore.

What each mechanism can and cannot undo

The boundary a mechanism crosses defines its reach. Crossing the boundary is the rollback; failing to cross it is the limit.

• Image rollback cannot undo a database schema the new application wrote. It can roll the application back; the data is whatever the new application wrote.
• Kubernetes rollback cannot undo a Node that drained on a new taint, a CRD that was removed, or a StorageClass that no longer exists. The Deployment’s previous ReplicaSet can be re-applied, but the cluster resources it depends on may not be there.
• Terraform rollback cannot undo a resource deleted out-of-band. State records what Terraform owns; if the resource was deleted by hand, applying the reverted configuration will recreate it, but the recreation may collide with the lingering state of the deleted resource.
• Configuration management rollback cannot undo a package the previous run installed and the new run uninstalls, if the uninstallation breaks a dependency. The next run restores the declared state; it does not remember intermediate states.
• Database rollback cannot undo a column drop if a forward fix has not been written. Expand-and-contract makes every rollback possible by leaving the old column in place until the new column is proven.

Choosing the right mechanism

The right rollback mechanism is the one whose boundary the failure crossed. Diagnose first; the diagnosis dictates the action.

• 5xx spike after image change, no schema change. Image rollback. Promote the previous digest; the Deployment rolls back automatically on the next GitOps sync, or explicitly via kubectl rollout undo.
• 5xx spike after manifest change, same image. Kubernetes rollback. Revert the manifest commit; the controller rolls back via argocd app rollback or kubectl rollout undo --to-revision=<n>.
• Resource drift after Terraform change. Terraform rollback. Revert the commit, review a fresh terraform plan against the current state, and apply; terraform state list confirms what Terraform owns.
• Host config drift after playbook change. Config-mgmt rollback. The next Ansible run restores the declared state; nothing else is needed.
• Data corruption after migration. Database forward-fix. Write a compensating migration; do not attempt to restore from backup unless the corruption is unrecoverable.

Production discipline

Five rules govern cross-boundary rollback:

1. Diagnose the failure origin before choosing the mechanism. A 5xx spike can be image, manifest, IAM, or schema. Read the diff, then act.
2. Roll back at the widest boundary that contains the failure. If the failure crosses boundaries, roll back at the boundary that contains them all.
3. Verify the rollback at every boundary. A successful image rollback is not a successful schema rollback. Check each artifact.
4. Treat a partial rollback as a failure. If you cannot roll back one of the affected boundaries, the rollback is incomplete; either extend it or escalate.
5. Document the boundary choice in the post-mortem. A future team needs to know which boundary the failure crossed so they can recognise the same pattern faster.

Cross-course references

• Kubernetes for Production Sysadmins - Part XXII (Updates) covers Deployment rollback and the ReplicaSet history mechanism.
• Terraform for Production Sysadmins - Part XII (State) covers state-driven rollback and the import-after-delete recovery path.
• Ansible for Production Sysadmins - Part XXXIV (ConfigMgmt) covers idempotency and the next-run restoration property.

Quiz