Data rollback versus state rollback — the two meanings of "rollback" for an infrastructure change

Advanced⏱ ~24 mingit

What you'll learn

Distinguish the rollback of the desired state from the rollback of the data the previous state wrote
Identify which artifacts the state rollback touches and which the data rollback must touch
Recognise why data rollback is forward-only in most production systems
Plan a rollback that handles the state half with git revert and the data half with a compensating forward-fix

Prerequisites

Verified against Git 2.55.x teaching target; 2.40+ minimum · GitHub Actions continuous service; Aug 2026 documentation baseline · Argo CD v3.5.x teaching target; v3.0+ minimum · Flux v2.9.x · Sigstore Cosign v3.1.x · SLSA v1.2 · OCI Distribution Specification v1.1 · Git LFS v3.7.1 · Kubernetes (cross-course target) 1.36.x

The word rollback in an infrastructure incident hides two distinct operations that an on-call engineer must reason about separately. A state rollback reverts the desired state: the Kubernetes manifests, the Terraform code, the Ansible playbook, the configuration files in the Git repository. A data rollback addresses the data the previous state wrote: the rows the new schema wrote, the files the old cron job wrote, the secrets the new rotation produced, the certificates the new CA signed. The two operations target different artifacts; they are governed by different rules; only one of them is recoverable with git revert. Treating them as the same operation is the source of the most expensive class of incident in production systems.

The two artifacts

Every infrastructure change writes into two stores: the declarative store (the manifests, the code, the configuration) and the imperative store (the rows, the files, the secrets, the certificates). A change that introduces a new column writes a schema migration into the declarative store and the rows produced by the migration into the imperative store. A change that rotates a credential writes the new Secret manifest into the declarative store and the new credential into the imperative store. A change that updates a config file writes the new file into the declarative store and any process restarts that depend on it into the imperative store.

flowchart LR
    A["Desired state change"] --> B["Declarative store (Git)"]
    A --> C["Imperative store (cluster, DB, secrets)"]
    B -. "git revert" .-> D["Declarative rollback"]
    C -. "forward-fix only" .-> E["Data rollback"]

The two stores have different rules for rollback:

The declarative store is a version-controlled set of files. The rollback is git revert (or an equivalent new commit that inverts the change). The audit trail is the branch history. The cost is small: a commit and a reconciliation.
The imperative store is the running system. The rollback is a forward operation: a compensating migration, a rotation, a file restore. The audit trail is the change log or the audit log. The cost is high: the data the previous state wrote is the data the new state must reconcile with.

What `git revert` can and cannot undo

git revert is the right tool for the declarative half. The manifest that introduced the bad probe path is on the branch; git revert <commit> produces a new commit that removes it; the next reconciliation restores the cluster to the previous probe path. The new commit’s tree is the inverse of the old commit’s tree, and the controller’s reconciliation makes the cluster match.

What git revert cannot undo is the data the reverted state wrote. If the bad schema migration added a column and rows in that column were written before the rollback, the rows remain in the database after the rollback. The schema is reverted; the data is not. If the bad credential rotation replaced a secret in the vault, the new secret remains after the manifest is reverted; only the manifest’s declaration of the secret reference is reverted, not the secret itself.

flowchart LR
    subgraph Before["Before rollback"]
        M1["Manifest: schema v2"] --> D1["DB: column added, 1000 rows"]
    end
    subgraph After["After git revert"]
        M2["Manifest: schema v1"] --> D2["DB: 1000 rows in the old shape"]
    end
    Before --> After

The asymmetry is what makes data rollback hard. The schema is revertible with git revert; the rows are not. The Secret reference is revertible; the Secret value is not. The configuration is revertible; the running processes that loaded the configuration are not.

The data rollback is forward-only

The production discipline for data rollback is that it is a forward operation, not a backward operation. A backward operation would be a database restore from backup, a secret vault restore, or a file system restore from snapshot. These operations exist; they are used as a last resort when the data is corrupt or unrecoverable; they are not the standard rollback path because they lose every change made after the backup, including the changes unrelated to the bad commit.

The standard data rollback is a compensating change. A compensating migration adds the rows back in the old shape (or drops the new column in a backward-compatible way). A compensating rotation re-issues the old secret as the new active credential, and lets the natural expiration take the new credential out of service. A compensating config reload forces the running processes to re-read the reverted config and re-establish the old behaviour.

The expand-and-contract discipline

The discipline that makes data rollback safe is expand and contract. The discipline replaces each breaking change with two non-breaking changes: an expand step that adds the new column (or table, or secret) without removing the old one, and a contract step that removes the old one once the new one is proven. A change that followed expand-and-contract has the old shape and the new shape coexisting in production during the rollout. The rollback can drop the new shape and restore the old one without losing data, because the data in the old shape was never removed.

The three examples:

Schema migration. Expand: add the new column, write to both old and new, backfill the new from the old. Contract: drop the old column once the new column is the only source of reads. The rollback of the contract step is to drop the new column; the data in the old column is preserved.
Secret rotation. Expand: issue the new secret alongside the old one; the manifest references both. Contract: deprecate the old secret. The rollback of the contract step is to re-enable the old secret; the data in the old secret is preserved because it was never removed.
Configuration change. Expand: introduce the new config key alongside the old one; the application reads the new key if present, falls back to the old. Contract: remove the fallback. The rollback of the contract step is to remove the new key; the application falls back to the old.

A change that did not follow expand-and-contract has no safe data rollback. The forward fix is the only option, and the forward fix is harder than the change would have been if the discipline had been applied at change time.

Choosing the right rollback for the failure

The two halves of a rollback are sequenced by the failure:

State bad, data untouched. A bad manifest that has not yet been applied to live data. The rollback is git revert; the data rollback is unnecessary.
State bad, data written, data still in the new shape. The bad migration ran for ten minutes before the rollback. The rollback is git revert for the state and a compensating migration for the data. The compensating migration is forward-only: it does not delete the rows in the new shape; it makes them invisible to the old shape (or vice versa, depending on which way the migration ran).
State bad, data corrupt. The data is in a shape no version of the application can read. The rollback is a forward fix that reconstructs the data from another source (a backup, a replication target, a derived view). The forward fix may take hours; the team accepts the outage while the data is repaired.

Why the declarative store and the imperative store cannot share a rollback mechanism

The declarative store is a content-addressed file system: every state is a tree of files identified by a SHA, every change is a diff between two SHAs, and the inverse of a diff is computable. git revert exploits this: the inverse of a commit is the diff that takes the tree at the bad commit back to the tree at its parent.

The imperative store is not content-addressed. A row in a database is a tuple of values; the row’s identity is the primary key, not a SHA. A file on disk is a sequence of bytes; the file’s identity is the path, not a SHA. A secret in a vault is a value; the secret’s identity is the name, not a SHA. The inverse of an imperative change is not computable from the change alone; it requires knowing what the imperative store contained before the change, and the imperative store does not remember.

The reason git revert cannot undo an imperative change is that the imperative change’s intent is not recorded in any structure that supports inversion. The change’s effect is recorded; the pre-change state is not. A forward fix reconstructs the pre-change state from external evidence; a backward restore reconstructs it from a backup. Neither is what git revert does.

Production discipline

Treat state and data as two distinct artifacts. A rollback that names one but not the other is incomplete.
Apply expand-and-contract to every breaking change. The discipline makes data rollback possible; its absence makes data rollback impossible.
Forward-fix the data; backward-fix only when the data is unrecoverable. A backward data rollback loses every change between the backup and the rollback; a forward fix loses nothing.
Document the data half in the post-mortem. The state half is in the branch history; the data half is not. The post-mortem is the only durable record of what the compensating change did.

Cross-course references

This course, Part LIX-06 (Database rollback) - the data question in detail, including expand-and-contract in schema migrations.
Postgres for Production Sysadmins - Parts on migrations cover the forward-fix discipline.
This course, Part XIV (Reverting commits) - git revert for the state half.

Quiz

Knowledge check · 4 questions

Q1. A team deploys a new application version that writes rows into a new `orders_v2` column. After five minutes, a bug is found and the team runs `git revert` on the deployment commit. Which half of the rollback is incomplete?

The state half; the manifest still references the new column The data half; the rows written to `orders_v2` during the five-minute window remain in the database and the reverted application cannot read them Both halves; `git revert` cannot undo a Kubernetes Deployment change Neither half; `git revert` reverts both the schema and the data
Q2. The expand-and-contract discipline makes data rollback safe by keeping the old shape of the data in production during the rollout, so the rollback can drop the new shape without losing data.

True False
Q3. Name the two artifacts an infrastructure rollback must distinguish between, and identify which one `git revert` can undo and which one requires a compensating forward-fix.
Q4. Plan a rollback that addresses both the state and the data halves, and identify the failure mode of treating data rollback as if it were state rollback.

A team deploys a schema migration that drops the `customer_email` column and replaces it with `customer_contact` (a JSONB column containing email and phone). The migration runs in production for 90 minutes before a bug is reported: customers cannot receive password-reset emails because the application reads from `customer_email`. The team runs `git revert` on the migration commit and deploys. After the revert, the application reads from `customer_email` but the column does not exist in the database. The team is in a panic.

Passing score: 75%. Answers are checked in this browser.