The decision framework — what makes rollback safe and what makes forward-fix the only option

The reflex answer to a production incident is “roll back”. The reflex answer to a bad data migration is “forward-fix”. Both reflexes are wrong as defaults. The discipline of Part LX is to make the choice before the change ships, on the basis of four properties of the change itself.

The four properties

Every change has four properties that together determine whether rollback or forward-fix is safe:

• Idempotency. Can the change be applied twice without a different effect? An idempotent change can be re-applied; a non-idempotent change (allocates resources, increments counters) can be applied once and only once.
• Reversibility. Can the change be reverted by removing the new state? An additive change is trivially reversible; a destructive change may require a backup restore.
• Data effects. Does the change write data the rollback cannot unread? A schema migration that adds a column is data-neutral; one that drops a column writes data the rollback cannot restore.
• Dependency reach. How many systems does the change touch? A workload-only change has a narrow reach; a change that crosses schema, config, and cloud has a wide reach.

flowchart LR
    A["Change classified"] --> B{"Data written?"}
    B -- "yes" --> C["Forward-fix by default"]
    B -- "no" --> D{"State destroyed?"}
    D -- "yes" --> E{"Backup available?"}
    E -- "no" --> C
    E -- "yes" --> F["Restore or forward-fix"]
    D -- "no" --> G{"Idempotent?"}
    G -- "yes" --> H["Rollback is safe"]
    G -- "no" --> I["Forward-fix with care"]

The decision matrix

The four quadrants: idempotent with no data effects rolls back (config, image, deployment bug); idempotent with data effects forward-fixes (additive schema migration); non-idempotent with no data effects forward-fixes with care (resource allocation); non-idempotent with data effects forward-fixes (destructive migration).

Why classification is not a reflex

The reflex to roll back is correct in the common case (idempotent, no data effects) and wrong when the change has data effects. The discipline is to classify the change before it ships:

1. At design time. The author declares the classification in the pull-request template.
2. At review time. The reviewer confirms the classification against the four properties.
3. At deploy time. The deploy annotation carries the classification; the on-call engineer sees it without re-deriving it.
4. At incident time. The on-call engineer follows the documented path for that classification.

Production discipline

1. Classify every change at design time. The pull-request template asks for the rollback-versus-forward-fix classification.
2. Record the classification in the deploy annotation. The Kubernetes change-cause annotation, the GitOps commit message, the CI deploy summary all carry it.
3. Document the rollback path for every change. Even forward-fix changes need a documented rollback path.
4. Treat the classification as a deliverable. A “forward-fix” classification without a documented forward-fix is a guess.

Cross-course references

• This course, Part LIX-03 and LIX-04 cover the mechanisms the classification decides between.
• This course, Part LIX-06 covers the data effects quadrant in depth.
• PostgreSQL for Production Sysadmins - Part XXIV covers expand-and-contract.

Quiz