When forward-fix is the right answer — schema migrations, breaking changes, data issues

Three categories of production change cannot be rolled back safely: schema migrations, breaking changes, and data issues. For these, forward-fix is the only path that preserves the system’s state.

Schema migrations

A schema migration alters the structure the application reads and writes. The migration’s up script writes data the application’s rollback cannot unread; the down script cannot undo the data. Examples: an additive migration that adds a column with a default; a backfill migration that sets values for old users; a destructive migration that drops a column; an alter migration that rewrites every row.

-- Bad migration
ALTER TABLE users DROP COLUMN email;
ALTER TABLE users ADD COLUMN email_address TEXT;

-- Forward-fix
ALTER TABLE users ADD COLUMN email TEXT;
UPDATE users SET email = email_address;

The forward-fix is a compensating migration that brings the database forward to a state where both columns exist. By the four properties: idempotent (yes), reversible (partially - schema can be reverted, data cannot), data effects (yes), dependency reach (wide - shared by every consumer). The default response is forward-fix.

Breaking changes

A breaking change removes or renames a capability the system depends on. It does not destroy data but breaks the contract the data or calling code relies on. Examples: a Kubernetes API version bump that removes a field; a Terraform argument rename that triggers destroy-and-create; a REST endpoint removal that breaks a downstream service; a message-broker topic rename that breaks consumers.

By the four properties: idempotent (no), reversible (no), data effects (indirect - the old contract’s data may still exist), dependency reach (wide). The default response is forward-fix: write a new version of the contract, migrate consumers, deprecate the old version.

Data issues

A data issue alters the data the system holds in a way the change did not intend. The application and schema may be unchanged; the data is the issue. Examples: a bulk update that set values for the wrong cohort; a data migration that backfilled from the wrong source; a deletion script that matched the wrong rows; a replication lag that applied a write to the wrong shard.

By the four properties: idempotent (yes), reversible (no - previous values are typically not available), data effects (yes), dependency reach (wide). The default response is forward-fix: a new migration that corrects the data, tested in staging, applied in production.

The operational pattern

The pattern for the three categories:

1. Classify the change. Confirm the four properties match the category. If any fails, fall back to Part LX-02.
2. Start the forward-fix immediately. Do not wait for the rollback to fail.
3. Write the compensating change. A migration that corrects the data, a new API version that consumers migrate to, a data fix-up script that restores previous values.
4. Test in staging. The forward-fix must work on a copy of production data.
5. Apply in production. The forward-fix is the rollback; the original change is not reverted.

sequenceDiagram
    participant Op as On-call engineer
    participant FB as Forward-fix branch
    participant ST as Staging
    participant PR as Production
    Op->>FB: write compensating migration
    FB->>ST: apply to staging copy
    ST-->>Op: verify correctness
    Op->>FB: merge to main
    FB->>PR: deploy via CI
    PR-->>Op: confirm data correct

Production discipline

1. Classify before shipping. A change that crosses the workload-data boundary is forward-fix only; the classification is in the deploy annotation.
2. Start the forward-fix immediately when classified as forward-fix only. Do not wait for the rollback to fail.
3. Test the forward-fix in staging on a copy of production data.
4. Verify the forward-fix’s effect on the application, not just on the data.
5. Document the forward-fix in the incident record. The audit trail must show the original change, the forward-fix, and the verification.

Cross-course references

• This course, Part LIX-06 (Database rollback) covers expand-and-contract.
• PostgreSQL for Production Sysadmins - Part XXIV covers PostgreSQL migrations.
• This course, Part LVIII-04 (Blue-green) covers routing that buys time for the forward-fix.

Quiz