Rollback and data integrity — when rollback cannot be undone and the cascading data effects

A change can be rollback-able at the workload level and still be operationally destructive at the data level. The Deployment rolls back; the data the new ReplicaSet wrote is left behind. The replicas that already processed the new behaviour keep their state; the cache that warmed from the new responses holds the new values; the queue holds the new message format.

The cascade model

A rollback produces cascading data effects when the change’s behaviour has propagated beyond the workload that received the change:

flowchart LR
    A["Workload change"] --> B["Replica state"]
    B --> C["Cache layer"]
    C --> D["Queue / stream"]
    D --> E["Downstream consumer"]
    E --> F["Cascading rollback damage"]

Each layer the change touched becomes a layer the rollback does not reconcile:

• Replica state. Each replica processed the change’s new behaviour. In-memory state (sessions, counters, deduplication windows) reflects the new behaviour. Rolling back the workload does not roll back the in-memory state.
• Cache layer. The cache warmed from the new responses. Rolling back does not invalidate the cache.
• Queue / stream. The queue accepted messages in the new format. Rolling back does not drain the queue.
• Downstream consumer. The downstream service wrote its own state based on the new behaviour. Rolling back does not roll back that state.

The cascade is asymmetric. The change’s effects propagated forward; the rollback’s effects do not propagate backward.

Change patterns that produce cascades

1. Stateful service changes. A service that holds session state in memory produces a cascade: the new sessions are lost, in-progress transactions abandoned, deduplication windows reset.
2. Cache-warming changes. A service that warms a shared cache (CDN, Redis) produces a cascade: the cache holds entries the old code did not produce.
3. Message-format changes. A service that publishes in a new format produces a cascade: the queue holds messages the old consumer cannot parse.
4. Idempotency-key changes. A service that uses a different key algorithm produces a cascade: the downstream sees the same logical operation with two different keys and processes it twice.

Recognising the cascade

The cascade is recognisable in the change review:

• Does the service hold session state in memory? If yes, the rollback abandons the sessions. Forward-fix must include a session-migration step.
• Does the service warm a shared cache? If yes, the rollback leaves the cache warm. Forward-fix must include a cache-invalidation step.
• Does the service publish messages in a new format? If yes, the queue holds new messages. Forward-fix must include a queue-drain step or a dual-format consumer.
• Does the service use an idempotency key? If yes, the key algorithm change is observable downstream. Forward-fix must include a key-mapping step.

A change that answers yes to any of the four questions produces a cascade. The classification is “forward-fix by default”.

The forward-fix acceptance

1. Document the cascade in the deploy annotation. The on-call engineer sees the cascade warning and knows the forward-fix is the path.
2. Pre-stage the forward-fix. The compensating change is written, tested, and ready to merge.
3. Set the MTTR budget for the cascade case. The budget is the apply-and-verify time (7-12 min), not the write-and-test time.
4. Track the cascade in the incident record. Every cascade case builds the evidence that the cascade is a real cost.

Production discipline

1. Recognise the cascade in the change review. The four questions are part of the reviewer’s checklist.
2. Document the cascade in the deploy annotation.
3. Pre-stage the forward-fix for cascading changes.
4. Accept the forward-fix over the faster rollback. The acceptance is a deliberate decision documented before the change ships.
5. Track every cascade case in the incident record.

Cross-course references

• This course, Part LIX-06 covers the change-level classification.
• This course, Part LVIII-04 covers routing that isolates the cascade.
• Linux for Production Sysadmins - Part XXXI covers the metrics that detect the cascade.

Quiz