Abort criteria and the point of no return - rollback as part of the procedure

Most change plans have a rollback section, and most of them are a paragraph written after the procedure was finished, describing in general terms how the change could be undone. It is filed, approved, and never read until the night it is needed - at which point it turns out to describe a package version that no longer exists, or to assume a state the failed change did not leave the system in.

The course has already covered what to roll back to: capture the artefact before you replace it. This lesson is about the decision - when to reach for it, who decides, and the part of the procedure after which reaching for it is no longer possible.

Abort criteria are written before the change

An abort criterion is a statement, written while nobody is under pressure, of an observation that ends the change. It has to be evaluable by the person at the keyboard at 02:00 without convening anyone.

Bad: “abort if the change is causing problems.”

Good, from a real change plan:

ABORT IMMEDIATELY IF:
  - health endpoint returns non-200 for more than 60 s after
    the service reports active
  - p99 latency exceeds 250 ms (pre-change p99 was 90 ms) for
    two consecutive 1-minute samples
  - error log rate exceeds 10/min on the changed host
  - replication lag on the standby exceeds 30 s and is rising
  - any host in the cohort fails its health gate
  - the change has taken more than 25 minutes on one host
    (expected 6 minutes)

ABORT AT THE END OF THE WINDOW REGARDLESS OF PROGRESS.

Four properties make those usable.

1. Each has a number and a comparison, so two people reading it reach the same answer
2. Each names where the number comes from, so it can be checked in one command
3. Each is compared against the pre-change capture rather than an absolute that may always have been breached
4. Elapsed time is one of them - a change taking four times its expected duration is failing, whatever the metrics say

The last one is the one people omit and the one that saves windows. A change that has run long is a change that has met a condition nobody predicted, and it consumes the time the rollback will need.

The point of no return

Every procedure has a step after which rollback stops meaning “put it back” and starts meaning “restore from backup”. Find it during planning, mark it in the document, and treat reaching it as an explicit decision rather than as the next line.

Steps that create one:

The pattern for making a procedure reversible is the same in every case: separate adding from removing, and put a soak between them. Add the new column, write to both, verify, and drop the old one next week. Add the new node, resynchronise, verify, and remove the old one next week. The cleanup is a second change, with its own record, made when the first is proven.

$ sudo lvcreate --size 10G --snapshot --name pre-upgrade /dev/vg0/root; sudo lvs -o lv_name,lv_size,origin,data_percent vg0

  Logical volume "pre-upgrade" created.
LV           LSize   Origin Data%
root         200.00g
pre-upgrade   10.00g root     0.00

Budget the rollback time

The rollback needs to fit in the window too. Work backwards from the window’s end.

Window:                       22:00 - 02:00   (4 h)
Change duration, measured:    6 min per host x 12 hosts = 72 min
Rollback duration, measured:  9 min per host x 12 hosts = 108 min
Validation:                                               30 min
                                                        ---------
Latest safe start for the last host:  02:00 - 108 min = 00:12

So the rollout has a hard cut-off at 00:12, whatever progress has been made. Past that, a full rollback no longer fits, and continuing means accepting that a failure will be resolved during business hours with users watching.

Both durations must be measured rather than estimated, and the rollback figure is the one nobody has. Timing the rollback in staging is the single cheapest thing you can do to make a change plan honest - and the exercise routinely reveals that the documented rollback does not work at all.

Rolling back versus fixing forward

Rolling back is the default. Fixing forward is a decision that must be justified, because it is a change made under pressure, without review, on a system already in an unknown state.

Fix forward only when all of these hold:

• The cause is understood - not suspected - and the fix follows from it directly
• The fix is smaller than the rollback and can be validated the same way
• You are past the point of no return, so rollback was never available
• The rollback itself is the risky operation, which happens when the change has already altered state that reverting would corrupt

“We are nearly finished” is not on that list. Neither is “the rollback will take longer” on its own - a slow return to a known state is better than a fast arrival at a new unknown one.

The trap is sunk cost. Three hours into a four-hour window, having solved four unexpected problems, the fifth one looks like another twenty minutes. It usually is not, and the cost of being wrong is now paid in daylight.

Close the loop

A rollback is not a failure of the change process; it is the change process working. But it does end the change, and the change record has to say so:

• what was observed and which abort criterion it met
• what was rolled back, on which hosts, and what was left in place
• how the rolled-back state was verified - the same validation the change would have had
• whether any host was left in a mixed state, and what is being done about it
• what has to be true before the change is attempted again

That last line is what separates a rollback from a retreat. A change rolled back without it will be attempted again next month in exactly the same form.

Knowledge check