The decision cost and time — MTTR trade-offs and the production discipline

For some changes, both rollback and forward-fix are technically possible. The decision turns on cost and time: which path lands within the MTTR budget, which preserves the data, which the team can execute under incident pressure.

The MTTR budget

MTTR is the time from incident detection to resolution. The MTTR budget is the upper bound the team commits to for a given incident class. The two paths spend the budget differently:

• Rollback. Detection (1-3 min), classification (1-2 min), rollback command (1 min), rollout (2-5 min), verify (2-3 min). Total: 7-14 minutes.
• Forward-fix. Detection (1-3 min), classification (1-2 min), write the compensating change (10-30 min), test in staging (5-15 min), apply (2-5 min), verify (2-3 min). Total: 21-58 minutes.

gantt
    title MTTR budget allocation
    dateFormat X
    axisFormat %s
    section Rollback
    Detect     :a1, 0, 2
    Classify   :a2, 2, 4
    Rollback   :a3, 4, 5
    Rollout    :a4, 5, 9
    Verify     :a5, 9, 12
    section Forward-fix
    Detect     :b1, 0, 2
    Classify   :b2, 2, 4
    Write fix  :b3, 4, 25
    Test       :b4, 25, 35
    Apply      :b5, 35, 40
    Verify     :b6, 40, 43

The forward-fix spends most of its budget writing the compensating change. The rollback spends most of it waiting for rollout. The difference is the 20-30 minutes the forward-fix needs to write the change.

When the budget is the decision

When both paths are technically possible, the MTTR budget determines the answer:

• 15-minute budget, change is rollback-safe: Rollback. The forward-fix exceeds the budget.
• 60-minute budget, change is forward-fix only: Forward-fix. The rollback is impossible.
• 30-minute budget, rollback-safe but data-effect boundary exists: This is the case where the two arguments disagree.

The third case is the one the production discipline must resolve. The rollback is faster; the forward-fix is safer for the data. The choice depends on the relative weight of latency and data preservation. A payment service losing revenue every minute may choose the rollback; a healthcare system that holds patient records may choose the forward-fix. The discipline is to make this choice before the change ships.

The pre-staging discipline

1. At design time. The author writes the forward-fix alongside the original change. Reviewed, tested, merged to a branch that is not deployed.
2. At deploy time. The original change deploys. The forward-fix branch is ready to merge.
3. At incident time. The on-call engineer merges the pre-staged branch, applies the migration, and verifies. The MTTR budget accommodates apply and verify, not write.

The cost is real: two changes to review, two CI runs. The benefit is that the on-call engineer does not spend the MTTR budget writing under incident pressure.

The runbook pattern

The runbook is the mechanism that lets the on-call engineer choose without re-deriving the answer:

• The deploy annotation carries the classification (rollback-safe, forward-fix only, both possible).
• The runbook for the workload links from the classification to the documented path.
• The incident record captures the path taken, the time per step, and the budget consumed.

kubectl rollout undo deployment/$NAME --to-revision=$N

The on-call engineer who runs this command has already classified the change, confirmed the previous revision is the expected good state, and verified the rollback is within the MTTR budget.

Production discipline

1. Set the MTTR budget per incident class. Customer-facing: tight (15 min). Batch: relaxed (60 min). Data-only: hours.
2. Pre-stage the forward-fix for every change classified as forward-fix only. The forward-fix is a deliverable.
3. Document the choice in the deploy annotation. The on-call engineer sees the classification without re-deriving it.
4. Track MTTR per incident and per class. A class that consistently exceeds its budget needs more pre-staging or a different mechanism.
5. Review the cost-and-time choice in the post-mortem.

Cross-course references

• This course, Part LV-06 (The decision framework) feeds the MTTR budget.
• This course, Part LVIII (Deployment patterns) buys time for the forward-fix.
• Linux for Production Sysadmins - Part XXXI covers detection latency.

Quiz