Upgrade anti-patterns — skip lab test, no rollback, double-firmware upgrade

Every upgrade failure mode is an anti-pattern that some operator wrote, some peer review approved, and some deployment executed. The anti-pattern is consistent across teams and across years: an operator under time pressure takes a shortcut that bypasses one of the production safeguards.

This lesson catalogues the seven upgrade anti-patterns the RunBook Academy has seen repeatedly:

1. Skip lab test — the operator deploys the upgrade without testing in a lab first.
2. No rollback path — the operator does not verify that the rollback procedure works.
3. Double-firmware upgrade — the operator upgrades two versions at once (e.g. from 1.4 to 1.5 and from FRR 9 to FRR 10 in the same change).
4. No staged rollout — the operator upgrades the entire fleet at once without a canary.
5. No monitoring — the operator deploys the upgrade and does not monitor for regressions.
6. Undocumented upgrade — the operator deploys the upgrade without a ticket or a runbook.
7. Untested playbook — the operator runs an upgrade playbook in production without testing it in a lab.

Each anti-pattern has a discipline that catches it and a production alternative.

Anti-pattern 1: skip lab test

Symptom: the operator deploys the upgrade without testing in a lab. The upgrade may have a boot failure, a configuration regression, or a performance regression that the lab would have caught.

Failure mode: the upgrade fails in production. The dual-image rollback restores the previous image, but the production traffic was affected during the failure.

Discipline that catches it: a mandatory lab test before deploying. The lab test uses the production configuration (or a representative subset of it).

Production alternative: every upgrade is preceded by a lab test. The lab test verifies the boot, the configuration, the routing protocols, and the traffic.

Anti-pattern 2: no rollback path

Symptom: the operator deploys the upgrade without verifying that the rollback works. The dual-image fallback may be misconfigured; the manual rollback may not be tested.

Failure mode: a regression surfaces; the operator attempts to roll back; the rollback fails; the router is stuck with the broken image.

Discipline that catches it: a mandatory rollback test before deploying. The operator sets the default boot to the previous image and reboots; verifies the rollback works; sets the default boot back to the new image.

Production alternative: every upgrade has a tested rollback procedure. The operator knows the rollback works before deploying the upgrade.

Anti-pattern 3: double-firmware upgrade

Symptom: the operator upgrades two versions at once (e.g. from VyOS 1.4 to 1.5 LTS and from FRR 9 to FRR 10 in the same change). The two upgrades interact; a failure in one is hard to attribute to the other.

Failure mode: the upgrade fails; the operator does not know whether the failure is due to the VyOS upgrade or the FRR upgrade; debugging is complicated.

Discipline that catches it: a one-version-at-a-time discipline. Each upgrade is a separate change with a separate ticket, a separate lab test, and a separate rollback procedure.

Production alternative: the operator upgrades VyOS and FRR in separate changes. The VyOS upgrade is deployed first; the FRR upgrade is deployed after the VyOS upgrade is verified.

Anti-pattern 4: no staged rollout

Symptom: the operator upgrades the entire fleet at once. The canary step is skipped.

Failure mode: a regression that affects the entire fleet surfaces; the operator cannot roll back one router; the operator must roll back the entire fleet.

Discipline that catches it: a mandatory staged rollout. The canary is upgraded first; the canary is verified; the rollout proceeds.

Production alternative: every upgrade is deployed to a canary first. The canary is representative of the fleet; the canary is verified before proceeding.

Anti-pattern 5: no monitoring

Symptom: the operator deploys the upgrade and does not monitor for regressions. The upgrade may have a performance regression that surfaces hours later.

Failure mode: a regression surfaces 24 hours after the upgrade. The operator detects the regression from an external system; the regression has been ongoing.

Discipline that catches it: a mandatory monitoring stage after the upgrade. The operator monitors for at least 24 hours; a regression triggers the rollback.

Production alternative: every upgrade is followed by at least 24 hours of monitoring. The operator watches for regressions; the monitoring catches issues that the lab and canary did not.

Anti-pattern 6: undocumented upgrade

Symptom: the operator deploys the upgrade without a ticket or a runbook. The post-incident review cannot find the intent.

Failure mode: the upgrade is in the production configuration; the rationale is missing; the rollback procedure is not documented.

Discipline that catches it: a mandatory ticket and runbook. Every upgrade is associated with a ticket that captures the intent, the implementation, and the rollback plan.

Production alternative: every upgrade has a ticket and a runbook. The ticket captures the intent; the runbook documents the procedure; the post-incident review can trace the upgrade to the ticket.

Anti-pattern 7: untested playbook

Symptom: the operator runs an upgrade playbook in production without testing it in a lab. The playbook has a bug that the lab would have caught.

Failure mode: the playbook’s bug is applied to production. The operator must debug in production.

Discipline that catches it: a mandatory lab test for the playbook. The playbook is run in the lab first; the lab catches the bug.

Production alternative: every playbook is tested in a lab before it is applied to production. The lab is the operator’s defence against a playbook bug.

The discipline

The discipline is a set of safeguards, each catching a different anti-pattern:

flowchart TB
  ANTI[Anti-pattern] -->|caught by| DISC[Discipline]
  DISC --> D1[Lab test\nskip lab test]
  DISC --> D2[Rollback test\nno rollback path]
  DISC --> D3[One version at a time\ndouble-firmware upgrade]
  DISC --> D4[Canary rollout\nno staged rollout]
  DISC --> D5[Production monitoring\nno monitoring]
  DISC --> D6[Ticket + runbook\nundocumented upgrade]
  DISC --> D7[Lab test for playbook\nuntested playbook]

The diagram shows the mapping: each anti-pattern is caught by a specific discipline. The discipline is the operator’s defence against the anti-pattern.

Failure modes

Discipline decays

The team implements all seven disciplines. Six months later, the team is under time pressure; the operator skips the lab test “just this once”. The upgrade fails in production.

Diagnostic: the upgrade ticket shows no lab test evidence; the post-incident review identifies the discipline decay as the cause.

Fix: re-implement the disciplines. The defensive idiom: the disciplines are configured in the platform (lab environment, branch protection, change ticket templates), not in the operator’s discipline.

Discipline is implemented inconsistently

The team implements the disciplines for some upgrades but not others. The new-team uses the full discipline; the existing-team uses only the upgrade command. The two teams produce different upgrade quality.

Fix: standardise the discipline across the team. The defensive idiom: the discipline is the same for every upgrade; the platform enforces the discipline.

Rollback

The upgrade rollback is the dual-image model. The operator sets the default boot to the previous image and reboots. The previous image boots; the upgraded routers are rolled back.

The anti-patterns themselves do not require a rollback; they require the discipline that prevents the anti-pattern.

Production discipline

Cross-course references

• LVI-VyOS-Upgrades (vyos-lvi-01-image-management through vyos-lvi-05-upgrade-rollback) cover the upgrade procedures that the anti-patterns bypass.
• LIV-VyOS-Automation (vyos-liv-06-automation-anti-patterns) covers the automation anti-patterns that complement the upgrade anti-patterns.
• LV-VyOS-Backup (vyos-lv-06-dr-validation) covers the DR drill that catches the discipline decay.

Quiz