Upgrade validation — lab test, staged rollout, monitoring

An upgrade validation is the operator’s defence against a VyOS upgrade that produces a regression in production. The validation has three stages: lab test, staged rollout, and production monitoring. Each stage catches a different class of issue; the combination is the production alternative to free-form deployment.

This lesson covers each stage, the production patterns for combining them, and the failure modes where upgrade validation is missing.

Stage 1: lab test

The lab test is the operator’s first defence. The operator boots the new image in a lab environment that mirrors the production fleet:

flowchart LR
  PROD[Production fleet\nN routers] -->|mirror| LAB[Lab environment\nN routers]
  LAB -->|boot new image| BOOT[Boot verification]
  BOOT -->|verify config| CONFIG[Configuration verification]
  CONFIG -->|test routing| ROUTING[Routing protocol test]
  ROUTING -->|test traffic| TRAFFIC[Traffic verification]
  TRAFFIC -->|all pass| PASS[Lab test passed]
  TRAFFIC -->|fail| FAIL[Lab test failed]

The diagram shows the lab test procedure. The lab environment mirrors the production fleet (same model, same interface layout, same configuration). The operator boots the new image, verifies the configuration, tests the routing protocols, and verifies the traffic flows.

The lab test catches:

• Boot failures — the new image fails to boot on the hardware.
• Configuration errors — the configuration is rejected or behaves incorrectly on the new image.
• Routing protocol regressions — OSPF adjacencies do not establish; BGP sessions do not come up.
• Traffic regressions — the lab traffic does not flow correctly through the upgraded router.

Stage 2: staged rollout

The staged rollout deploys the upgrade to a subset of production routers first. The operator upgrades one router (or a small group) and verifies the upgrade before proceeding to the rest of the fleet:

flowchart LR
  LAB[Lab test passed] -->|deploy to| CANARY[Canary router\nfirst in production]
  CANARY -->|verify| CANARY_CHECK{Canary\nOK?}
  CANARY_CHECK -->|yes| ROLLOUT[Roll out to rest of fleet]
  CANARY_CHECK -->|no| ROLLBACK[Roll back canary]
  ROLLOUT -->|verify each router| MONITOR[Production monitoring]
  ROLLOUT -->|fail| ROLLBACK

The diagram shows the staged rollout procedure. The operator deploys to a canary router first; the canary is verified; the operator rolls out to the rest of the fleet. A failure on the canary triggers a rollback without affecting the rest of the fleet.

The staged rollout catches:

• Production-specific issues — the lab environment may not exactly mirror production; the canary surfaces production-specific issues.
• Fleet-wide issues — the canary is one router; the rest of the fleet may have different hardware or configuration.
• Operator process issues — the canary tests the operator’s upgrade procedure in production.

Stage 3: production monitoring

After the upgrade is deployed to the fleet, the operator monitors the production routers for regressions:

• Routing protocol sessions — show ip ospf neighbor and show ip bgp summary show the session states.
• Interface counters — packet drops, errors, and throughput.
• CPU and memory usage — show system resource shows the system resource usage.
• Log messages — the system log shows errors, warnings, and informational messages.

The monitoring runs for at least 24 hours after the upgrade. The operator watches for:

• Adjacency flaps — OSPF or BGP adjacencies flap repeatedly.
• Configuration warnings — the commit log shows warnings about deprecated options.
• Performance regressions — CPU or memory usage is higher than before the upgrade.
• Traffic regressions — packet drops or throughput regressions.

A regression triggers the rollback procedure. The operator rolls back the upgraded routers to the previous image using the dual-image model.

Failure modes

Lab test with simplified configuration

The operator runs the lab test with a simplified configuration. The lab test passes. The production upgrade fails because the production configuration has an option the lab did not have.

Diagnostic: the production commit fails with a schema error that the lab did not surface.

Fix: re-run the lab test with the production configuration; catch the schema change; migrate the configuration; re-deploy.

Canary not representative

The operator selects a canary router that is not representative of the fleet (e.g. the smallest router when the fleet has larger routers). The canary upgrade succeeds. The rollout to larger routers fails because of a hardware-specific issue.

Diagnostic: the rollout to the larger routers fails with a hardware error.

Fix: select a canary router that is representative of the fleet (e.g. the most common model); verify the upgrade on the canary; proceed with the rollout.

Production monitoring not running

The operator deploys the upgrade and does not monitor the production routers. A regression surfaces 24 hours later when a downstream system reports an issue.

Diagnostic: the regression is detected by an external system; the operator investigates; the regression has been ongoing for 24 hours.

Fix: implement the production monitoring stage; the operator watches for regressions for at least 24 hours after the upgrade.

Rollback not tested

The operator deploys the upgrade but has not tested the rollback procedure. A regression surfaces; the operator attempts to roll back; the rollback fails because the dual-image model is not configured correctly.

Diagnostic: the rollback fails; the operator must manually boot the previous image from the bootloader.

Fix: test the rollback procedure before deploying the upgrade; verify the dual-image model is configured correctly.

Rollback

The upgrade validation’s rollback is the dual-image model. The operator sets the default boot to the previous image and reboots. The previous image boots; the upgraded routers are rolled back.

The defensive idiom: the rollback is tested before deploying the upgrade. The operator knows the rollback works; the operator can deploy the upgrade with confidence.

Production discipline

Cross-course references

• LVI-VyOS-Upgrades (vyos-lvi-02-rolling-upgrade, vyos-lvi-03-frr-compat) cover the rolling upgrade and FRR compatibility that the validation procedure verifies.
• XLIX-VyOS-Monitoring (vyos-xlix-06-prometheus-exporters) covers the monitoring tools that the production monitoring stage uses.
• LVI-VyOS-Upgrades (vyos-lvi-05-upgrade-rollback, the next lesson) covers the upgrade rollback that the validation procedure’s regression handling uses.

Quiz