Upgrade rollback — dual-image, automatic, manual, verify alt boot

An upgrade rollback is the procedure for restoring the previous system image when the new image produces a regression. The dual-image model is the rollback mechanism: the previous image is in the alternate slot; the operator (or the bootloader) selects the alternate slot for booting.

This lesson covers the dual-image automatic rollback, the manual rollback, the verification that the alternate boot is working, and the production failure modes where the rollback does not fire.

The dual-image automatic rollback

The bootloader is configured with a fallback: if the primary slot fails to boot (kernel panic, missing image, corrupted filesystem), the bootloader falls back to the alternate slot.

flowchart TB
  P[Primary slot\nnew image\nfails to boot] -->|fallback| A[Alternate slot\nprevious image]
  A -->|boot| B[Router boots\nprevious image]
  B -->|operator investigates| I[Investigation]

The diagram shows the automatic rollback. The new image fails to boot; the bootloader falls back to the alternate slot; the router boots the previous image; the operator investigates.

The automatic rollback is the operator’s first line of defence. It catches:

• Boot failures — the new image fails to boot due to a hardware incompatibility.
• Kernel panics — the new image’s kernel panics on hardware that was not tested in the lab.
• Missing drivers — the new image does not include a driver that the hardware requires.

The automatic rollback does NOT catch:

• Configuration regressions — the new image boots but the configuration produces a regression (e.g. BGP session does not establish).
• Performance regressions — the new image boots but CPU usage is higher than before.
• Behavioural regressions — the new image boots and the configuration is accepted, but the behaviour is incorrect.

For these cases, the operator must use the manual rollback.

The manual rollback

For regressions that the automatic rollback does not catch, the operator performs a manual rollback:

# Set the default boot to the previous image
set system image 1.4
commit

# Reboot into the previous image
reboot

The operator sets the default boot to the previous image and reboots. The router boots the previous image; the new image is in the alternate slot for future reference.

flowchart LR
  NEW[New image\nin primary slot] -->|set system image| PREV[Previous image\ndefault boot]
  PREV -->|reboot| BOOT[Router boots\nprevious image]
  BOOT -->|verify| V[Verification\nrouting protocols up\ntraffic flows]
  V -->|all OK| DONE[Rollback complete]
  V -->|issue| DEBUG[Debug]

The diagram shows the manual rollback. The operator sets the default boot to the previous image, reboots, and verifies the previous image is working correctly.

Verifying the alternate boot

After the rollback, the operator verifies that the alternate boot is working correctly:

# Verify the router is running the previous image
show version
# Version: VyOS 1.4-rolling-...

# Verify the routing protocols are up
show ip ospf neighbor
show ip bgp summary

# Verify the configuration is applied
show configuration

# Verify traffic flows
ping <known-good-IP>

The verification is the operator’s confirmation that the rollback succeeded. A router that boots the previous image but does not have the routing protocols up is a router that requires additional troubleshooting.

Failure modes

Bootloader fallback not configured

The dual-image fallback is not configured on the router. The new image fails to boot; the bootloader does not fall back; the router is stuck in a boot loop.

Diagnostic: the router repeatedly reboots into the new image; the operator connects via console; the operator sees the boot loop.

Fix: connect via console; manually select the alternate slot from the bootloader menu; boot the previous image.

Manual rollback fails

The operator sets the default boot to the previous image and reboots. The previous image fails to boot (e.g. the previous image has a different bug). The router is stuck in a boot loop with no working image.

Diagnostic: the router repeatedly reboots; the operator connects via console; neither image boots correctly.

Fix: connect via console; boot the previous image from the bootloader; boot the rescue image from the ISO; the operator investigates.

Configuration not rolled back

The operator rolls back the image but does not roll back the configuration. The previous image boots but with the new configuration; the configuration may have references to options that the previous image does not support.

Diagnostic: the commit fails on the previous image with a schema error.

Fix: roll back the configuration to the previous revision (rollback N and commit); the previous image’s configuration is restored.

New image deleted before debugging

The operator deletes the new image before investigating the regression. The new image is no longer available for debugging.

Diagnostic: show system image shows only the previous image; the operator cannot boot the new image.

Fix: re-add the new image; investigate the regression.

Rollback

The upgrade rollback is itself a rollback mechanism. The dual-image model allows the operator to roll back the image; the configuration history allows the operator to roll back the configuration. The combination is the operator’s defence against a regression in either the image or the configuration.

The defensive idiom: the rollback is tested before deploying the upgrade. The operator knows the rollback works; the operator can deploy the upgrade with confidence.

Production discipline

Cross-course references

• LVI-VyOS-Upgrades (vyos-lvi-01-image-management, vyos-lvi-02-rolling-upgrade, vyos-lvi-04-upgrade-validation) cover the image management, rolling upgrade, and validation that the rollback procedure depends on.
• LV-VyOS-Backup (vyos-lv-01-saved-configuration) covers the saved configuration that the rollback uses.

Quiz