Runbook: Perform a Production Router Upgrade

1 · Prerequisites

Confirm every item is in place before any state change.

• Image management — dual-image model, add system image, alt slot
• Rolling upgrade — dual-router topology, upgrade one at a time, verify traffic
• FRR compatibility — what an FRR version bump changes on VyOS, and what it does not
• Upgrade validation — lab test, staged rollout, monitoring
• Upgrade rollback — dual-image, automatic, manual, verify alt boot
• Upgrade anti-patterns — skip lab test, no rollback, double-firmware upgrade
• Saved configuration — /config/config.boot, scp, off-box copy, versioning
• save — making the configuration persist
• Out-of-band access — OOB management VRF, console server, IPMI, the lockout safety net

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Out-of-band console tested on this router, from the path you would actually use at 02:00 rather than the one that happens to be open. set system image default-boot followed by reboot is a bet that the box comes back on the network. The console is what you hold if it does not, and it is the only thing in this procedure that the new image cannot take away.
• · Current state captured and read, not just saved: show version for the running release, show system image for what is installed and which entry is default-boot. If both slots already hold the same release family, this router has no fallback and every rollback line below is fiction until you fix that.
• · The FRR version currently running is written down: vtysh -c "show version". A VyOS image upgrade is an FRR upgrade whether or not anybody planned it that way, and the routing daemon is the component that decides whether your BGP and OSPF configuration still means what it meant yesterday.
• · The release notes for the target release have been read by the person running the change, not forwarded by them. Configuration families moved between 1.3 and 1.4 — BGP to system-as, firewall rule sets to set firewall ipv4 name, per-interface firewall bindings to set firewall ipv4 forward filter — and a migration that cannot map a node drops it rather than refusing to boot.
• · A current off-box configuration backup exists and its checksum has been verified against the router, per the backup runbook. The commit archive on this router is not a backup for this change: the router is about to boot into a different image with its own separate /config.
• · Free space checked with df -h /. add system image unpacks a complete second system; a partially written image on a full disk is discovered at boot, which is the worst possible moment.
• · The drain mechanism is decided and its propagation time is known, in seconds, for this topology: VRRP priority, a BGP export policy, or an administrative shutdown on named sessions. "We will drain it" without a mechanism means the reboot is the drain.
• · It is written down, and agreed, whether this is a rolling upgrade of a redundant pair or a planned outage of a single router. They are different changes with different windows and different approvals, and the second one is frequently run under the paperwork of the first.
• · The change window covers the reboot and the soak and a rollback, because a rollback is a second reboot. A window sized for the reboot alone forces the decision you least want to be forced: ship it broken or run over.
• · Whoever watches the other side has been told: the VRRP peer, the upstream provider, the iBGP mesh, the monitoring team. A planned failover that reaches somebody else as an unexplained alert costs more in trust than it saves in notice.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Confirm the shape of the change out loud before touching anything: redundant pair (traffic moves, the estate stays up) or single router (traffic stops). Everything below assumes the first; if it is the second, the drain steps become "declare the outage" and the soak still applies.
2. 2Capture the pre-change baseline and keep it off the router: show version, show system image, show configuration commands | strip-private, show interfaces, show ip route summary, show bgp summary, show ip ospf neighbor, show vrrp. Prefix counts from before are the only thing that makes the counts after meaningful.
3. 3Read the release notes for the target release and note the FRR version it ships. Compare it against vtysh -c "show version" from the baseline. A two-step FRR jump in one change is the anti-pattern the Part LVI lessons call the double-firmware upgrade, and its defining property is that a failure cannot be attributed to either half.
4. 4Freeze the configuration on this router and say so in the change channel. No commits from anyone until the upgrade is finished. This step has no command and prevents the most confusing failure in the procedure — see the callout on per-image configuration.
5. 5Persist what is running: save. Then prove the file matches: from configure, load /config/config.boot, compare (expect no output), discard. A diff here is exactly the state the new image would never see.
6. 6Take the off-box copy and verify it with sha256sum on both ends. The backup that matters for this change is the one that survives the router booting into an image with a different /config.
7. 7Drain the traffic. VRRP: set high-availability vrrp group NAME priority 50, commit. BGP: set protocols bgp neighbor 203.0.113.2 shutdown, commit, or an export route-map that makes this path unattractive. Do not save the drain — you want it gone at the next boot if everything else goes wrong.
8. 8Prove the drain from the other side, not from this router. show vrrp on the peer should show it as master; show bgp summary at the peer or upstream should show the prefixes arriving by the other path; the interface counters on the surviving router should have risen. A drain confirmed only by the box you drained is not confirmed.
9. 9add system image URL. When it asks whether to copy the current configuration and SSH host keys into the new image, say yes — declining leaves you booting a router with no configuration and new host keys, on the far side of a reboot. Note the time: this is the instant the configuration inside the new image is frozen.
10. 10Confirm the install landed: show system image lists both the running release and the new one. Both present means the dual-image fallback exists; only one means it does not, and the rollback plan needs rewriting before you go further.
11. 11set system image default-boot and select the new image. This is operational mode. There is no commit and no save to follow it — the change is to the bootloader, not to the configuration tree, and looking for a commit that never appears has sent operators back to re-run the command.
12. 12reboot, watching the console rather than the SSH session. Expect the router to be out of forwarding for the whole boot, typically a few minutes. If it has not returned in twice the time you expect, you are in the rollback path, not in a slow boot.
13. 13Confirm what actually booted before believing anything else: show version for the release, show system image for which entry is running and which is default-boot. A router that fell back to the alternate slot looks healthy and is running the old release.
14. 14Confirm the configuration survived migration: show configuration commands | strip-private, diffed against the pre-change capture. Read for missing lines, not changed ones. A migration that could not map a node drops it, and the page you get is a valid configuration that is quietly smaller than the one you had.
15. 15Confirm the control plane in dependency order — interfaces, then IGP, then BGP, then overlays: show interfaces, show ip ospf neighbor, show bgp summary, show ip route summary, show vrrp, and show interfaces wireguard or show vpn ipsec sa where those exist. Each layer explains the one above it, so read them in that order or you will debug BGP for an interface problem.
16. 16Confirm forwarding before restoring traffic: ping and traceroute through the router along a real path, not to its management address. A router with every adjacency up and a firewall or NAT rule that did not migrate answers pings perfectly and forwards nothing.
17. 17Undo the drain one step at a time, watching the counts return between steps: restore the VRRP priority, or delete protocols bgp neighbor 203.0.113.2 shutdown, then commit. Confirm with show ip bgp neighbors 203.0.113.2 advertised-routes that you are exporting what you exported before, at the same count.
18. 18Soak, with a named owner and a named end time, before touching the second router. The lessons ask for at least 24 hours of watching; anything shorter is a decision to accept regressions that only appear under a full traffic day, and it should be recorded as that decision rather than as an oversight.
19. 19save only if you changed configuration after the reboot, and understand where it lands: /config/config.boot inside the RUNNING image, which the previous image will never read.
20. 20Leave the previous image in place until the soak ends. delete system image is the step that converts a two-minute rollback into a reinstall, and it buys nothing but disk.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓show version reports the intended release and show system image shows it as both running and default-boot. If those two disagree, the router booted the fallback and the upgrade did not happen.
• ✓show configuration commands | strip-private diffed against the pre-change capture shows only differences you can name. Zero missing lines is the bar, not "nothing obviously wrong".
• ✓vtysh -c "show version" reports the FRR version the release notes said it would. A surprise here means the image is not the one you think it is.
• ✓show ip ospf neighbor shows every expected adjacency in Full, with the same neighbour list as the baseline — not merely a non-empty list.
• ✓show bgp summary shows every session Established with prefix counts within the normal range for this router. A session that is up while receiving a fraction of its usual prefixes is a policy or capability regression, and it looks like success in a screenshot.
• ✓show ip bgp neighbors PEER advertised-routes exports the same prefix set as before the change. Redistribution and export policy are the first things a schema change quietly alters.
• ✓show ip route summary totals match the baseline per protocol. This is the cheapest single check that a whole route source stopped feeding the table.
• ✓show vrrp shows this router back in its intended role, and the peer agrees. Two masters, or two backups, is a worse state than the one you started in.
• ✓Forwarding is proved end to end along a real path, through the router, after the drain is lifted — not only to the router.
• ✓The firewall is still doing its job: show firewall counters are moving on the rules that matter, and the input chain still has its default-action drop.
• ✓The soak has completed with its named owner reporting, and the previous image is still installed and still listed by show system image.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Fastest path, and the one to reach for first: set system image default-boot back to the previous release, then reboot. Two commands and one boot, and it is available at any point after the reboot until somebody deletes the old image.
• ↶If the new image never boots at all, the bootloader falls back to the alternate slot on its own and you arrive at the previous release without doing anything. Check show system image before assuming the upgrade succeeded — a successful automatic fallback and a successful upgrade look identical from a ping.
• ↶READ THIS BEFORE ROLLING BACK: each image has its own configuration. Booting the previous image restores the /config/config.boot belonging to that image, which is the state as of the moment you ran add system image — not the state as of the reboot, and not any change made after the upgrade. Retrieve the newer file first if you need it: copy file IMAGENAME://config/config.boot to /tmp/config.boot.saved.
• ↶A configuration mistake made on the new image, with the image itself fine, is a configuration rollback and not an image rollback: configure, rollback 1, compare, commit, save. Reaching for the image when the configuration is the problem costs a reboot you did not need.
• ↶Locked out over the network with the console alive: the console is unaffected by the firewall, by SSH, and by which image booted. set system image default-boot and reboot work from it exactly as they do from SSH.
• ↶POINT OF NO RETURN: delete system image on the previous release. After that the fallback is a reinstall and a restore, measured in hours rather than in one boot. Do not run it during the change; run it at the end of the soak, deliberately.
• ↶POINT OF NO RETURN: a configuration change committed and saved on the new image is not carried backwards by a rollback. If the change was needed to make the upgrade work, rolling the image back also rolls that change away, and the router returns to service without it.
• ↶Rolling back one router of a pair leaves a mixed-version estate. That is an acceptable intermediate state and a poor resting state: same-version is the configuration you have actually operated. Decide before you start how long you will tolerate mixed, and who decides to finish or reverse.

6 · Escalation

When the runbook isn't enough, contact:

• · Escalate to the data-centre or remote-hands team the moment the console does not answer, and stop the upgrade there. Every step after the pre-checks assumes a recovery path you have just discovered you do not have.
• · Escalate to network engineering before rebooting if the drain cannot be proved from the far side. A reboot without a confirmed drain is an unannounced outage of whatever share of traffic this router was still carrying.
• · Escalate to network engineering, and stop, if the post-boot configuration diff shows anything missing. A dropped node means the migration did not fully understand your configuration, and the rest of the fleet has the same configuration.
• · Escalate to the upstream provider or peer AS if prefix counts do not return to normal after the drain is lifted. The problem may be at their end of a session that reset during your reboot, and they will not correlate it with your change unless you tell them.
• · Escalate to the change owner rather than improvising if the window will not cover the soak. Shortening the soak is a decision somebody owns; letting it lapse quietly is not.
• · Escalate to security before returning the router to service if the upgrade changed anything about SSH host keys, API certificates, or user authentication in a way you did not plan. Operators who see a changed host-key warning and click through it are the exact population this router is meant to protect.
• · Escalate to the platform or vendor support channel with the console output before retrying a boot that failed twice. A second attempt at the same boot is rarely diagnosis and often just another outage.

References

1. VyOS documentation — updating VyOS
2. VyOS documentation — image management
3. VyOS documentation — installation and image verification
4. VyOS documentation — high availability (VRRP)
5. VyOS documentation — BGP
6. FRRouting releases and changelog
7. NIST SP 800-40 Rev. 4 — Enterprise Patch Management