Rolling upgrade — dual-router topology, upgrade one at a time, verify traffic

A rolling upgrade is the procedure for upgrading a production VyOS deployment without taking the network offline. The operator upgrades one router at a time; the upgraded router is verified before the next router is upgraded; the network is operational throughout the procedure.

The rolling upgrade relies on a dual-router topology: the production traffic is forwarded by both routers (VRRP, ECMP, or a similar mechanism); when one router is rebooted for the upgrade, the other router continues to forward traffic. The upgraded router returns to service after reboot and resumes its share of the traffic.

This lesson covers the rolling upgrade procedure, the verification steps, the handling of a failed upgrade on the second router, and the production failure modes where the rolling upgrade is interrupted.

The dual-router topology

The production deployment has two (or more) routers that share the traffic:

flowchart LR
  CLIENT[Client traffic] --> LB[Load balancer / switch]
  LB --> E1["edge-01<br/>image 1.4"]
  LB --> E2["edge-02<br/>image 1.4"]
  E1 --> UPSTREAM[Upstream provider]
  E2 --> UPSTREAM

The diagram shows the dual-router topology. Both edge-01 and edge-02 forward traffic; the load balancer (or ECMP) distributes traffic between them. If one router fails or is rebooted, the other router continues to forward traffic.

The rolling upgrade procedure relies on this topology: when one router is rebooted for the upgrade, the other router continues to forward traffic; the upgraded router returns to service after reboot and resumes its share of the traffic.

The rolling upgrade procedure

The procedure has five steps for each router:

1. Drain the traffic — shift traffic away from the router being upgraded.
2. Add the new image — install the new image into the alternate slot.
3. Set the default boot to the new image — update the bootloader.
4. Reboot — reboot into the new image.
5. Verify and restore traffic — verify the new image is working correctly; restore traffic to the router.

sequenceDiagram
  participant OP as Operator
  participant E1 as edge-01
  participant E2 as edge-02
  Note over E1,E2: Both routers running image 1.4
  OP->>E1: drain traffic (route-map / VRRP priority)
  OP->>E1: add system image 1.5
  OP->>E1: set system image default-boot 1.5
  OP->>E1: reboot
  Note over E1: edge-01 rebooting...
  Note over E2: edge-02 carries all traffic
  E1->>E1: boot 1.5
  E1->>OP: edge-01 ready
  OP->>E1: verify (show version, show ip route, show bgp ipv4 summary)
  OP->>E1: restore traffic (remove route-map / VRRP priority)
  Note over E1,E2: edge-01 running 1.5, edge-02 running 1.4
  OP->>E2: drain traffic
  OP->>E2: add system image 1.5
  OP->>E2: set system image default-boot 1.5
  OP->>E2: reboot
  E2->>E2: boot 1.5
  E2->>OP: edge-02 ready
  OP->>E2: verify traffic
  OP->>E2: restore traffic
  Note over E1,E2: Both routers running 1.5

The diagram shows the rolling upgrade procedure. The operator upgrades edge-01 first; verifies; then upgrades edge-02. At each step, the other router carries the traffic; the production network is operational throughout.

Step 1: drain the traffic

The operator drains traffic away from the router being upgraded. The mechanism depends on the topology, and the first thing to settle is which direction you are draining, because the two directions need different levers:

• VRRP — lower the VRRP priority so the router becomes the backup and the other router takes over as master. This drains the traffic that LAN hosts send to the virtual address. Note that this only works if the group will actually preempt: if the group carries no-preempt, lowering the priority on the current master changes nothing until it stops advertising.
• BGP, outbound from your network — you want your own iBGP speakers to stop choosing this router as the exit. Lower the local-preference on the routes this router imports from its upstream, so the other router’s copies win the iBGP best-path comparison.
• BGP, inbound to your network — you want the upstream to stop sending traffic to this router. Local-preference cannot do that; it is a local, non-transitive attribute and the upstream never sees it. Prepend your own AS on export instead, so your prefixes look longer via this router.
• ECMP — there is no clean drain. The hashes are recomputed when the nexthop goes away, and flows on the router being rebooted are dropped in the gap. Plan the reboot inside a maintenance window rather than pretending there is a graceful path.

On VyOS 1.5 the VRRP group lives under high-availability, not under the interface:

configure
# Lower the VRRP priority (VRRP topology)
set high-availability vrrp group LAN priority 50
commit

The BGP drain is a route-map applied to the neighbour’s address family, with an explicit import or export direction:

configure
# Drain outbound: make this router a less attractive exit
# for our own iBGP speakers.
set policy route-map DRAIN-IN rule 10 description 'maintenance drain, restore after upgrade'
set policy route-map DRAIN-IN rule 10 action permit
set policy route-map DRAIN-IN rule 10 set local-preference 50
set protocols bgp neighbor 10.0.0.1 address-family ipv4-unicast route-map import DRAIN-IN

# Drain inbound: make our prefixes look longer to the upstream.
set policy route-map DRAIN-OUT rule 10 description 'maintenance drain, restore after upgrade'
set policy route-map DRAIN-OUT rule 10 action permit
set policy route-map DRAIN-OUT rule 10 set as-path prepend '64512 64512 64512'
set protocols bgp neighbor 10.0.0.1 address-family ipv4-unicast route-map export DRAIN-OUT
commit

A route-map with a single permit rule and no match clause applies its set to every route in that direction. That is what you want for a drain — but it is also why a DRAIN route-map left in place after the maintenance window is such a common and quiet mistake. Nothing alarms; the router simply stays half-drained until someone wonders why one edge carries twice the traffic of the other.

The drain is not instant. VRRP takes a few advertisement intervals; BGP takes as long as the far side needs to run best-path and propagate. Do not measure the drain by the clock — measure it by the counters, on the other router.

Step 2: add the new image

The operator adds the new image to the alternate slot:

add system image https://example.net/vyos-1.5.1-generic-amd64.iso

The image is downloaded and installed alongside the running one; the running image is not touched. The command also fetches the release’s PGP signature and verifies it — but only LTS releases are signed, so on an unsigned build VyOS prompts Do you want to continue without signature check? (yes/no). That prompt is a decision, not a formality. Answering yes on an image pulled from anywhere other than a source you control is how a router ends up running somebody else’s firmware.

add system image also accepts latest, a vrf name if the management path is in a VRF, and username/password for an authenticated mirror.

Step 3: set the default boot

The operator sets the default boot to the new image:

show system image
set system image default-boot 1.5.1

Both of these are operational-mode commands. There is no configure and no commit here — the boot slot is not part of the configuration tree, which is precisely why it survives a configuration rollback and why rollback will not undo it. Run show system image first: it prints the installed images and marks the current default, so you set the boot slot by the name the system actually uses rather than by the version string you expected.

The previously running image stays installed and remains selectable, which is the whole basis of the rollback plan below.

Step 4: reboot

The operator reboots the router:

reboot

The router reboots into the new image. The reboot takes 2-5 minutes. During the reboot, the other router carries the traffic.

Step 5: verify and restore traffic

After the reboot, the operator verifies that the new image is working correctly:

# Confirm which image actually booted
show version
show system image

# Verify the routing protocols are up
show ip ospf neighbor
show bgp ipv4 summary

# Verify the routes are installed, and that there are as many
# as before — a session that is Established but carrying a
# fraction of the prefixes is a policy problem, not a win
show ip route
show ip route summary

# Verify VRRP is where you expect it
show vrrp

# Verify traffic flows to a destination you know was reachable
# before the upgrade
ping 198.51.100.1 count 4

Check show version first. An upgrade that silently booted the old image looks identical to a successful one from every routing command in that list, and every minute spent verifying the wrong image is a minute you will have to spend again.

If the verification passes, the operator restores traffic to the upgraded router:

configure
# Restore VRRP priority (VRRP topology)
set high-availability vrrp group LAN priority 100

# Remove the BGP drain: unbind the route-maps, then delete them
delete protocols bgp neighbor 10.0.0.1 address-family ipv4-unicast route-map import DRAIN-IN
delete protocols bgp neighbor 10.0.0.1 address-family ipv4-unicast route-map export DRAIN-OUT
delete policy route-map DRAIN-IN
delete policy route-map DRAIN-OUT
commit
save

Delete the route-maps as well as the bindings. A DRAIN route-map left defined but unbound is harmless today and is exactly what the next operator will bind to the wrong neighbour at three in the morning.

The upgraded router is now back in service, running the new image. The operator proceeds to upgrade the next router.

Failure modes

Second router upgrade fails

The operator upgrades edge-01 successfully and proceeds to edge-02. Something about the new image does not work on edge-02, and the operator has to put it back.

Be clear about what “put it back” involves, because this is where the rolling upgrade is most often described wrongly. VyOS does not roll the image back by itself. Keeping the previous image installed is what makes a rollback possible; it is not a rollback. Somebody has to run set system image default-boot <previous> and reboot again. If the new image will not boot far enough to reach a login prompt, “somebody” needs console or IPMI access to pick the previous entry from the GRUB menu — which is why an out-of-band path to each router is a prerequisite of this procedure and not a nice-to-have.

Diagnostic: show version and show system image on edge-02 name the running image and the default boot entry. Compare them with edge-01.

Fix: roll edge-02 back deliberately, then investigate with both routers on the same known-good image.

Drain takes longer than expected

The operator drains traffic from edge-01 and proceeds to reboot. The drain has not actually finished — BGP convergence on the far side is still in progress, or the VRRP group did not preempt — so edge-01 was still carrying production traffic when it went down.

Diagnostic: the honest one is on the other router, and it is a comparison rather than a reading. Take ip -s link show eth0 on edge-02 before the drain and again a minute after it, and check that the received-byte counter has climbed by roughly edge-01’s share. show vrrp should show edge-02 as master before you reboot edge-01, not after.

Fix: wait for the counters to move, not for a number of minutes to elapse. The drain is complete when the other router’s traffic has gone up, and there is no substitute observation for that.

Verification fails

The operator upgrades edge-01 and reboots. The image boots but the routing protocols do not come up.

Diagnostic: show ip ospf neighbor empty; show bgp ipv4 summary showing peers in Idle or Active. Before reaching for a rollback, check show version — a router that booted the old image because the default-boot slot was never set produces different symptoms but the same initial panic, and it is a two-second thing to rule out. Then read the release notes for the configuration migrations that ran during the first boot: VyOS rewrites the stored configuration to the new syntax as part of the upgrade, and a migration that could not translate a construct is the usual reason a protocol comes back up with less configuration than it had.

Fix: diff the migrated configuration against the show configuration commands output you saved before the upgrade. Save it off the box — VyOS keeps a configuration per image, so the pre-upgrade revision history is not reliably the thing you are now looking at. That diff is the answer most of the time, and it exists only if you took the capture first.

Mixed-version state

The operator upgrades edge-01 successfully but cannot upgrade edge-02 (the new image fails on edge-02). The operator has a mixed-version state: edge-01 running the new image; edge-02 running the previous image. The mixed-version state may have interop issues (e.g. BGP session works but OSPF session does not).

Fix: roll back edge-01 to the previous image to restore the same-version state. Investigate the edge-02 failure; retry the upgrade.

Rollback

The rolling upgrade is reversible because the previous image is still installed. Reversing it is the same procedure run backwards, one router at a time:

show system image
set system image default-boot 1.4.2
reboot

Three things to be honest about:

• It is manual. Nothing detects a bad image and reverts it. The operator decides, sets the boot slot, and reboots. If the router will not boot far enough to reach a prompt, this is done from the GRUB menu over console or IPMI — so out-of-band access to every router is a prerequisite of the procedure, not an optional extra.
• It is a reboot per router, so the rollback needs the same drain, verify, restore discipline the upgrade did. Rolling both routers back at once turns a degraded network into a down one.
• rollback <N> is a different thing entirely. That command reverts the configuration to a stored revision, and on VyOS it currently requires a reboot to take effect. It has no bearing on which image boots. Reaching for it during an image rollback is a category error that costs an extra reboot at the worst possible moment.

The mixed-version state is the intermediate state during both the upgrade and the rollback: one router on each image, the other carrying the traffic. It is a normal place to be for the length of one verification, and a bad place to camp overnight.

Production discipline

Cross-course references

• LVI-VyOS-Upgrades (vyos-lvi-01-image-management, the previous lesson) covers the dual-image model that the rolling upgrade uses.
• XL-VyOS-VRRP (vyos-xl-01-vrrp-concept) covers the VRRP mechanism that the rolling upgrade may use for traffic draining.
• LVI-VyOS-Upgrades (vyos-lvi-04-upgrade-validation) covers the validation procedure that the rolling upgrade follows.

Quiz