Lab: VRRP High Availability

Objective

By the end of this lab two routers will share one gateway address, a client will keep working when the router holding that address is powered off, and you will have a number — in lost packets, measured from the client — for how long “keep working” actually took.

Between those two states sits the finding this lab is really built around. Your first failover will succeed by every measure VRRP offers: the surviving router will report itself master, the virtual address will be on its interface, and the client’s traffic will still not work. VRRP will have done its whole job correctly and the service will still be down, because first-hop redundancy is redundancy for one hop and somebody has to own the other one.

Architecture

Four VyOS nodes: two routers forming the VRRP pair, one client on the LAN they protect, and one upstream node standing in for everything beyond them.

                    up
                    eth1 203.0.113.1/24
                    dum0 198.51.100.9/32   <- the destination we test against
                            |
                       vmbr93 — uplink segment, 203.0.113.0/24
                        /            \
        eth2 203.0.113.2/24        eth2 203.0.113.3/24
     +--------------------+      +--------------------+
     |         r1         |      |         r2         |
     |   priority 200     |      |   priority 100     |
     +--------------------+      +--------------------+
        eth1 192.168.20.2/24       eth1 192.168.20.3/24
                        \            /
                       vmbr94 — LAN segment, 192.168.20.0/24
                            |
                    client  192.168.20.50/24
                            default route via 192.168.20.1

     VRRP group LAN : vrid 20, virtual address 192.168.20.1/24
     VRRP group WAN : vrid 30, virtual address 203.0.113.10/24  (added in Task 5)

     every node also has eth0 on vmbr0 — your management LAN, untouched

vmbr93 and vmbr94 are isolated bridges with no physical uplink, so nothing in this lab can reach anything real. vmbr0 carries management only, and no step here touches it — which is what makes it safe to power off a router mid-lab.

Addresses come from the RFC 5737 documentation ranges on the routed side and from RFC 1918 space on the protected LAN, which is what a LAN behind a redundant gateway pair usually is.

Requirements

• A hypervisor able to run four VMs with two or three NICs each. The commands are written for Proxmox VE; any KVM/libvirt host works.
• 1 GB RAM and 8 GB disk per node — 4 GB and 32 GB in total.
• The VyOS 1.5 LTS ISO uploaded to hypervisor storage.
• Console access to all four VMs. Task 5 powers a router off from the hypervisor, which is the point; Task 7 disables an interface on a live router. Neither touches management, but keep the console available.
• Roughly two hours, of which the VM build is about 40 minutes.

Scenario

A site has one router, r1, and one client network behind it. The router has been a single point of failure for two years and has finally been funded a partner, r2. The requirement as written on the ticket is “make the gateway redundant”. The requirement as it will be judged is “a user on the LAN does not notice when we reboot a router”.

Nothing is configured yet. You will build the starting state.

Tasks

Task 1: Build the starting state

Two isolated bridges first. On the Proxmox host, add two Linux bridges with no ports — no ports is what makes them isolated. This is config text for /etc/network/interfaces, not a command:

auto vmbr93
iface vmbr93 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0

auto vmbr94
iface vmbr94 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0

Apply with ifreload -a, or reboot the node if your host has no ifupdown2. Then build the VMs. The two routers get three NICs; the client and the upstream node get two:

# Run on the Proxmox host.
# Substitute your own values before running. The ISO volume ID must match
# exactly what `pvesm list local` prints for your upload.
ISO=local:iso/vyos-1.5-lts-amd64.iso
STORE=local-lvm

for VMID in 270 271; do
  qm create "$VMID" --memory 1024 --cores 1 \
    --net0 "virtio,bridge=vmbr0,firewall=0" \
    --net1 "virtio,bridge=vmbr94,firewall=0" \
    --net2 "virtio,bridge=vmbr93,firewall=0" \
    --scsihw virtio-scsi-single --scsi0 "$STORE:8" \
    --ide2 "$ISO,media=cdrom" --boot order=ide2 --ostype l26
done

qm create 272 --name client --memory 1024 --cores 1 \
  --net0 "virtio,bridge=vmbr0,firewall=0" \
  --net1 "virtio,bridge=vmbr94,firewall=0" \
  --scsihw virtio-scsi-single --scsi0 "$STORE:8" \
  --ide2 "$ISO,media=cdrom" --boot order=ide2 --ostype l26

qm create 273 --name up --memory 1024 --cores 1 \
  --net0 "virtio,bridge=vmbr0,firewall=0" \
  --net1 "virtio,bridge=vmbr93,firewall=0" \
  --scsihw virtio-scsi-single --scsi0 "$STORE:8" \
  --ide2 "$ISO,media=cdrom" --boot order=ide2 --ostype l26

qm set 270 --name r1
qm set 271 --name r2

for VMID in 270 271 272 273; do qm start "$VMID"; done

firewall=0 on every NIC matters here for a reason specific to this lab: VRRP advertisements are sent to the multicast address 224.0.0.18 as IP protocol 112, and a host-side filter that quietly drops them produces two routers that each believe the other is dead. That is the split-brain failure mode, arriving from outside the guest, and it is very hard to diagnose from inside.

Install on each VM from the console — log in with the live-boot credentials, run install image, accept the defaults — then move the boot order back to disk and detach the ISO:

# Run on the Proxmox host, after `install image` completes on all four VMs.
for VMID in 270 271 272 273; do
  qm set "$VMID" --boot order=scsi0
  qm set "$VMID" --delete ide2
done

Task 2: Baseline, and a path that works with no redundancy at all

Configure the four nodes. On r1:

configure
set system host-name r1
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth0 description "management - not part of the lab"
set interfaces ethernet eth1 address 192.168.20.2/24
set interfaces ethernet eth1 description "LAN - protected segment"
set interfaces ethernet eth2 address 203.0.113.2/24
set interfaces ethernet eth2 description "uplink"
set protocols static route 198.51.100.0/24 next-hop 203.0.113.1
set service ssh
commit
save

On r2, the same with its own addresses:

configure
set system host-name r2
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth0 description "management - not part of the lab"
set interfaces ethernet eth1 address 192.168.20.3/24
set interfaces ethernet eth1 description "LAN - protected segment"
set interfaces ethernet eth2 address 203.0.113.3/24
set interfaces ethernet eth2 description "uplink"
set protocols static route 198.51.100.0/24 next-hop 203.0.113.1
set service ssh
commit
save

On up. Note the single static route back towards the LAN, pointing at r1 and only at r1 — this is deliberate, and it is the most realistic detail in the whole lab:

configure
set system host-name up
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth1 address 203.0.113.1/24
set interfaces dummy dum0 address 198.51.100.9/32
set interfaces dummy dum0 description "destination under test"
set protocols static route 192.168.20.0/24 next-hop 203.0.113.2
set service ssh
commit
save

On client, with r1 as its gateway for now:

configure
set system host-name client
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth1 address 192.168.20.50/24
set protocols static route 0.0.0.0/0 next-hop 192.168.20.2
set service ssh
commit
save

Save a named baseline on all four nodes; Cleanup restores from it:

save /config/pre-lab.boot

Prove the path before adding anything to it:

$ ping 198.51.100.9 count 3

PING 198.51.100.9 (198.51.100.9) 56(84) bytes of data.
64 bytes from 198.51.100.9: icmp_seq=1 ttl=63 time=0.702 ms
64 bytes from 198.51.100.9: icmp_seq=2 ttl=63 time=0.488 ms
64 bytes from 198.51.100.9: icmp_seq=3 ttl=63 time=0.503 ms

--- 198.51.100.9 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2042ms

This is the state the site is in today: one working path, entirely dependent on r1. Everything from here is about what happens when r1 stops.

Task 3: One virtual address, and three independent ways to see it

Configure the LAN-side group on r1:

configure
set high-availability vrrp group LAN interface eth1
set high-availability vrrp group LAN vrid 20
set high-availability vrrp group LAN address 192.168.20.1/24
set high-availability vrrp group LAN priority 200
set high-availability vrrp global-parameters version 3
commit-confirm 5

And on r2, identical except for the priority:

configure
set high-availability vrrp group LAN interface eth1
set high-availability vrrp group LAN vrid 20
set high-availability vrrp group LAN address 192.168.20.1/24
set high-availability vrrp group LAN priority 100
set high-availability vrrp global-parameters version 3
commit-confirm 5

Type confirm on both. commit-confirm 5 cannot save you from anything in this task, because management is on eth0 — but the habit belongs in your fingers before the task that powers a router off.

The group name LAN is a local label. The identifier that appears on the wire and that both routers must agree on is vrid 20, and the two are completely independent: two routers with different group names and the same VRID form one virtual router, while two routers with the same group name and different VRIDs form two.

Now read the state from three places that do not share a source.

The operational view — what keepalived believes.

$ show vrrp

Name        Interface      VRID  State    Last Transition
----------  -----------  ------  -------  -----------------
LAN         eth1             20  MASTER   41s

$ show vrrp

Name        Interface      VRID  State    Last Transition
----------  -----------  ------  -------  -----------------
LAN         eth1             20  BACKUP   39s

Exactly one MASTER. Two would be split brain; zero means the group never initialised, usually because the interface named in the configuration is not the one carrying the segment.

The interface view — where the address actually is. This one is worth more than the operational command, because it is not keepalived reporting on itself:

$ ip -br addr show eth1

eth1             UP             192.168.20.2/24 192.168.20.1/24

Run the same command on r2 and the virtual address is absent. That is the mechanism in one line: the master holds the address, the backup does not, and the transition is an address moving between two machines.

The wire view — the advertisements. Capture on the LAN segment from r2, which should be hearing r1 and saying nothing itself:

$ sudo tcpdump -ni eth1 'ip proto 112' -c 4 -v

12:41:02.118432 IP (tos 0xc0, ttl 255, id 0, offset 0, flags [none], proto VRRP (112), length 40)
  192.168.20.2 > 224.0.0.18: VRRPv3, Advertisement, vrid 20, prio 200, intvl 100cs, length 20
12:41:03.119004 IP (tos 0xc0, ttl 255, id 0, offset 0, flags [none], proto VRRP (112), length 40)
  192.168.20.2 > 224.0.0.18: VRRPv3, Advertisement, vrid 20, prio 200, intvl 100cs, length 20

One source, sending once per second. The source is r1’s real address, not the virtual one — the advertisement is a statement by a physical router about a virtual one. ttl 255 is required by the protocol and is how a receiver knows the advertisement was not forwarded from another subnet.

And the question almost nobody asks. The client has not spoken to the virtual address yet, so give it a reason to resolve it — ping 192.168.20.1 count 2 from the client — and then look at what it holds:

$ ip neigh show dev eth1

192.168.20.2 dev eth1 lladdr 52:54:00:1a:2b:3c REACHABLE
192.168.20.1 dev eth1 lladdr 52:54:00:1a:2b:3c REACHABLE

Write both lines down. The virtual address and r1’s real address resolve to the same MAC — r1’s own interface MAC. The RFC 3768 virtual MAC in the 00:00:5e:00:01:xx range is not in use here, because VyOS does not enable it by default. Task 8 is where that matters.

Task 4: Move the client onto the virtual address

Point the client at the gateway it is supposed to use:

configure
delete protocols static route 0.0.0.0/0 next-hop 192.168.20.2
set protocols static route 0.0.0.0/0 next-hop 192.168.20.1
commit
save

Re-run the ping from Task 2. It succeeds, identically, because r1 is master and r1 was already the path. Nothing has been proved yet except that the virtual address forwards — which is worth confirming, since a virtual address that answers ARP but does not route is a real misconfiguration and it looks healthy from show vrrp.

Task 5: Fail the master, measure the outage, and find the half that is still broken

Start a long ping on the client and leave it running for the whole task:

$ ping 198.51.100.9 count 120

Now kill r1 the way a power supply would.

# Run on the Proxmox host, after confirming the ID with `qm list`.
qm stop 270

Watch the ping. It stops. Note the sequence number it stopped at — you will need it — and then keep watching, because what makes this task worth doing is what does not happen next.

Then look at what r2 believes:

$ show vrrp

Name        Interface      VRID  State    Last Transition
----------  -----------  ------  -------  -----------------
LAN         eth1             20  MASTER   14s

$ ip -br addr show eth1

eth1             UP             192.168.20.3/24 192.168.20.1/24

$ ip neigh show dev eth1

192.168.20.1 dev eth1 lladdr 52:54:00:4d:5e:6f REACHABLE

Every one of those is healthy. The state moved in about a second, the address moved with it, and the client’s ARP cache followed without waiting for a timeout, because the new master sent gratuitous ARP the moment it took over. VRRP did its entire job.

And the ping never resumes. Find out why before reading on — the evidence is two captures.

$ sudo tcpdump -ni eth2 icmp -c 4

12:58:11.402118 IP 192.168.20.50 > 198.51.100.9: ICMP echo request, id 3312, seq 44, length 64
12:58:12.403441 IP 192.168.20.50 > 198.51.100.9: ICMP echo request, id 3312, seq 45, length 64

They are. Requests only — no replies. Now the far end:

$ sudo tcpdump -ni eth1 icmp -c 4

12:58:11.402490 IP 192.168.20.50 > 198.51.100.9: ICMP echo request, id 3312, seq 44, length 64
12:58:11.402612 IP 198.51.100.9 > 192.168.20.50: ICMP echo reply, id 3312, seq 44, length 64

The reply is built and sent. It is sent to 192.168.20.50 via up’s only route towards that network — next-hop 203.0.113.2, which is r1, which is switched off. up never had an opinion about VRRP and nothing told it the topology changed.

Fix it with a second virtual address on the uplink side, and make the two groups move together. Bring r1 back first — qm start 270 on the Proxmox host — and wait for show vrrp on both routers to settle before configuring anything.

On r1:

configure
set high-availability vrrp group WAN interface eth2
set high-availability vrrp group WAN vrid 30
set high-availability vrrp group WAN address 203.0.113.10/24
set high-availability vrrp group WAN priority 200
set high-availability vrrp sync-group SITE member LAN
set high-availability vrrp sync-group SITE member WAN
commit
save

On r2, identical except for the priority, which is the same way round as the LAN group:

configure
set high-availability vrrp group WAN interface eth2
set high-availability vrrp group WAN vrid 30
set high-availability vrrp group WAN address 203.0.113.10/24
set high-availability vrrp group WAN priority 100
set high-availability vrrp sync-group SITE member LAN
set high-availability vrrp sync-group SITE member WAN
commit
save

Then repoint the upstream at the virtual address rather than at a machine. On up:

configure
delete protocols static route 192.168.20.0/24 next-hop 203.0.113.2
set protocols static route 192.168.20.0/24 next-hop 203.0.113.10
commit
save

Confirm show vrrp on r1 lists both groups as MASTER and on r2 lists both as BACKUP — a sync-group whose members disagree is a sync-group that is not working. Then repeat the whole failure: start the long ping on the client, qm stop 270, and count the lost packets. This time the ping resumes, and the number you record is the one that answers the ticket.

Task 6: Preemption, and the second outage nobody scheduled

r1 is still off after the second failover test. Start it with qm start 270, and watch the ping on the client while it boots.

There is a second gap. When r1 comes back it advertises priority 200, r2 sees a higher priority and steps down, and the address moves again — so a single hardware failure produced two interruptions: one when the router died and one when it came back, during working hours, unannounced, while somebody was already looking at the first one.

That is the behaviour to decide about deliberately rather than inherit. Turn it off on both routers:

configure
set high-availability vrrp group LAN no-preempt
set high-availability vrrp group WAN no-preempt
commit
save

With no-preempt, a recovered r1 stays backup until r2 fails. The roles are now “whoever last survived” rather than “whichever we designated” — which is fine, and is the point, provided your monitoring alerts on which router is master rather than assuming.

The middle option is a delay: preempt, but not immediately. It is an alternative to no-preempt, not a companion to it, so remove that first:

delete high-availability vrrp group LAN no-preempt
set high-availability vrrp group LAN preempt-delay 180

Three minutes, in seconds. The delay exists because a router that has just booted is not ready to forward: interfaces come up before routing protocols converge, and a switch port that has just come up may still be in a spanning-tree listening state. A router that takes the virtual address in that window is a black hole with a healthy show vrrp.

Task 7: Tracking, or what to do about an uplink that is up and useless

Both routers are currently willing to be master with a dead uplink. The LAN group watches eth1 and knows nothing about eth2, so a failure of r1’s uplink alone leaves r1 holding the gateway address and dropping everything.

Prove it first. Check show vrrp on both routers and work on whichever one currently holds MASTER — after Task 6 that may not be r1, which is itself the point of no-preempt. Start a ping on the client so you can see the effect, then on the master:

configure
set interfaces ethernet eth2 disable
commit

The client’s ping stops and stays stopped. show vrrp on that router still reports MASTER for the LAN group, because from VRRP’s point of view nothing has happened: the interface it was told to watch is still up, and it was never told to watch the other one.

Re-enable the interface, then add the link between the two facts on both routers:

configure
set high-availability vrrp group LAN track interface eth2
set high-availability vrrp group WAN track interface eth1
commit
save

Re-run the failure on the master and watch the state:

$ show vrrp

Name        Interface      VRID  State    Last Transition
----------  -----------  ------  -------  -----------------
LAN         eth1             20  FAULT    6s
WAN         eth2             30  FAULT    6s

Re-enable the interface with delete interfaces ethernet eth2 disable and commit, and confirm the groups return to the election.

Task 8: The virtual MAC, and what your client is really holding

Return to the ARP entries from Task 3. On VyOS the virtual address is answered with the master’s own interface MAC by default, so a failover changes the MAC behind the gateway address and relies on gratuitous ARP to tell every host on the segment. That works, and it has two costs: hosts that ignore gratuitous ARP keep the old MAC until their cache ages out, and the LAN switch has to relearn which port the MAC lives on.

RFC 3768 defines a virtual MAC per virtual router precisely to avoid this. Enable it on both routers, on the LAN group:

configure
set high-availability vrrp group LAN rfc3768-compatibility
commit
save

VyOS creates a dedicated interface for the group, named for the parent interface and the VRID:

$ show interfaces ethernet eth1v20

eth1v20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
  link/ether 00:00:5e:00:01:14 brd ff:ff:ff:ff:ff:ff
  inet 192.168.20.1/24 scope global eth1v20

00:00:5e:00:01:14 is not arbitrary: 00:00:5e is the IANA-reserved OUI, 00:01 marks IPv4 VRRP, and 14 is 20 in hexadecimal — your VRID. A group with VRID 30 would end 1e. This is why a real NIC can never collide with it.

Fail the master again — qm stop against whichever VM currently holds it, confirmed with qm list — and re-read the client’s neighbour table. The lladdr for 192.168.20.1 is the same value before and after, because this time the address and the MAC moved together and the client had nothing to relearn. Start the VM again before moving on to Cleanup.

Production notes — running this as a change

Build it in the order that cannot break the working state. Configure the backup first, with the low priority, and commit. It will sit in BACKUP and change nothing, because the address it wants is already answered by a router that is not running VRRP at all. Then configure the intended master. Then, and only then, move clients onto the virtual address. The last step is the one users notice and it is a single-line change on the client side, which is also the rollback.

The change window is the client cutover, not the VRRP configuration. Adding a group to a router with the right priority is non-disruptive. Repointing a default gateway is not, and neither is the first real failover test — which you must schedule, because a redundancy pair that has never failed over is a redundancy pair that has never been tested.

Test it in both directions and from the client. The evidence that counts is lost packets on a host, not a state on a router. Run the ping, stop the master, count. Then start it, count again for the preemption event. Two numbers, and they belong in the handover document.

Alert on the state, not just on the routers. Both routers up with both claiming master is an outage that no per-device availability check will see. So is both in FAULT. Monitor which router holds each group and alert on the count of masters being anything other than one.

Validation

• show vrrp on both routers shows exactly one MASTER per group, and the two groups agree on which router that is.
• ip -br addr show eth1 shows 192.168.20.1/24 on the master and not on the backup; the same holds for 203.0.113.10/24 on eth2.
• A capture of ip proto 112 on the LAN segment shows advertisements from one source only, with vrid 20 and the master’s configured priority.
• ping 198.51.100.9 count 3 from the client succeeds with the client’s default route pointing at 192.168.20.1.
• Your journal holds a measured failover: the count of lost sequence numbers from a running ping while the master was powered off, taken both before and after the WAN group and sync-group were added. The first number should be “it never recovered”.
• Your journal holds the Task 5 evidence pair — a healthy show vrrp on r2 alongside a client ping that was still failing — and one sentence naming the mechanism that was missing.
• Disabling eth2 on the master puts both groups into FAULT and the other router takes over; re-enabling returns them to the election.
• The client’s ip neigh entry for 192.168.20.1 changes MAC across a failover before rfc3768-compatibility is enabled, and does not change after.
• A written preemption decision for this topology: which setting, which delay if any, and why.

Expected Outcome

Two routers sharing two virtual addresses in a sync-group, each tracking the other’s interface, with preemption set deliberately. A client whose default route is a virtual address and which survives the loss of either router. An upstream node whose route back into the LAN points at a virtual address rather than at a machine.

All four nodes still hold /config/pre-lab.boot, so the topology resets to its baseline without a rebuild. The journal holds every capture named in Validation, plus two failover measurements taken from the client.

Troubleshooting

Both routers report MASTER. They are not hearing each other. Capture ip proto 112 on both LAN interfaces: if each sees only its own advertisements, the segment is not carrying multicast between them — check that both eth1 interfaces are on the same bridge and that firewall=0 was set on the NICs. If each sees the other’s advertisements and both are still master, check that the vrid matches and that global-parameters version is the same on both.

No router reports MASTER, or show vrrp is empty. The group never initialised. The usual cause is an interface name that does not carry the segment; the second is a group whose address is not in a subnet configured on that interface.

The failover works but takes far longer than the ping suggests it should. Look at what is re-learning rather than at VRRP. Without rfc3768-compatibility the switch must move a MAC between ports and hosts must accept a gratuitous ARP; a host that ignores gratuitous ARP waits for its own cache to expire, which is minutes.

Traffic works from the LAN but not back. This is Task 5. Check what the upstream device has as its route towards the LAN, and whether that next hop is a virtual address or a machine.

Both groups sit in FAULT. A tracked interface is down on both routers, so both have withdrawn. show interfaces on each will name it. This is the failure mode of tracking too much.

set is rejected with an unknown-node error. You are using the 1.3 spelling — virtual-address, or preemption. See the release note at the top of this lab and check show version.

Cleanup

Everything here is on isolated bridges and management was never touched, so cleanup is about leaving a reproducible state.

Step 1. Confirm both routers are running, then on each of the four nodes load the baseline saved in Task 2:

configure
load /config/pre-lab.boot
compare
commit
save

compare before commit shows exactly what loading the file will change. An empty comparison means you are already at the baseline.

Step 2. Confirm the virtual addresses are gone — ip -br addr show eth1 on both routers should show only the real address — and that the client is back on a static next hop of 192.168.20.2.

Step 3. Remove the VMs, then the bridges.

# Run on the Proxmox host, after confirming these IDs with `qm list`.
for VMID in 270 271 272 273; do
  qm stop "$VMID"
  qm destroy "$VMID" --purge
done

Then remove the vmbr93 and vmbr94 stanzas from /etc/network/interfaces and run ifreload -a. Confirm with ip link show vmbr93, which should report that the device does not exist.

What You Learned

• VRRP state is an address, and the address is the honest check. show vrrp is keepalived describing itself; ip -br addr show is the kernel. You used both, and they agree until the interesting moment.
• A successful failover can leave the service down. Task 5 produced a healthy master, a moved address, an updated ARP cache and a broken ping, because the return path pointed at a machine rather than at a virtual address.
• A sync-group is a decision to fail together. It trades a working interface on a partly-failed router for traffic that stays on one machine, which is the trade any stateful device in the path requires.
• Preemption schedules a second outage. You measured it. Whether that is worth paying depends on whether the two routers are equivalent, and the answer belongs in writing.
• Tracking on VyOS 1.5 is a withdrawal, not a decrement. A tracked interface that is down means FAULT, so tracking too much takes the gateway away from a LAN that was still working.
• The virtual MAC is not on by default. Without rfc3768-compatibility, failover changes the MAC behind the gateway address and depends on every host believing a gratuitous ARP.