Skip to main content
RunBook Academy

← All runbooks in VyOS

high riskservice affecting~60 min

Runbook: Configure VRRP

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Both routers are on the same Layer 2 segment for the interface VRRP will run on. VRRP is multicast to 224.0.0.18 as IP protocol 112, and it does not cross a routed boundary
  • · The segment and both local firewalls pass IP protocol 112 multicast. A path that drops it produces two masters, not a failed commit
  • · The virtual address is not already configured as a real address on either router, unless IP-owner mode with priority 255 is the deliberate design
  • · Priorities are decided and different, and the gap is larger than any tracking decrement that will be applied to the higher one
  • · The preemption decision is made and written in the ticket: enabled returns the pair to its designed state after a recovery, disabled leaves the survivor as master
  • · The tracking decision is made. A master with a dead uplink stays master and blackholes the LAN unless something decrements its priority
  • · The VRRP version is agreed and identical on both routers. A v2 and a v3 router on one segment are two masters
  • · Baseline captured from both routers: show vrrp, show vrrp detail, show configuration commands | grep vrrp, show interfaces, and the current default gateway the LAN hosts are using
  • · The failover test is scheduled inside this window, with an agreed abort condition. A pair that has never failed over has not been verified

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Confirm out-of-band access to both routers and keep both console sessions open. A VRRP mistake can take the LAN gateway down for every host on the segment.
  2. 2Capture the baseline on both routers, and record what the LAN hosts currently use as their default gateway.
  3. 3Confirm the virtual address is free: it must not appear as a real interface address on either router, and nothing on the segment may already answer ARP for it.
  4. 4On the intended master, enter configure and build the group: interface, virtual-address, priority, advertisement-interval, version, and preemption per the ticket.
  5. 5Add tracking on the intended master if the ticket calls for it: set high-availability vrrp group 1 track interface eth1 priority-decrement 150.
  6. 6Run compare on the intended master and have the peer reviewer read it. Confirm the group id, the virtual address and the version.
  7. 7Commit the intended master first with commit-confirm 10, then exit. Committing the backup first makes it master by default, because nothing else is advertising yet.
  8. 8Verify on the master: show vrrp reports Master, and the virtual MAC follows the 00:00:5E:00:01:XX pattern with the group id in the last byte.
  9. 9Build the same group on the backup with the lower priority and identical interface, virtual-address, version and advertisement-interval. Apply with commit-confirm 10.
  10. 10Verify the pair: the master reports Master and the backup reports Backup. Two Masters is a stop condition, not something to tune around.
  11. 11Read the committed timers back with show vrrp detail and confirm the daemon reports the values you intended. Timer units are the commonest place for a VRRP change to be quietly wrong.
  12. 12Confirm tracking is loaded and its objects are up: show vrrp track.
  13. 13Confirm the LAN can use the virtual address: from a host on the segment, resolve the virtual IP and check its ARP entry is the virtual MAC, then reach something beyond the router through it.
  14. 14Run the failover test by failing the tracked uplink from the far end or physically. Do not induce it with a second commit while a commit-confirm timer is still pending on this router. Confirm the backup takes over, and time the recovery.
  15. 15Restore the uplink and confirm the pair settles into the state the preemption decision says it should.
  16. 16If everything matches: confirm and save on both routers. A confirmed master with an unconfirmed backup is a pair that reverts to a single router when the timer expires.
  17. 17If anything does not match: capture show vrrp detail and a protocol-112 capture from the segment, then let the timers expire on both routers.
  18. 18If the failover could not be induced from outside, run it after both routers are confirmed and saved. With no timer pending, set interfaces ethernet eth1 disable and commit is a safe way to fail the tracked uplink, and deleting the disable restores it.

4 · Verification

Confirm the procedure actually fixed the problem.

  • show vrrp reports Master on exactly one router and Backup on the other. Two Masters means the advertisements are not crossing the segment
  • show vrrp detail on both routers reports the advertisement interval, priority and preemption values that were intended, not the defaults and not a different unit
  • The virtual MAC seen on the segment matches 00:00:5E:00:01:XX, where the last byte is the group id in hex
  • show vrrp track lists every tracked object with its state, and the effective priority reflects any decrement that is currently applied
  • A LAN host has an ARP entry mapping the virtual IP to the virtual MAC, and reaches a destination beyond the router through it
  • A capture of IP protocol 112 on the segment shows advertisements from exactly one source address
  • The failover test moved the LAN to the backup, and the recovery time was measured and recorded in the ticket
  • After the master is restored, the pair settles into the state the preemption decision predicts
  • systemctl status keepalived is active on both routers, and /var/log/keepalived.log shows the transitions the test produced and no others
  • Both routers have been confirm-ed and save-d

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • Preferred path on either router: do not run confirm. Let the commit-confirm timer expire and that router restores its previous configuration on its own.
  • Roll back the backup first, then the master. Rolling back the master first hands the virtual address to a backup you are about to remove.
  • Explicit path: delete high-availability vrrp group 1 then commit inside configure.
  • Removing VRRP removes the virtual address from the segment. Every host using it as a default gateway loses its gateway at that moment, so the LAN must be pointed back at a real router address as part of the same change.
  • If the pair is in split brain, roll back to a single router rather than adjusting priorities. One working gateway beats two routers arguing about which is the gateway.
  • Capture show vrrp detail from both routers and a protocol-112 capture from the segment before rolling back. After the rollback there is nothing left to diagnose.
  • After rollback, confirm on both routers that show vrrp reports no groups and that the virtual address is no longer answered on the segment.
  • Check the LAN hosts' ARP caches. A host that cached the virtual MAC keeps sending to it until the entry ages out, so recovery is not instant even after the routers are clean.

6 · Escalation

When the runbook isn't enough, contact:

  • · Escalate to network engineering when both routers report Master. That is a Layer 2 problem — filtered multicast, a VLAN mismatch, or a switch that is not forwarding protocol 112 — and it is not fixed from the routers.
  • · Escalate to network engineering if failover works but traffic does not recover. The likely cause is asymmetric upstream paths and a stateful firewall that has no session for the new path, and that is a design question, not a VRRP setting.
  • · Escalate to the LAN owner before the failover test if the segment carries traffic that cannot absorb a few seconds of interruption. The test is service-affecting by design.
  • · Escalate to the platform team if a router stops answering on both the data and the management path. Recover from the console sessions opened before the change.
  • · Escalate to the network architect if the pair flaps repeatedly after a recovery. Repeated transitions usually mean preemption is fighting a slow-converging switch port, and the fix is a preemption-delay decision rather than a priority change.
  • · Holding is a legitimate outcome: let both timers expire, leave the LAN on its previous single gateway, and return with the Layer 2 evidence that was missing.

VRRP is the only change in this toolkit where doing half of it produces something that looks finished. Configure one router, and you get a working virtual gateway address: hosts resolve it, traffic flows, show vrrp says Master. What you do not get is redundancy — and nothing on the page says so. The pair is the unit of work here, and the runbook is written for two operators on two routers in one window.

The second thing to hold onto: VRRP protects the host-to-router hop and nothing else. It moves a gateway address between two routers on the same segment. It does not choose a path, it does not know whether the router that wins the election can actually reach the Internet, and it will cheerfully keep a master that has lost its uplink. Making it react to that is the tracking decision, and it is the difference between VRRP as redundancy and VRRP as decoration.

When this runbook applies

  • A LAN segment is getting a redundant default gateway across two VyOS routers for the first time.
  • A second router is being added to a segment that currently has one.
  • An existing VRRP group is being extended with tracking, or having its priority or preemption policy changed.

When it does not apply

  • You need redundancy between routers, not for hosts. VRRP is first-hop redundancy. Path redundancy between routers is what OSPF and BGP are for, and reaching for VRRP there produces a design that fails in ways neither protocol was meant to handle.
  • The two routers are not on the same Layer 2 segment. VRRP advertisements are multicast to 224.0.0.18 with TTL 255 and do not cross a routed boundary. If the routers cannot see each other’s advertisements, both become master.
  • The pair is already up and misbehaving. Two masters, a failover that never fires, or traffic that dies after a failover are diagnoses. Use the VRRP troubleshooting runbook.
  • The virtual address is currently a live gateway on one router. Converting a real gateway address into a virtual one is a higher-risk change than this runbook covers: for a period, two things on the segment want to answer for the same IP.

Blast radius

Every host on the segment that uses this address as its default gateway.

That is the number for the ticket. Not “two routers” — the count of hosts whose off-network traffic stops if this goes wrong. It is also the reason the failover test is not optional and not free: the test is a deliberate, short outage for all of them, and it needs the LAN owner’s agreement before the window, not during it.

The three decisions to make before the window

Priority, with the tracking decrement in mind

The higher priority wins the election; ties break on the higher IP address. The default is 100, and a typical pair uses 200 and 100.

The number that actually matters is the effective priority once tracking applies. If the master is 200 and tracks its uplink with a decrement of 150, its effective priority when that uplink dies is 50 — comfortably below the backup’s 100, which is the point. Choose the decrement so the result genuinely crosses the backup’s priority; a decrement that leaves the master still on top is a tracking configuration that does nothing at all.

Work out what happens when both tracked objects fail, too. The priorities collapse together, the election falls back to the tie-break, and whichever router wins has no working uplink either. That is not a bug you can configure away — it is the honest answer that the LAN has lost both paths — but you should know which router you will find as master when it happens.

Preemption: return to the design, or stay put

Enabled (the default), a recovered master takes the role back, which returns the pair to its designed state at the cost of a second short interruption. Disabled, the survivor stays master and the pair is left in the state the failure produced.

The argument that settles it is upstream symmetry. If both routers reach the outside world the same way, preemption is harmless and returning to the design is worth having. If each router has its own upstream and there is a stateful firewall anywhere in the return path, preemption moves live sessions to a path that has no state for them, and the recovery of the master becomes a second outage. In that estate, disable it.

Tracking: what should cost this router the master role

Without tracking, a master whose uplink has failed keeps advertising at full priority and keeps attracting all the LAN’s traffic into a blackhole. Interface tracking is the cleanest form because a port that goes down is an unambiguous event. Route tracking suits an upstream that is learned dynamically, where the failure shows up as a route disappearing rather than a link dropping. Remote-IP tracking is for where neither is true, and it needs a threshold so a single lost probe does not move the gateway.

Prove the way back in, on both routers

Read-only / Safeon each console path
show version
show interfaces

Two routers, two console sessions, both open for the whole window. A VRRP change that goes wrong does not take one router away — it takes the segment’s gateway away.

Capture the baseline, including what the hosts use today

Read-only / Safeon both routers
show vrrp
show vrrp detail
show configuration commands | grep vrrp
show interfaces

Then record, from the LAN side, what the hosts currently use as their default gateway. This is the thing you have to be able to put back, and it is not visible from the router.

Confirm the virtual address is free. Nothing on the segment may already answer ARP for it, and it must not be a real address on either router — unless IP-owner mode is the deliberate design, in which case the router that owns it takes priority 255 and always wins. Priority 255 on a router that does not hold the address as a real interface address is the misconfiguration to avoid: the router insists on being master without being able to own the address, and the result is erratic transitions.

Build the group on the intended master

Configuration changeintended master
configure

set high-availability vrrp group 1 interface eth0
set high-availability vrrp group 1 virtual-address 192.168.1.1/24
set high-availability vrrp group 1 priority 200
set high-availability vrrp group 1 advertisement-interval 1000
set high-availability vrrp group 1 version 3
set high-availability vrrp group 1 preemption delay 0

# React to the uplink, not just to the router being alive
set high-availability vrrp group 1 track interface eth1 priority-decrement 150

compare

Read the compare output for the fields that have to be identical on the backup: the group id, the interface, the virtual address, the version and the advertisement interval. Priority is the only field that is supposed to differ — plus the tracked interface name, if the two routers track different uplinks, which they usually do.

Commit the master first

Service impact possiblemaster, behind a timer
commit-confirm 10
exit

show vrrp
show vrrp detail

Order matters. Whichever router you commit first becomes master, because nothing else is advertising yet. Committing the intended backup first means the pair comes up inverted and then has to transition once the master appears — an extra, avoidable state change on a live segment.

Confirm the virtual MAC while you are here. For IPv4 it is 00:00:5E:00:01:XX, where the last byte is the group id in hex: group 1 is 00:00:5E:00:01:01, group 10 is 00:00:5E:00:01:0A. That OUI is reserved, so it can never collide with a real NIC — which means an ARP entry showing it is unambiguous evidence that the host is talking to the virtual router.

Build the backup

Service impact possiblebackup, behind a timer
configure

set high-availability vrrp group 1 interface eth0
set high-availability vrrp group 1 virtual-address 192.168.1.1/24
set high-availability vrrp group 1 priority 100
set high-availability vrrp group 1 advertisement-interval 1000
set high-availability vrrp group 1 version 3
set high-availability vrrp group 1 preemption delay 0
set high-availability vrrp group 1 track interface eth1 priority-decrement 150

compare
commit-confirm 10
exit

Read back what the daemon actually has

Read-only / Safeverify the pair
show vrrp
show vrrp detail
show vrrp track
show vrrp summary

show vrrp detail is the one that matters, because it reports what the daemon is using rather than what you typed. Timer units are the commonest place for a VRRP change to be quietly wrong — treat a difference between the value you set and the value reported here as a stop condition and resolve it before the failover test, not after.

show vrrp track confirms the tracked objects loaded and are up. A tracking stanza that committed but shows nothing here is a tracking configuration that will not fire when you need it.

Confirm the wire agrees with both routers:

Read-only / Safewhat is actually on the segment
# Substitute your own values before running:
LAN_IF=eth0

sudo tcpdump -ni "$LAN_IF" 'ip proto 112' -c 10 -vv

Exactly one source address should be sending advertisements. The capture is the arbiter when the two routers disagree: configuration can be wrong, daemon state can be stale, but the segment shows what is really happening.

Prove the failover

A VRRP pair that has never failed over has not been verified. This is the step that gets cut when the window runs short, and it is the step that decides whether the pair works when it matters.

Fail the tracked uplink, not the router. That is the better test for two reasons: it exercises the tracking path, which is the part most likely to be misconfigured, and it leaves you logged into the master so you can watch the transition and reverse it.

Service impact possibleinduce the failure from outside this router
# Admin-down the uplink at the far end, or unplug it. Then, on the master:
show vrrp
show vrrp track

While it is failing over, measure three things from a host on the segment: how long traffic stops, whether it comes back on its own, and whether the host’s ARP entry still points at the virtual MAC. The third is what tells you the failover happened at the VRRP layer rather than the host re-resolving. Record the recovery time in the ticket; it is the number someone will ask for during the next incident.

Expect the recovery to take longer than the protocol timers suggest. The switch also has to relearn which port the virtual MAC lives on, and the gratuitous ARP the new master sends is what prompts it. On a large segment that relearn is a visible part of the total.

Then restore the uplink and confirm the pair settles the way the preemption decision says it should: back to the original master if preemption is enabled, staying put if it is not. A pair that does something else has a preemption setting that does not match the ticket.

Confirm both routers, or hold both

Configuration changeon each router
configure
confirm
save
exit

Both routers. A confirmed master with an unconfirmed backup reverts to a single router the moment the backup’s timer expires, and it does so quietly — the pair was working when everyone stopped looking.

Holding is a first-class option, and here it is symmetric. If anything failed, let both timers expire so the segment goes back to the gateway it had this morning. Leaving one router configured is the worst of the three states: it looks like redundancy, and it is not.

Rollback

Service impact possibleremove the group
configure
delete high-availability vrrp group 1
commit
save
exit

Roll the backup back first, then the master. The other order hands the virtual address to a router you are about to strip it from.

Removing VRRP removes the virtual address from the segment. Every host pointing at it as a default gateway loses its gateway at that instant, so restoring the LAN to a real router address is part of the rollback, not a follow-up. And hosts that cached the virtual MAC keep sending to it until the entry ages out, so recovery is not instantaneous even once both routers are clean.

Common patterns

SymptomLikely causeNext check
Both routers report MasterAdvertisements not crossing the segment, or a v2/v3 version mismatchCapture protocol 112 on the segment; compare the version field on both
Backup never takes overGroup id, virtual address or interface differs between the routersshow vrrp detail side by side, field by field
Failover works, traffic does not recoverAsymmetric upstreams and a stateful firewall with no session for the new pathCompare the default route on each router
Pair flaps after the master recoversPreemption firing before the switch port has convergedshow vrrp detail for the preemption setting; check the switch port state
Master keeps the role with a dead uplinkNo tracking configured, or a decrement too small to cross the backupshow vrrp track, then compare effective priorities
Gateway moves back and forth on its ownA tracked object that flapsshow vrrp track and the tracked link, then /var/log/keepalived.log
Recovery is far slower than the timers predictSwitch relearning the virtual MAC after the gratuitous ARPTime the test from a host; check the switch MAC aging timer
Committed values differ from what the daemon reportsWrong timer unitshow vrrp detail against the ticket

Provenance

This runbook has not been executed end to end against a live router pair, which is why last_verified reads pending rather than carrying a date. Its command set comes from the course’s own VRRP lessons — vyos-xl-01-vrrp-concept, vyos-xl-02-priority-and-skew, vyos-xl-03-preemption, vyos-xl-05-vrrp-tracking and vyos-xl-06-vrrp-troubleshoot — the telemetry material in vyos-xlix-04-vrrp-telemetry, and the commit discipline in vyos-vi-02-commit-confirm. No command output is reproduced here, because none of it was captured.

References

  1. VyOS 1.5 LTS documentation — High availability (VRRP)
  2. VyOS documentation — CLI, configuration mode and commit-confirm
  3. RFC 5798 — Virtual Router Redundancy Protocol (VRRP) Version 3
  4. RFC 3768 — Virtual Router Redundancy Protocol (VRRP)
  5. Keepalived documentation