high riskservice affecting~45 min

Runbook: Troubleshoot VRRP

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The report has been converted into a symptom before any command is run. "VRRP is broken" is what was said. "Both routers report Master", "the master died and the backup stayed Backup", "the failover happened and traffic did not come back" and "one Master, one Backup, and the LAN still cannot reach anything" are four unrelated faults, and this sentence decides which one you are working on.
· An operator is on BOTH routers, and show vrrp has been read from each of them, before any hypothesis is formed. A VRRP fault is a statement about two routers and a segment; the router that looks healthy is very often the one carrying the misconfiguration.
· Whether this pair has EVER failed over successfully is established. A pair that has never been tested has never been verified, and "it worked until Tuesday" points at a change rather than at the protocol.
· Out-of-band console access to both routers is confirmed by using it, before any remediation. Several fixes below remove or re-commit the virtual address, and on a router reached across the segment that address may be part of the access path.
· Nobody has restarted keepalived, bounced the LAN interface, or re-committed the VRRP group yet. If someone has, say so in the ticket: the transition history in /var/log/keepalived.log and the daemon counters are the evidence, and a restart resets both.
· The intended design is to hand: group id, virtual address, VRRP version, advertisement interval, the priority on each router, the preemption decision, and every tracked object with its decrement. Without it you cannot tell a misconfiguration from a design you disagree with.
· The change record for the last 72 hours covers the SWITCH as well as the two routers. A switch firmware upgrade that re-enabled IGMP snooping is a textbook cause of split brain and it appears in neither router commit log.
· Access to the LAN switch, or to somebody who has it, is arranged before diagnosis starts. Most VRRP faults that get as far as a runbook are Layer 2 faults, and none of those are fixed from the routers.
· The blast radius is written in the ticket as a host count: every host on the segment using the virtual address as its default gateway. Not "two routers" — the number of machines whose off-network traffic is at stake.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Capture the failing state on BOTH routers before touching either: show vrrp, show vrrp detail, show vrrp track, show vrrp summary, and show configuration commands | grep vrrp. Attach it to the ticket. Everything below can destroy some of it.
2Read the two show vrrp outputs side by side and place the fault in one of four states. STATE A: both routers report Master. STATE B: the master is gone and the backup still reports Backup. STATE C: the roles changed correctly and traffic did not recover. STATE D: one Master, one Backup, and the LAN still reaches nothing. The four share almost no further steps.
3Confirm show vrrp detail reports the values the design intended — priority, advertisement interval, preemption, version — rather than the values you believe were committed. The daemon reports what it is using; the configuration reports what somebody typed, and a wrong timer unit is the commonest way for those two to differ.
4Take the capture. sudo tcpdump -ni eth0 "ip proto 112" -c 10 -vv on the LAN segment, from a span port or a monitor port where one exists. Advertisements carry the source address, the VRRP version, the group id and the priority, and that is the only evidence that neither routers nor configuration can contradict.
5STATE A — read the version, vrid and source address fields in the capture before anything else. Two source addresses advertising for the same vrid confirms split brain; different version numbers, or advertisements from only one router visible at each end, name the cause.
6STATE A — do NOT adjust priorities. Priorities only settle an election between routers that can hear each other. Two routers that cannot hear each other stay Master at any priority, and the change loses you the evidence.
7STATE B — check the obvious local causes on the backup first: show interfaces for the VRRP interface state, and show vrrp for an Initialize state. A router does not run VRRP on a down interface, and Initialize with an empty capture is a configuration or interface fault, not an election fault.
8STATE B — compare group id, virtual address, VRRP version and advertisement interval field by field between the two routers with show vrrp detail. A backup configured for a different group is not a backup; it is a second, independent VRRP router that happens to share a segment.
9STATE C — the election worked, so stop looking at VRRP. Compare show ip route 0.0.0.0/0 on both routers. Different upstreams mean the failover moved live sessions onto a path whose stateful firewall has no entry for them, and that is a design fault surfaced by VRRP rather than a VRRP fault.
10STATE C — if both routers share an upstream, look at the segment instead: the switch has to relearn which port the virtual MAC lives on, and the hosts have to accept the new master gratuitous ARP. Check a host ARP entry for the virtual IP against the 00:00:5E:00:01:XX virtual MAC for this group.
11STATE D — read show vrrp track on the master. An empty output on a router whose uplink is down is the whole diagnosis: nothing decrements its priority, so it keeps the role and blackholes every host on the segment.
12STATE D — where tracking IS configured, compare the effective priority against the peer priority rather than the configured one. A decrement that leaves the master above the backup is tracking that runs and changes nothing.
13Correlate the shape of the failure with the clock before proposing a cause. A gateway that moves back and forth on a rhythm is a flapping tracked object or preemption fighting a slow-converging switch port. A pair that broke once and stayed broken is a change. Neither is diagnosed by watching it happen again.
14Only now choose remediation, and prefer the narrowest action the evidence supports. Where the cause is on the switch, the router fix is to collapse to one working gateway and hand the Layer 2 problem over — not to reconfigure VRRP around it.
15Apply any configuration fix with commit-confirm 10 rather than a bare commit, on one router at a time, and re-run the same commands captured in step 1 before running confirm and save.
16Write the diagnosis as a layer, not as an outcome. "Split brain — IGMP snooping re-enabled by the switch firmware upgrade dropped 224.0.0.18" is usable next time. "VRRP was flapping, restarted keepalived, fine now" guarantees a repeat.

4 · Verification

Confirm the procedure actually fixed the problem.

✓show vrrp reports Master on exactly one router and Backup on the other, read from BOTH routers rather than inferred from one
✓A capture of IP protocol 112 on the segment shows advertisements from exactly one source address, with the vrid, version and priority the design specifies
✓show vrrp detail on both routers reports the intended advertisement interval, priority, preemption setting and version — the values the daemon is using, not the values in the ticket
✓show vrrp track lists every tracked object with its state, and the effective priority reflects any decrement currently applied
✓A host on the segment has an ARP entry mapping the virtual IP to the 00:00:5E:00:01:XX virtual MAC for this group id, and reaches a destination beyond the router through it
✓A real end-to-end flow is tested from a host on the segment, not a router-sourced ping. Router-to-upstream proves the router has a path; only host-to-destination proves the gateway, the routing and the firewall together
✓systemctl status keepalived is active on both routers, and /var/log/keepalived.log shows the transitions this incident produced and no further ones after the fix
✓Where the fault was a repeated transition, the pair has been observed stable for longer than the interval at which it was flapping — a quiet minute proves nothing about a five-minute cycle
✓Any temporary change made during diagnosis — a disabled interface, a widened firewall rule, an adjusted priority — has been removed, and the two routers match the intended design again
✓The ticket names a layer: Layer 2 multicast, version mismatch, group mismatch, tracking, preemption, upstream asymmetry, or switch relearning — with the evidence that established it

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Diagnosis up to the first commit is read-only and there is nothing to roll back. Everything below concerns undoing a fix or an emergency action.
↶A fix still inside its commit-confirm window that made things worse: do nothing. Let the timer expire and the router restores itself. On a live gateway that is the cheapest and least dramatic undo available.
↶A committed and confirmed fix: delete the specific nodes you added, then commit, inside configure. Restoring the whole high-availability subtree in one step is safer than unpicking it by hand when several nodes were touched.
↶Rolling the pair back to a single router is a legitimate emergency action, and the order matters: remove VRRP from the BACKUP first, then decide about the master. Removing it from the master first hands the virtual address to a router you are about to strip.
↶Removing VRRP removes the virtual address from the segment, so every host using it as a default gateway loses its gateway at that instant. Pointing the LAN back at a real router address is part of the same change, not a follow-up.
↶Hosts that cached the virtual MAC keep sending to it until the entry ages out, so recovery after a rollback is not instantaneous even once both routers are clean. Check a host ARP cache before declaring the rollback successful.
↶Any priority, preemption or timer value changed during diagnosis is temporary until it is agreed and recorded. A priority edited at 03:00 to make the election go the way you wanted outlives everyone who remembers why.
↶If keepalived was restarted or the interface bounced to restore service, that is a service restoration and not a diagnosis. Record that the evidence was lost, and keep the ticket open until the cause is found or the next occurrence is instrumented.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the team that owns the LAN switch as soon as the capture shows advertisements from both routers, or shows a router hearing only itself. That is filtered multicast, IGMP snooping, a VLAN mismatch or a forwarding fault, and no amount of router configuration fixes it.
· Escalate to network engineering when the election is demonstrably correct and traffic still fails after a failover. Asymmetric upstreams behind a stateful firewall are a design question with an owner, and reconfiguring VRRP only adds variables.
· Escalate to the network architect when the pair flaps repeatedly after a recovery. Repeated transitions are usually preemption firing before a switch port has converged, or a tracked object that is not a clean event; both are decisions to be made, not values to be tuned during an incident.
· Escalate to the LAN owner before inducing any failover as a test. The test is a deliberate short outage for every host on the segment, and consent for it belongs before the window rather than during it.
· Escalate to the platform team if a router stops answering on both the data and the management path. Recover from the console session opened in the pre-checks; do not attempt further remediation across a path that may be the thing that is broken.
· Escalate to the security owner if the capture shows advertisements for this group from a source address that is neither router. Unauthenticated VRRPv3 on an open segment is a first-hop redundancy protocol, not an authorisation mechanism.
· Holding is a first-class outcome. Collapse to one working gateway, leave the pair unconfigured, record the hold with an owner and an end time, and come back with the Layer 2 evidence that was missing. One gateway that works beats two routers arguing about which is the gateway.

“VRRP is broken” arrives in four different shapes and they have almost nothing in common.

Somebody reports intermittent connectivity on a LAN and both routers turn out to think they are master. Somebody reports that a router died and the gateway went with it, while the backup sat in Backup the whole time. Somebody reports that the failover worked perfectly and every TCP session died anyway. Somebody reports that the pair looks textbook — one Master, one Backup, no transitions — and no host on the segment can reach the internet.

Those are four faults. Two of them are not VRRP faults at all. The first job of this runbook is to work out which one you have, because the diagnostic paths diverge immediately and running the wrong one costs you the window.

When this runbook applies

A VRRP pair on a VyOS 1.5 router is not producing the redundancy it was built for.
Hosts on a segment report intermittent or absent connectivity and the gateway is a virtual address.
A failover happened and something about the result was wrong — too slow, incomplete, or repeated.

When it does not apply

There is no VRRP pair yet. Building one is a different change with a different risk profile; use the VRRP configuration runbook.
The redundancy you need is between routers, not for hosts. VRRP is first-hop redundancy and nothing else. If the question is which path leaves the site, that is OSPF, BGP or policy routing.
The pair is fine and one router is unwell. A router with a saturated CPU or a failing disk will eventually disturb VRRP, but VRRP is the symptom there. Diagnose the router.
The complaint is throughput, not reachability. VRRP moves an address. It does not carry traffic, share load, or influence how fast anything goes.

Blast radius, and the three things not to do first

The blast radius of this diagnosis is every host on the segment that uses the virtual address as its default gateway. Write that number in the ticket before you start; it is what decides whether an induced failover is acceptable and who has to agree to it.

Three actions feel natural at the start of a VRRP incident and all three are expensive:

Do not restart keepalived. It is the reflex, it sometimes appears to work, and it resets the daemon counters and the transition history that were about to tell you what happened. On a router that is currently master it also drops the virtual address, so the “diagnostic” is an outage.

Do not adjust priorities to break a tie. Priority is how two routers that can hear each other settle an election. Two routers that cannot hear each other will both stay master at any priority, so the change cannot fix split brain — it can only make the configuration no longer match the design you are about to compare it against.

Do not induce a failover to “see if it works”. Once the incident is open, a test failover is a second outage on a segment that is already degraded, and the LAN owner has not agreed to it. Test failovers belong in a planned window.

Step 1: capture both routers before you change either

Read-only / Safeon each router, in this order

show vrrp
show vrrp detail
show vrrp track
show vrrp summary
show configuration commands | grep vrrp
show interfaces

show vrrp gives you the state word. show vrrp detail is the one that matters, because it reports what the daemon is actually using — priority, advertisement interval, preemption, version — rather than what somebody typed. A difference between those two outputs and the ticket is a finding in itself, and timer units are the commonest place for it to appear.

show vrrp track is the fastest way to eliminate an entire failure class: an empty tracking output on a router whose uplink is down tells you why the LAN is blackholed, in one command.

Then take the same capture from the other router. This is the step that gets skipped under pressure and the step that most often ends the investigation, because the two outputs read together answer a question neither answers alone: do these two routers agree about what is happening?

Step 2: place the fault in one of four states

Both routers say	The fault is	Go to
Master, Master	Advertisements are not crossing the segment	State A
Master gone, Backup still Backup	The two are not in the same group, or the backup cannot run VRRP	State B
Roles changed correctly, traffic still broken	Not VRRP — routing, firewall state, or the switch	State C
Master, Backup, and the LAN reaches nothing	Nothing decrements the master priority	State D

The rest of this runbook is those four paths. Read the one you are in.

State A — both routers report Master

Split brain is a Layer 2 diagnosis wearing a Layer 3 costume. Both routers are behaving exactly as the protocol requires: each hears no advertisement with a higher priority, so each takes the role. The segment now has two devices answering ARP for the same address and a switch learning the virtual MAC on two ports, which is why hosts see intermittent rather than absent connectivity — it depends which answer they cached.

The capture settles it:

Read-only / Safeon the segment, ideally from a span or monitor port

# Substitute your own values before running:
LAN_IF=eth0

sudo tcpdump -ni "$LAN_IF" 'ip proto 112' -c 10 -vv

VRRP advertisements are sent to the multicast group 224.0.0.18 as IP protocol 112, with the sending router’s real interface address as the source. The fields visible in the capture are the entire diagnosis: the source address tells you who is advertising, the version field tells you whether the two routers are speaking the same protocol, the vrid tells you whether they are in the same group, and the priority tells you whether tracking has already fired.

Four causes account for nearly all of it:

The segment is not forwarding the multicast. IGMP snooping, a filtering rule, or a switch that stopped flooding 224.0.0.18. Each router hears only itself.
A version mismatch. One router on VRRPv2 and one on VRRPv3 cannot participate in the same group, and both become master.
A VRRPv2 authentication mismatch. The receiver discards advertisements whose authentication does not match, which looks exactly like the multicast never arriving. VRRPv3 has no protocol-level authentication, so this cause disappears when both ends are v3.
A group id mismatch. Two routers configured for different vrids are not a pair. They are two independent VRRP routers that happen to share a segment, and each is correctly the master of its own group.

Causes 2, 3 and 4 are visible in the capture and fixable from the routers. Cause 1 is not fixable from the routers at all.

A note on timing that saves an argument: if this appeared immediately after somebody upgraded the switch, that is not a coincidence. Firmware upgrades routinely reset multicast defaults, and the change is invisible in both routers’ commit logs — which is exactly why the pre-checks ask for the switch change record as well as the routers’.

State B — the master failed and the backup stayed Backup

Start with the two cheap local checks, because they are also the most common answers.

Read-only / Safeon the router that should have taken over

show vrrp
show interfaces
show vrrp detail

A router in Initialize state is not participating in an election at all. VRRP does not run on a down interface, so a VRRP interface that is administratively or physically down produces exactly this: a router that is up, configured, and invisible to the protocol. Combined with an empty capture on the segment, Initialize is a configuration or interface fault, not an election fault.

If the backup is genuinely in Backup and simply never transitioned, the question is whether it is in the same group as the master at all. Compare, field by field, from show vrrp detail on both routers:

the group id
the virtual address
the VRRP version
the advertisement interval

Those four must be identical. Priority is the only field that is supposed to differ. A “backup” configured for a different virtual address is a router waiting patiently for advertisements about an address nobody is advertising, and it will wait forever without logging anything interesting.

How long the backup is supposed to wait

A backup declares the master down after the master-down interval, which is three advertisement intervals plus a skew. With the defaults — a one-second advertisement interval and no configured skew — that is roughly three seconds.

RFC 5798 derives an implicit skew from the router’s own priority, so a higher-priority backup waits marginally less than a lower-priority one and the election settles without both of them announcing at once. VyOS also exposes a skew value in centi-seconds, and that one is additional: it is added on top of the RFC-implicit skew rather than replacing it. A skew of 200 turns a three-second failover into a five-second one.

This matters here for one reason. If somebody set a skew to stop a pair flapping, the pair now fails over more slowly than the design says, and the gap shows up as “the failover took ages” rather than as a configuration error. show vrrp detail reports what the daemon is using; read it there rather than reasoning from the ticket.

State C — the roles changed and traffic did not come back

This is the state where the most time is wasted, because everything about VRRP is working and VRRP is the only thing being looked at. The election fired, the address moved, the new master is advertising. Stop looking at show vrrp and start looking at what changed underneath the address.

First, compare the default route on both routers.

Read-only / Safeon both routers

show ip route 0.0.0.0/0

If each router leaves the site by a different upstream, the failover did not just move a gateway — it moved every live session onto a path whose stateful firewall has no entry for any of them. The return packets arrive at a firewall that never saw the outbound half and are dropped. The symptom is TCP sessions that hang rather than reset, which reads like a firewall fault, and the root cause is the interaction between the election and the routing.

There is no VRRP setting that fixes this. The choices are to align the upstreams so the two routers are interchangeable, or to accept the asymmetry and disable preemption so that at least the pair stops moving back and forth between two incompatible paths. Both are decisions with an owner, made outside an incident.

Second, if the upstreams are the same, look at the segment.

A failover has to be believed by three parties, not one. The new master sends a gratuitous ARP so hosts update their caches. The switch has to relearn which port the virtual MAC now lives behind. And the hosts have to accept the gratuitous ARP, which some deliberately do not.

Read-only / Safefrom a host on the segment

# Substitute your own values before running:
VIP=192.168.1.1

arp -n "$VIP"

The virtual MAC for an IPv4 group follows 00:00:5E:00:01:XX, where the last byte is the group id in hex — group 1 is 00:00:5E:00:01:01, group 10 is 00:00:5E:00:01:0A. That OUI is reserved, so an ARP entry showing it is unambiguous evidence the host is talking to the virtual router rather than to a real interface.

A host still pointing at the correct virtual MAC while traffic fails means the address moved and the frames are not following it: the switch is still forwarding the virtual MAC toward the old port. A host pointing at something else entirely means the host never accepted the gratuitous ARP and is caching a stale answer, which will resolve on its own when the entry ages out — in minutes, not seconds.

Both explain a recovery that takes far longer than the protocol timers predict, and neither is visible from the routers.

State D — one Master, one Backup, and the LAN reaches nothing

The pair looks perfect and the segment is blackholed. This is VRRP doing precisely what it was asked to do: the master is alive, it is advertising, it wins the election. It simply cannot reach anything.

Read-only / Safeon the master

show vrrp track
show interfaces
show ip route 0.0.0.0/0

An empty show vrrp track is the whole diagnosis. Without tracking, VRRP has no opinion about whether the router can forward anything; it only knows the router is running. A master whose uplink has failed keeps advertising at full priority and keeps attracting every host’s off-network traffic into a dead end.

Where tracking is configured, the failure is usually arithmetic. Compare the effective priority against the peer’s priority, not the configured one. A master at 200 tracking its uplink with a decrement of 30 falls to 170 when that uplink dies — still comfortably above a backup at 100, so the election does not move and the tracking configuration has achieved nothing except the appearance of safety. The decrement has to genuinely cross the backup’s priority.

VyOS 1.5 offers three tracking forms and they suit different failures: interface tracking for a link that goes hard down, route tracking for an upstream learned dynamically where the failure shows up as a route disappearing, and remote-IP tracking with an interval and a threshold for cases where neither is true. A threshold matters on the last one — a single lost probe should not move a LAN’s gateway.

What each action costs if it is wrong

Action	If it is right	If it is wrong
Restart keepalived	Service returns	The master drops the address; the transition history and counters are gone, and the cause is now unknowable
Adjust a priority	The intended router holds the role	The configuration no longer matches the design you are comparing against, and split brain is untouched
Remove VRRP from a router	The pair collapses to one working gateway	Removed from the master first, the address lands on a backup you are about to strip, and the LAN loses its gateway twice
Enable or disable preemption	Flapping stops, or the design is restored	On asymmetric upstreams, enabling it moves live sessions onto a path with no firewall state — a second outage during the recovery
Shorten the advertisement interval	Failover is quicker	On a loaded router or a lossy segment, transient delays now look like a dead master and the pair starts flapping
Induce a failover to test	The pair is proven	It is a deliberate outage for every host on the segment, during an incident, without the LAN owner’s agreement

Holding is a first-class option

If the evidence points at the switch and you cannot get the switch fixed inside this window, the right outcome is not to keep configuring routers.

Collapse the pair to one working gateway: remove VRRP from the backup first, confirm the LAN has a gateway it can reach, and record a hold with an owner and an end time. A single router with no redundancy is a known, documented state that the next shift can act on. Two routers arguing about which of them is the gateway is not.

The half-measure to avoid is leaving one router configured for VRRP and the other not. It looks like redundancy on every dashboard, and it is not.

Common patterns

Symptom	Likely cause	Next check
Both routers report Master	Multicast not crossing the segment, or a v2/v3 mismatch	Capture protocol 112; compare the version field from both ends
Both Master, appeared after switch work	IGMP snooping re-enabled by the firmware upgrade	The switch change record, then the capture
Backup never takes over	Group id, virtual address or version differs between routers	`show vrrp detail` side by side, field by field
A router reports Initialize	The VRRP interface is down, or no group is configured on it	`show interfaces`, then the VRRP configuration on that router
Failover fires, sessions still die	Asymmetric upstreams and a stateful firewall with no state for the new path	`show ip route 0.0.0.0/0` on both routers
Recovery far slower than the timers predict	Switch relearning the virtual MAC, or hosts ignoring gratuitous ARP	A host ARP entry for the virtual IP, then the switch MAC table
Master holds the role with a dead uplink	No tracking, or a decrement too small to cross the backup	`show vrrp track`, then effective against peer priority
Gateway moves back and forth on a rhythm	A flapping tracked object, or preemption firing before the switch port converges	`show vrrp track` and `/var/log/keepalived.log`
Committed values differ from observed behaviour	Wrong timer unit, or a skew somebody added to stop a flap	`show vrrp detail` against the design
Everything correct, no advertisements anywhere	keepalived is not running	`systemctl status keepalived` on both routers

Provenance

This runbook was written from the course’s own Part XL lessons — vyos-xl-01-vrrp-concept, vyos-xl-02-priority-and-skew, vyos-xl-03-preemption, vyos-xl-04-advertisements, vyos-xl-05-vrrp-tracking and vyos-xl-06-vrrp-troubleshoot — together with vyos-xlix-04-vrrp-telemetry for the keepalived state and log material, and vyos-lii-02-evidence-first for the capture-before-change discipline. It has not been executed against a live pair, and no output is reproduced here as if it had been captured.

Why this matters

The reason VRRP incidents run long is that VRRP is almost never the thing that is broken.

Three of the four states in this runbook resolve somewhere else. Split brain is a switch. A failover that does not restore service is a routing design, or a switch relearning a MAC address. A master that blackholes a segment is a tracking decision that was never made. Only the fourth — two routers configured into different groups — is a VRRP configuration fault, and it is the least common of the four.

That is why the discipline that pays here is the boring one: get the state from both routers and the capture from the wire before forming a hypothesis. The protocol is simple enough that the evidence is unambiguous once you have it, and confusing enough from one router that a confident, entirely wrong conclusion is available within thirty seconds.

And the habit worth keeping past this incident is treating “the pair looks healthy” as a claim rather than a result. A VRRP pair reports healthy right up until the moment somebody needs it, which is the only moment it has ever been asked to do anything.