VRRP Failover Fails

Reported symptoms

At 04:12 the LAN behind a VRRP pair stopped reaching anything off-network. It came back at 04:53, when an engineer administratively shut R1’s LAN interface and R2 took over.

The pair is the ordinary shape: R1 and R2 share the virtual address 192.168.1.1 on the LAN segment, R1 has an uplink to ISP-A on eth1, R2 has an uplink to ISP-B on eth1, and the hosts use the virtual address as their default gateway.

What the on-call engineer found, in the order they found it:

• Hosts could ping 192.168.1.1 the entire time. The gateway answered ARP and answered ICMP.
• Hosts could reach each other across the LAN normally.
• show vrrp on R1 said Master. show vrrp on R2 said Backup. That is exactly the output the troubleshooting runbook says to expect from a healthy pair, so VRRP was ruled out in the first five minutes.
• No alert fired. The VRRP alert is bound to state transitions and there were none.
• Tracking was not missing. show vrrp track on R1 listed eth1 and reported it down, which is the tracking feature working.
• The same cable had been pulled deliberately during a failover test in June and R2 had taken over within seconds.

The only change on either router since June was a standardisation commit applied by automation to eleven sites, setting every VRRP master to priority 200 and every backup to priority 100. Ten sites were unaffected.

Evidence provided

vyos@R1:~$ show vrrp

Virtual Router 1 - Master
State: Master
Virtual IP: 192.168.1.1
Virtual MAC: 00:00:5E:00:01:01
Priority: 200
Advertisement interval: 1s
Preemption: enabled

vyos@R2:~$ show vrrp

Virtual Router 1 - Backup
State: Backup
Virtual IP: 192.168.1.1
Virtual MAC: 00:00:5E:00:01:01
Priority: 100
Advertisement interval: 1s
Preemption: enabled

vyos@R1:~$ show interfaces ethernet eth1

eth1: state: down
  link/ether 52:54:00:1a:2b:3c
  inet 198.51.100.2/30

vyos@R1:~$ show ip route 0.0.0.0/0

% Network not in table

vyos@R1:~$ show configuration commands | match vrrp

set high-availability vrrp group 1 interface eth0
set high-availability vrrp group 1 priority 200
set high-availability vrrp group 1 virtual-address 192.168.1.1/24
set high-availability vrrp group 1 preemption delay 0
set high-availability vrrp group 1 version 3
set high-availability vrrp group 1 track interface eth1 priority-decrement 90

vyos@R2:~$ show configuration commands | match vrrp

set high-availability vrrp group 1 interface eth0
set high-availability vrrp group 1 priority 100
set high-availability vrrp group 1 virtual-address 192.168.1.1/24
set high-availability vrrp group 1 preemption delay 0
set high-availability vrrp group 1 version 3
set high-availability vrrp group 1 track interface eth1 priority-decrement 90

Work the evidence before reading on

Nothing here is broken in the sense of being unable to do its job. The uplink is down and the router knows it. Tracking is configured and is reporting the tracked object correctly. VRRP is running on both routers, they can hear each other, and they agree on who is master.

1. R1’s configured priority is 200 and its tracked decrement is 90. With eth1 down, what number is R1 advertising?
2. R2’s configured priority is 100 and its own uplink is up. What number is R2 advertising?
3. Given those two numbers, what is the correct VRRP election result?
4. In June the test passed. The configuration file has one line different from June. Which line, and what were the two numbers then?

Before continuing: was this a VRRP failure at all? State what VRRP was asked to do and whether it did it.

Root cause

1. Tracking subtracts; it does not decide

The mental model that produces this outage is “tracking makes the router fail over when the uplink dies”. It does not. Tracking changes one field in the advertisement — the priority — and the election is then the same comparison it always was: highest advertised priority wins, ties broken by the interface address.

So the question is never “is tracking configured”, which is what show vrrp track answers. The question is what number comes out of it:

110 is greater than 100. R1 stayed master. Every component in the path behaved exactly as configured, and the aggregate behaviour was a forty-one-minute black hole.

This is why the pair looked healthy to the runbook. One master and one backup is the healthy steady state, and it is also the steady state of a pair that has completely failed to protect the service. The two are not distinguishable from show vrrp alone.

2. The June test was run against a different configuration

The site was built with R1 at priority 150 and a decrement of 90. That put R1’s degraded priority at 60, forty points below R2’s 100, which is a sensible margin and is the configuration the June failover test exercised.

The standardisation commit raised R1 from 150 to 200 and left the decrement at 90. It did not raise R2, which was already at 100.

# What the change was, in effect
- set high-availability vrrp group 1 priority 150
+ set high-availability vrrp group 1 priority 200

One line. Nothing about it looks like a failover change, and the standard it implements — master 200, backup 100 — is the right standard. The failure is that priority and decrement are one setting split across two nodes, and only one of them was in scope.

3. Ten sites were fine, which is why nobody looked

The other ten sites already carried a decrement of 150. At 200 minus 150 they land on 50, comfortably below 100, and the standardisation improved them. A change that is correct almost everywhere and silently wrong at one place will not be caught by a spot check, and it will not be caught by a post-change validation that asks whether the commit applied.

Resolution

1. Establish the current numbers on both routers before changing anything: configured priority, configured decrement, and which object each router tracks. Write down the effective priority each router would advertise with its tracked object down. This is the artefact the incident review needs and it takes two commands per router.
2. Decide the interim action deliberately. If the uplink is still down and the site is still black-holed, raising R2 priority above R1 degraded value forces the election and is trivially reversible. Shutting R1 LAN interface also works and is what was done at 04:53, but it removes R1 from the segment entirely, so a subsequent failure of R2 has nothing to fail back to.
3. Set the decrement so the degraded priority falls strictly below the peer, with margin: with 200 and 100 in force, a decrement of 150 gives an effective priority of 50. Apply the same value on both routers so the pair is symmetric and each protects against its own uplink, not a shared one.
4. Commit under commit-confirm on the router that currently owns the virtual address. The change alters what that router advertises for the LAN default gateway, and if your session runs through it, an unattended mistake takes the session and the site together.
5. Re-check the uplink itself. Tracking has been reporting eth1 down since 04:12 and that is a separate fault with its own owner; the VRRP fix makes the site survive it, it does not repair it.
6. If the uplink is known to flap rather than fail cleanly, track the default route instead of the interface. A link that returns without restoring routing restores an interface-tracked priority and hands the traffic back to a router that still cannot forward it.
7. Record the numbers in the change record: priority, decrement, effective priority, peer priority. The next standardisation will read this record or repeat the incident.

Verification

1. The arithmetic holds independently on each router. On R1, configured priority minus decrement is strictly less than R2 configured priority; on R2, the same statement with the roles swapped. Check it by reading the configuration, not by inference from show vrrp track, which reports whether an object is tracked and not what subtracting it produces.
2. The failure actually flips the election. Take R1 uplink down administratively, then confirm show vrrp on R2 reports Master. Three seconds is the expected order of magnitude at the default advertisement interval.
3. Traffic follows the election. From a host on the LAN, reach a destination off the network — not the gateway address. Reaching 192.168.1.1 was true throughout the outage and proves only that something on the segment answers for it.
4. Recovery returns the pair to its designed state. Restore R1 uplink and confirm R1 becomes master again, since preemption is enabled here. If it does not, preemption is the next thing to read, not the decrement.
5. The reverse direction is covered too. Repeat the test against R2 uplink and confirm R1 keeps or takes the master role. A pair verified in one direction only is half verified, and the decrement was symmetric in this configuration purely by luck.
6. The new alert fires. Force the condition — master with no default route — on a lab pair and confirm the alert is raised. An alert that has only ever been silent is untested, and the silent alert is exactly what this incident had.
7. The other ten sites are re-checked against the same arithmetic rather than assumed correct because they were unaffected. They were unaffected by this change; that is a different claim.

Prevention

• Change priority and decrement together or not at all. They are one setting expressed as two configuration nodes, and every incident of this shape comes from editing one of them.
• Make estate-wide standardisation state its effect. A standard that fixes priorities without fixing decrements is under-specified; the automation applying it should refuse to run where the resulting effective priority would sit above the peer’s, rather than applying cleanly and reporting success.
• Validate the computed outcome, not the presence of configuration. show vrrp track listing an object is not evidence of failover; the evidence is priority minus decrement compared against the peer.
• Alert on the black-hole condition. VRRP master with no default route is a page, and it is independent of transitions. The transition-based alert was correct and useless here, because the bug is that no transition happened.
• Re-run the failover test after any change to priority, decrement, preemption or tracked object, and record which numbers the test was run against. A green test from June is a statement about June’s configuration.
• Track each router’s own uplink, never a shared object. Both routers tracking the LAN, or both tracking the same upstream, decrement together and leave the election exactly where it was — a different route to the same outage.