Skip to main content
RunBook Academy

← All break/fix scenarios in VyOS

advancedvyos-vrrp~30 min

Both Routers Become VRRP MASTER

Reported symptoms

  • ●Users have reported flaky internet for three days, while the intranet, the file shares and everything else inside the LAN are perfect
  • ●Short things work reliably - DNS resolves, pages load, ping to anywhere is clean and a speed test passes - which is why the first three tickets were closed as user error
  • ●Long things fail. Client VPN tunnels to a partner site drop every few minutes, uploads to a customer portal stall at a different percentage each attempt, and a large page will occasionally half-render
  • ●Staff are being logged out of a SaaS application at random and have to sign in again, which was raised with the vendor as an application fault
  • ●`show vrrp` on both routers of the HA pair reports MASTER, and each router has been in that state continuously since Tuesday
  • ●The HA dashboard is green, because its check asks the primary router whether it is master and the answer is yes
  • ●The access switch has been logging MAC address flapping for a 00:00:5e address between two ports since Tuesday afternoon, which the network team read as normal VRRP behaviour
  • ●The only change in the window was a firewall hardening rollout, signed off as input-filter-only with no impact on forwarded traffic, and no OSPF adjacency dropped during or after it

Evidence

  • · `show vrrp` on rtr-ha-a reports state MASTER, priority 200; `show vrrp` on rtr-ha-b reports state MASTER, priority 100
  • · `show vrrp detail` on rtr-ha-b records its transition to master at 14:07 on Tuesday, within seconds of that node commit
  • · `tcpdump -ni eth1 'ip proto 112' -vv` on rtr-ha-a captures advertisements from 10.20.0.252 and from 10.20.0.253, both vrid 10, ten per ten seconds
  • · The same capture run on rtr-ha-b shows both streams as well, so each router is receiving the other advertisements on the wire
  • · Both streams print the same version, the same vrid 10 and authtype none, so nothing in the packets disagrees
  • · `show firewall` shows the input filter default-drop counter on both routers incrementing at roughly one packet per second per router
  • · `show configuration commands | match "firewall ipv4 input"` lists rules for established and related traffic, ICMP, SSH from the management prefix, OSPF and BGP, and a default action of drop
  • · `show system commit` on both routers records one commit at 14:06 and 14:07 on Tuesday, pushed by the configuration pipeline and described as input filter hardening
  • · `show ip ospf neighbor` shows the core adjacencies Full on both routers and unbroken since before the change
  • · On the access switch, the virtual MAC 00:00:5e:00:01:0a appears against two ports rather than one
Diagnosis and resolutionclick to reveal

Root cause

The hardening change replaced a permissive input path with a default-drop base chain and an explicit allow-list, and the allow-list does not contain IP protocol 112. VRRP advertisements are not TCP or UDP; they are carried directly in IP as protocol 112, addressed to the multicast group 224.0.0.18, and they are delivered locally to the router rather than forwarded through it, so they are evaluated by the input chain and dropped there. Each router therefore stopped hearing its partner from the moment its own commit applied. The backup, hearing nothing for longer than the master-down interval, did exactly what the protocol requires: it concluded the master was gone, took the virtual IP, sent a gratuitous ARP and entered MASTER. The primary never demoted itself, because it too was dropping the advertisements it would have needed in order to notice a peer at all - and even had it heard them, a master that receives a lower-priority advertisement keeps the role. Both routers now own the same virtual IP and the same virtual MAC, the access switch learns that MAC on two ports and flaps between them, and traffic to the default gateway lands on one router or the other unpredictably. That is the whole symptom set. Anything that completes inside a single exchange - a DNS lookup, an ICMP echo, a small page - almost always finishes on whichever router it started on and works perfectly. Anything long-lived crosses a MAC relearn at some point, and the packet that lands on the other router finds no connection-tracking entry for the flow, gets source-NATed as a brand new connection to that router's own upstream address, and is rejected by a server that has been talking to a different address for the last ten minutes. The SaaS logouts are the same mechanism seen from outside: a session bound to a source IP that keeps changing. What let the change through review is the shape of the configuration tree. Somebody did think about the control plane and wrote allow rules for OSPF and BGP, and those adjacencies stayed up throughout, which is the evidence the rollout was signed off on. VRRP does not live under `protocols`; it lives under `high-availability`, so it was not on the list anybody was reading.

Remediation

Permit protocol 112 from the peer on the HA interface, on both routers, and sequence the two commits deliberately because they do different jobs. The commit on the backup is the one that restores service: as soon as rtr-ha-b can hear rtr-ha-a again it sees a higher-priority master, demotes itself and releases the virtual IP, and the split brain ends with one change. The commit on the primary restores nothing visible and is the more important of the two, because without it the pair still has no working election and the next real failure of rtr-ha-b would leave rtr-ha-a unable to observe anything at all. Scope the rule rather than blanket-permitting the protocol: inbound interface, source address of the peer, destination 224.0.0.18. A rule that accepts protocol 112 from anywhere on any interface reintroduces a spoofing surface on a router that was being hardened for a reason. Name the cost of the fix, because it is real: the demotion tears down every flow currently pinned to the backup. Connection-tracking state on rtr-ha-b does not migrate, so every session that happens to be traversing it at that moment ends, and on this estate that includes client VPN tunnels and long uploads - the same users who have been raising tickets for three days. That makes holding until an agreed window a defensible choice rather than a delay: the estate is degraded, not down, and the repair is briefly worse than the fault for a subset of users. A hold needs a named owner, an end time, a note on the SaaS vendor ticket so somebody stops chasing an application bug, and the tickets consolidated so the VPN, portal and login complaints stop being worked as three faults. What is not defensible is rolling back the whole hardening change to make the symptom go away; the defect is one missing rule in it, not the change itself.

Verification

Confirm the pair, not the members, because split brain is a property of the relationship and no question asked of a single router can detect it. `show vrrp` must report exactly one MASTER and one BACKUP across the two boxes, read within the same minute, and read again several minutes later so that a transition caught mid-cycle is not mistaken for a stable state. Prove the rule is matching rather than merely present: the packet counter on the new input rule must be incrementing on both routers, and the input filter default-drop counter must stop advancing at one packet per second. Go to the wire on both - each router must see both advertisement streams in a capture and now act on them - and go to the switch, where the virtual MAC 00:00:5e:00:01:0a must appear against exactly one port with no further flapping entries in the log. Then prove the redundancy rather than the absence of the symptom, which is a different and larger claim: in an agreed window, take the primary out of the group and confirm that the backup takes over within the master-down interval, that the gratuitous ARP is sent, and that traffic recovers; then restore the primary and confirm the backup steps back down. Nobody in this estate has had a working failover for three days and nobody would have found out until they needed one. Verify from the user side too: a file transfer of several minutes completes, a client VPN tunnel stays up across a coffee break, and an external service reports a stable source address rather than one that alternates. Finally, confirm the hardening is still doing its job - the default drop still counting, the new rule scoped to the peer and the interface rather than accepting protocol 112 from anywhere.

Prevention

A default-deny input chain has to be derived from what the router speaks, not from what it listens on. TCP and UDP services are easy to enumerate and are the ones every hardening template covers; the protocols that keep a router in a cluster are carried in IP directly and have no ports to look for. Build the allow-list from an inventory of the box's own control plane - OSPF as protocol 89, VRRP as protocol 112, ESP as 50, IKE and NAT-T on their UDP ports, BFD, DHCP, the routing daemons' TCP sessions - and generate it from the configuration rather than writing it by hand, because the hand writing it reads `protocols` and never opens `high-availability`. Validate a hardening change against control-plane state, not against SSH: an input filter change that leaves you logged in has been tested for exactly one thing. The specific assertion missing here is a pair-level one - alert when a VRRP group has other than one master across its members, which is a two-line check over `show vrrp json` from both routers and would have fired within four seconds of the Tuesday commit - and alongside it alert on VRRP transition count, which catches the flapping cases the count check misses. Stagger control-plane changes across the members of an HA pair and check the pair's state between the two commits; rolling both out in one window removed the only comparison that would have shown the fault. Treat a MAC flapping alarm for an address in the IANA VRRP range as a page rather than as noise: that address is only ever supposed to exist behind one port at a time, and the switch was reporting the incident correctly from the first hour. And retire the dashboard check that asks the primary whether it is master, which is true in a healthy pair and true in the worst failure this pair has.

Reported symptoms

For three days the office has had flaky internet, and the service desk has closed three tickets as user error because it keeps failing to reproduce.

The failure is real and it has a shape nobody has written down yet. Short things work. DNS resolves instantly. Pings to anywhere are clean, with no loss and no jitter. Web pages load. A speed test runs at line rate. Every quick test anybody has run has passed.

Long things fail. Client VPN tunnels to a partner site drop every few minutes and reconnect on their own. Uploads to a customer portal stall, at a different percentage each attempt. A large page will occasionally half-render, sitting with images outstanding until it is reloaded.

And staff are being logged out of a SaaS application at random, mid-task, several times a day. That one has been raised with the vendor as an application fault and the vendor has found nothing.

Everything inside the LAN is perfect. The intranet, the file shares, the print queues, the internal ticketing system - nobody has a complaint about any of it, which is why the working hypothesis is that the ISP is having a bad week.

Three more facts are on the table and nobody has put them together.

show vrrp on both routers of the HA pair says MASTER. Both have said so continuously since Tuesday. The HA dashboard is green, because the check it runs asks the primary router whether it is master and the answer is yes.

The access switch has been logging MAC address flapping for a 00:00:5e address between two ports since Tuesday afternoon. The network team looked at it, recognised the address as a VRRP virtual MAC, and filed it as normal VRRP behaviour.

And the only change in the window was a firewall hardening rollout, pushed by the configuration pipeline on Tuesday afternoon and signed off as input-filter-only with no impact on forwarded traffic. No OSPF adjacency dropped during it or since.

The estate

rtr-ha-a and rtr-ha-b are a VyOS 1.5 LTS pair providing the default gateway for the office LAN.

  • eth1 on each faces the LAN, 10.20.0.0/24. rtr-ha-a is 10.20.0.252 and rtr-ha-b is 10.20.0.253.
  • VRRP group LAN, VRID 10, virtual address 10.20.0.1, which is the default gateway every host on the segment uses. Priority 200 on A, 100 on B, preemption enabled, default advertisement interval.
  • eth0 on each is a separate ISP tail. Each router source-NATs LAN traffic to its own public address: 203.0.113.10 on A, 198.51.100.10 on B. Each holds its own connection-tracking state and neither shares any of it.
  • eth2 on each goes to the core, running OSPF in area 0. This is how the site reaches the DC, and it is the part of the control plane everyone thinks of as the routing.

On Tuesday afternoon the security programme rolled out a default-deny input filter to every router in the estate: an explicit allow-list for management and control-plane traffic and a base-chain default action of drop. rtr-ha-a committed at 14:06 and rtr-ha-b at 14:07.

The allow-list contains rules for established and related traffic, ICMP, SSH from the management prefix, OSPF and BGP.

Evidence provided

Read-only / Safethe primary, master, and stable for a week
vyos@rtr-ha-a:~$ show vrrp
Name    Interface    VRID    State      Priority    Last Transition
----    ---------    ----    -----      --------    ---------------
LAN     eth1         10      MASTER     200         6d 21h 04m

Illustrative output

Read-only / Safethe backup, also master, since Tuesday at 14:07
vyos@rtr-ha-b:~$ show vrrp
Name    Interface    VRID    State      Priority    Last Transition
----    ---------    ----    -----      --------    ---------------
LAN     eth1         10      MASTER     100         3d 02h 51m

Illustrative output

Read-only / Safeboth routers are transmitting, and rtr-ha-a is receiving both streams
vyos@rtr-ha-a:~$ tcpdump -ni eth1 'ip proto 112' -c 4 -vv
17:02:11.114 IP (tos 0xc0, ttl 255, proto VRRP (112), length 40)
  10.20.0.252 > 224.0.0.18: VRRPv3, Advertisement, vrid 10, prio 200, authtype none, intvl 100cs, length 20
17:02:11.802 IP (tos 0xc0, ttl 255, proto VRRP (112), length 40)
  10.20.0.253 > 224.0.0.18: VRRPv3, Advertisement, vrid 10, prio 100, authtype none, intvl 100cs, length 20
17:02:12.117 IP (tos 0xc0, ttl 255, proto VRRP (112), length 40)
  10.20.0.252 > 224.0.0.18: VRRPv3, Advertisement, vrid 10, prio 200, authtype none, intvl 100cs, length 20
17:02:12.805 IP (tos 0xc0, ttl 255, proto VRRP (112), length 40)
  10.20.0.253 > 224.0.0.18: VRRPv3, Advertisement, vrid 10, prio 100, authtype none, intvl 100cs, length 20

Illustrative output

The identical capture on rtr-ha-b shows the same two streams. Both print the same version, the same vrid 10 and authtype none, so there is nothing in the packets for the two routers to disagree about.

Read-only / Safesomebody did think about the control plane; read the list for what is not on it
vyos@rtr-ha-a:~$ show configuration commands | match 'firewall ipv4 input'
set firewall ipv4 input filter default-action 'drop'
set firewall ipv4 input filter rule 10 action 'accept'
set firewall ipv4 input filter rule 10 state 'established'
set firewall ipv4 input filter rule 11 action 'accept'
set firewall ipv4 input filter rule 11 state 'related'
set firewall ipv4 input filter rule 20 action 'accept'
set firewall ipv4 input filter rule 20 protocol 'icmp'
set firewall ipv4 input filter rule 25 action 'accept'
set firewall ipv4 input filter rule 25 protocol 'tcp'
set firewall ipv4 input filter rule 25 destination port '22'
set firewall ipv4 input filter rule 25 source address '10.20.9.0/24'
set firewall ipv4 input filter rule 30 action 'accept'
set firewall ipv4 input filter rule 30 protocol 'ospf'
set firewall ipv4 input filter rule 31 action 'accept'
set firewall ipv4 input filter rule 31 protocol 'tcp'
set firewall ipv4 input filter rule 31 destination port '179'

Illustrative output

show firewall shows the input filter default-drop counter advancing at roughly one packet per second on each router. show vrrp detail on rtr-ha-b records its transition to master at 14:07 on Tuesday. On the access switch, the virtual MAC 00:00:5e:00:01:0a appears against two ports.

Work the evidence before reading on

Both routers are up, both are healthy, and neither has a complaint to make.

  1. The lesson’s first cause for split brain is a layer-2 partition - the routers cannot hear each other. One command settles that. Run it in your head against the capture above and say what it eliminates.
  2. The capture also carries the fields that a parameter mismatch would show. Name them, read them off the output, and cross off what they rule out.
  3. You have now eliminated the wire and the packet contents, and the advertisements are still not being acted on. Where in a Linux router can a packet be, such that a capture on the interface sees it and the process that cares about it does not?
  4. Short exchanges work and long ones fail. What does each router hold that the other does not, and what happens to a flow the moment it changes routers?
  5. Both routers report MASTER and the HA dashboard is green. Write down what the dashboard’s check actually asserts, then write down what you would have to ask to detect this.
  6. OSPF adjacencies never dropped, and that is what the change was signed off on. Look at where VRRP lives in the configuration tree compared with OSPF and BGP, and explain the omission without blaming anybody.

Before continuing: split brain is a property of the pair, not of a router. Every command run so far has been addressed to one member. Which question can only be answered by looking at both?

Root cause

1. Protocol 112 is not on the allow-list

VRRP advertisements are not TCP and not UDP. They are carried directly in IP as protocol 112, addressed to the multicast group 224.0.0.18, with a TTL of 255 so they cannot leave the segment. They are delivered locally to the router rather than forwarded through it, so the chain that evaluates them is the input chain - the one the hardening change converted to default-drop.

The allow-list covers established and related traffic, ICMP, SSH from the management prefix, OSPF and BGP. It does not cover protocol 112. From the moment each router committed, it stopped accepting its partner’s advertisements.

The backup did exactly what RFC 5798 requires of it. Having heard nothing for longer than the master-down interval, it concluded the master was gone, took the virtual IP, sent a gratuitous ARP, and entered MASTER. That is not a bug and not a race; it is the protocol working correctly on the only information it was allowed to have.

The primary never noticed. It was dropping the same advertisements in the same way - and even if it had heard them, a master that receives a lower-priority advertisement keeps the role. There is no state in which this pair recovers on its own.

2. The packets were visible and unreachable at the same time

This is the part worth carrying to every other Linux-router incident.

tcpdump attaches to the interface. On the receive path that tap sits before the netfilter hooks, so a capture shows every frame the NIC delivered, whether or not the firewall later dropped it. The advertisements are genuinely on the wire, genuinely received by the kernel, and genuinely never delivered to the daemon that was waiting for them.

That looks like a contradiction and it is the most useful diagnostic gap the platform has. Read it as a sequence:

  • A capture shows nothing. The packet did not arrive. Suspect the wire, the switch, the VLAN, the sender.
  • A capture shows it and the process ignores it. The packet arrived and something between the NIC and the process consumed it, or the process rejected it on its contents.
  • A capture shows it, the contents agree, and the process still does not act. The packet arrived and something between the NIC and the process consumed it. That is a very short list, and on a router that has just had its input chain rewritten it is a list of one.

The second and third bullets are separated by reading the fields in the capture you already have - version, vrid, priority, authtype - which is why this incident collapses in two commands once somebody runs tcpdump instead of comparing configurations.

3. Two masters, one virtual MAC, and a switch that cannot know

Both routers now own 10.20.0.1 and both answer ARP for it with the same virtual MAC, 00:00:5e:00:01:0a, because the virtual MAC is derived from the VRID and the VRID matches.

The access switch learns that MAC from whichever router last sent a frame with it as the source, and relearns it when the other one does. From the LAN’s point of view the default gateway is behind a port that changes every second or so. Frames to the gateway land on rtr-ha-a or rtr-ha-b depending on nothing the client can influence.

The switch reported this accurately, from the first hour, as MAC flapping on a 00:00:5e address. That address is only ever supposed to be behind one port at a time. The alarm was correct and it was dismissed.

4. Why short things work and long things break

Each router keeps its own connection-tracking table and NATs to its own upstream address. Neither shares any state with the other.

A DNS lookup, an ICMP echo, a small object fetch - each of these completes within one or two round trips, almost always on whichever router it started on. Nothing is wrong with them and nothing will be.

A long flow eventually straddles a relearn. The packet that lands on the other router finds no state for the connection, gets source-NATed as a brand new one to that router’s own public address, and arrives at a server that has been talking to a different address for the last ten minutes. The server rejects it or drops it, and the flow stalls or resets.

That is the VPN tunnels, the stalled uploads and the half-rendered pages, all at once. The SaaS logouts are the same mechanism observed from outside: a session bound to a source address that keeps alternating between 203.0.113.10 and 198.51.100.10.

Resolution

  1. Confirm the fault is the pair, not a router, before acting. Read show vrrp on both boxes within the same minute and record both outputs. Every command so far has been addressed to one member, and one member cannot report this.
  2. Decide whether you are fixing now or at an agreed window, and write it down with an owner and an end time. The repair is briefly worse than the fault for a subset of users, which makes a short hold defensible rather than a delay.
  3. Consolidate the tickets first, whichever you choose. The VPN drops, the portal stalls and the SaaS logouts are one fault, and somebody is currently chasing an application bug at a vendor who does not have one.
  4. Fix the backup first. As soon as rtr-ha-b can hear rtr-ha-a again it sees a higher-priority master, demotes itself and releases the virtual IP. That single commit ends the split brain.
  5. Expect the demotion to hurt. Connection-tracking state does not migrate: every flow currently pinned to rtr-ha-b ends at that instant. On this estate that means client VPN tunnels and in-flight uploads - the same users who have been raising tickets.
  6. Then fix the primary. This commit restores nothing you can see and matters more than the first: without it the pair still has no working election, and the next genuine failure would find rtr-ha-a unable to observe anything at all.
  7. Scope the rule rather than blanket-permitting the protocol. Inbound interface, source address of the peer, destination 224.0.0.18. Accepting protocol 112 from anywhere on any interface reintroduces a spoofing surface on a router that was being hardened for a reason.
  8. Read the change with compare and commit with commit-confirm. You are editing the input chain of a router you are reachable on; a mistake here locks you out, and the confirm window is what turns that into a five-minute wait rather than a site visit.
  9. Do not roll back the hardening change. The defect is one missing rule inside it, and reverting the whole thing to make a symptom disappear discards work that is otherwise correct.
# On rtr-ha-b first - the backup. This commit ends the split brain.
# The matching change on rtr-ha-a restores the election itself.
set firewall ipv4 input filter rule 40 description 'VRRP advertisements from the HA peer'
set firewall ipv4 input filter rule 40 action 'accept'
set firewall ipv4 input filter rule 40 protocol '112'
set firewall ipv4 input filter rule 40 inbound-interface name 'eth1'
set firewall ipv4 input filter rule 40 source address '10.20.0.252'
set firewall ipv4 input filter rule 40 destination address '224.0.0.18'
compare
commit-confirm 5

Verification

  1. Ask the pair, not the members. show vrrp on both routers within the same minute must show exactly one MASTER and one BACKUP. Read it again several minutes later, so a transition caught mid-cycle is not mistaken for a stable state.
  2. The rule is matching, not merely present. The packet counter on the new input rule is incrementing on both routers, and the input filter default-drop counter has stopped advancing at one packet per second.
  3. The wire agrees. A capture on each router still shows both advertisement streams - that never changed - and the state above proves they are now being acted on rather than dropped.
  4. The switch sees one gateway. The virtual MAC 00:00:5e:00:01:0a appears against exactly one port, and no further flapping entries appear in the switch log.
  5. The redundancy works, which is a larger claim than the symptom being gone. In an agreed window, take the primary out of the group and confirm the backup takes over within the master-down interval and sends its gratuitous ARP; then restore the primary and confirm the backup steps back down. Nobody here has had a working failover for three days.
  6. The user-visible fault is gone, proven from a user machine: a file transfer of several minutes completes, a client VPN tunnel survives a coffee break, and an external service reports one stable source address rather than two alternating ones.
  7. The hardening still hardens. The default drop is still counting, and the new rule is scoped to the peer address, the interface and the multicast destination rather than accepting protocol 112 from anywhere.
  8. The other routers in the estate are checked, not assumed. The same allow-list was pushed everywhere; any other VRRP pair in the fleet has the same defect and the same three days of it.

Prevention

  • Derive a default-deny input allow-list from what the router speaks, not from what it listens on. Ports are easy to enumerate and are the part every template covers. The protocols that keep a router in a cluster ride directly in IP and have no ports to find: OSPF as 89, VRRP as 112, ESP as 50, plus IKE and NAT-T on their UDP ports. Generate the list from the configuration rather than writing it by hand.
  • Alert on VRRP master count across the pair. One master per group, asserted over both members. It is a two-line check over show vrrp json from each router and it would have fired four seconds after the Tuesday commit. Add transition count alongside it, which catches the flapping cases a count check cannot.
  • Retire the dashboard check that asks the primary whether it is master. That assertion is true in a healthy pair and true in the worst failure this pair can have, which makes it worse than no check at all.
  • Stagger control-plane changes across HA members and check the pair between commits. Rolling both out in one window removed the only comparison that would have exposed this within a minute.
  • Validate a hardening change against control-plane state. Adjacency counts, cluster membership, tunnel state - not whether SSH still works, which is the one thing the engineer running the change has already proved by being logged in.
  • Treat MAC flapping on an address in the IANA VRRP range as a page. That address is only ever meant to exist behind one port. The switch diagnosed this correctly on day one and was overruled.
  • Write the estate’s protocol inventory down once, so the next hardening change has a list to check against instead of a section of the configuration tree to skim.