VRRP telemetry — master/backup state, gratuitous ARP, and failover event auditing

A “VRRP failover” report is not a VRRP issue until the operator has master/backup state, transition count, and gratuitous ARP evidence. VRRP (Virtual Router Redundancy Protocol) is the canonical HA pattern for first-hop redundancy: two or more routers share a virtual IP, and one router is the master while the others are backups. The operator who cannot read the VRRP state in five minutes will spend an hour guessing whether the failover is a false positive, a network-layer issue, or a real master failure.

This lesson is the production reference for VRRP telemetry on VyOS 1.5 LTS: the four transport options (show vrrp JSON, gNMI, SNMP, custom textfile), what each one gives the operator, what each one costs, and the validation discipline that prevents the VRRP telemetry layer from becoming the next source of incidents.

The four transports

flowchart LR
  subgraph ROUTER["VyOS 1.5 router (HA pair)"]
    K["keepalived daemon<br/>VRRP state, transitions"]
  end
  subgraph COLLECT["Collection plane"]
    VRRP["show vrrp json"]
    GNMI["gNMI telemetry<br/>subscription"]
    SNMP["SNMP vrrpMIB<br/>(RFC 6527)"]
    TF["Custom textfile<br/>collector"]
  end
  subgraph CONSUME["Consumption plane"]
    TS["Time-series DB<br/>(Prometheus)"]
    NMS["NMS / Grafana"]
  end
  K --> VRRP
  K --> GNMI
  K --> SNMP
  K --> TF
  VRRP --> TS
  GNMI --> TS
  SNMP --> NMS
  TF --> TS

show vrrp json is the canonical ad-hoc inspection pattern. The JSON output is machine-readable, the operator can parse it with jq, and the operator can write automation playbooks that alert on master/backup transitions. The cost: the operator must poll (the keepalived daemon does not push), and the polling rate is limited by the user’s patience.

gNMI telemetry is the streaming-evidence alternative. The operator subscribes to the keepalived YANG model and receives state updates on a configurable cadence. The cost: gNMI requires the vyos-gnmi package, and the keepalived YANG model is still maturing.

SNMP vrrpMIB (RFC 6527) is the legacy alternative. The vrrpNodeState, vrrpMasterState, vrrpOperMasterIpAddr, vrrpStatsMasterTransitions provide aggregate state. The cost: the vrrpMIB does not provide per-VRRP-instance tracked-interface state, and the counters are 32-bit.

Custom textfile collector is the modern pattern for derived metrics. The operator writes a shell script that extracts the transition count and the time-since-last-transition from show vrrp json and writes them in Prometheus text format. The cost: the operator must write and maintain the script.

The canonical VRRP state

The operator must know which VRRP state structures to monitor. The four canonical state structures:

flowchart TD
  subgraph VRRP_STATE["VRRP state"]
    NM["Node state<br/>(Initialize, Backup, Master)"]
    NB["VRRP instance state<br/>(per-group: state, priority, virtual IP)"]
    TI["Tracked interface state<br/>(per-tracked-object: weight, threshold)"]
    EV["Event state<br/>(transitions, last transition, gratuitous ARP)"]
  end
  NM --> NB
  NB --> TI
  TI --> EV

Node state is the keepalived daemon’s overall state. The operator monitors show vrrp summary to confirm the keepalived daemon is running and the VRRP instance is loaded.

VRRP instance state is the per-VRRP-group state. The operator monitors show vrrp to inspect the per-group state (Master, Backup, Initialize), the priority, the virtual IP, the advertisement interval, and the preemption flag. A group that is bouncing between Master and Backup indicates a network-layer issue (the advertisements are not arriving) or a priority issue (the priority is changing).

Tracked interface state is the per-tracked-object state. The operator monitors show vrrp track to inspect the tracked interfaces and the corresponding priority decrement. A tracked interface that transitions to down should decrement the priority and trigger a failover if the new priority is below the peer’s priority.

Event state is the transition history. The operator monitors the transition count and the timestamp of the last transition. A spike in transitions indicates a flapping link or a misconfigured priority. The operator also monitors the gratuitous ARP count: a gratuitous ARP is sent by the new master to update the upstream router’s MAC table; if the gratuitous ARP is delayed or missed, the failover is slow.

Step 1 — Inspect show vrrp json

The operator uses show vrrp json to inspect the VRRP state in a machine-readable format. This is the canonical pattern for ad-hoc inspection and automation.

$ show vrrp json | jq .
{
  "r1": {
    "groups": [
      {
        "name": "default",
        "interface": "eth0",
        "state": "MASTER",
        "virtual_ip": "10.0.0.1/24",
        "virtual_mac": "00:00:5e:00:01:01",
        "priority": 100,
        "advert_interval": 1,
        "preempt": true,
        "tracking": {
          "tracked_interface": "eth1",
          "weight": 10,
          "status": "up"
        }
      }
    ]
  },
  "r2": {
    "groups": [
      {
        "name": "default",
        "interface": "eth0",
        "state": "BACKUP",
        "virtual_ip": "10.0.0.1/24",
        "virtual_mac": "00:00:5e:00:01:01",
        "priority": 90,
        "advert_interval": 1,
        "preempt": true,
        "tracking": {
          "tracked_interface": "eth1",
          "weight": 10,
          "status": "up"
        }
      }
    ]
  }
}

The output is JSON. The operator uses jq to extract the relevant fields:

$ show vrrp json | jq '.r1.groups[] | {name, state, priority, virtual_ip}'

The operator can also inspect the transition count by enabling the keepalived daemon’s --log-detail flag and parsing the log file. The keepalived log records every state transition with a timestamp:

$ tail -f /var/log/keepalived.log
VRRP_Instance(default) Transition to MASTER STATE
VRRP_Instance(default) Entering MASTER STATE
VRRP_Instance(default) Sending gratuitous ARP on eth0 for 10.0.0.1

The discipline: use show vrrp json for the canonical state and the keepalived log for the transition history.

Step 2 — Stream VRRP state via gNMI

The operator enables the gNMI server and subscribes to the keepalived YANG model:

configure
set service gnmi listen-address '10.0.0.1'
set service gnmi port '57400'
set service gnmi authentication username 'gnmi-user'
set service gnmi authentication password 'GNm1-2026-Q3'
commit

The operator then uses a gNMI client to subscribe to the VRRP state:

$ gnmic -a 10.0.0.1:57400 -u gnmi-user -p 'GNm1-2026-Q3' \
  subscribe --path '/high-availability/vrrp' \
  --stream-mode sample --sample-interval 10s

The stream produces VRRP state updates every 10 seconds. The operator feeds the stream into a time-series database. The cost: gNMI requires the vyos-gnmi package, and the keepalived YANG model is still maturing.

Step 3 — Configure SNMP vrrpMIB

The operator enables SNMPv3 and the vrrpMIB on the router:

configure
set service snmp v3 engineid '0x80004f7e9c3a1b2c'
set service snmp v3 group default mode 'ro'
set service snmp v3 view full oid '1' include
set service snmp v3 user monitoring-auth group default auth plaintext-key 'AuthPass-2026-Q3'
set service snmp v3 user monitoring-auth group default auth type 'sha'
set service snmp v3 user monitoring-auth group default privacy plaintext-key 'PrivPass-2026-Q3'
set service snmp v3 user monitoring-auth group default privacy type 'aes'
commit

The operator then queries the vrrpMIB:

$ snmpwalk -v3 -u monitoring-auth -l authPriv -a SHA -A 'AuthPass-2026-Q3' -x AES -X 'PrivPass-2026-Q3' 10.0.0.1 vrrpNodeState
VRRP-MIB::vrrpNodeState.0 = INTEGER: master(2)

The vrrpMIB provides aggregate state (node state, master state, transition count) but not per-instance tracked-interface state. The operator uses vrrpMIB for the NMS-level view and the show vrrp json for the per-instance detail.

Step 4 — Configure the custom textfile collector

The operator writes a shell script that extracts the transition count and the time-since-last-transition from the keepalived log and writes them in Prometheus text format.

#!/bin/bash
TEXTFILE=/var/lib/node_exporter/textfile/vrrp-stats.prom
TMPFILE=$(mktemp)

echo "# HELP vyos_vrrp_transition_total Number of VRRP state transitions" > $TMPFILE
echo "# TYPE vyos_vrrp_transition_total counter" >> $TMPFILE
echo "# HELP vyos_vrrp_seconds_since_last Transition Time since last VRRP state transition" >> $TMPFILE
echo "# TYPE vyos_vrrp_seconds_since_last gauge" >> $TMPFILE

# Count transitions in the last 24 hours
TRANSITIONS=$(grep -c "Transition to" /var/log/keepalived.log | tail -1)
echo "vyos_vrrp_transition_total{instance=\"default\"} $TRANSITIONS" >> $TMPFILE

# Time since last transition
LAST=$(grep "Transition to" /var/log/keepalived.log | tail -1 | awk '{print $1, $2}')
LAST_EPOCH=$(date -d "$LAST" +%s 2>/dev/null || echo 0)
NOW_EPOCH=$(date +%s)
SECONDS_SINCE=$((NOW_EPOCH - LAST_EPOCH))
echo "vyos_vrrp_seconds_since_last{instance=\"default\"} $SECONDS_SINCE" >> $TMPFILE

mv $TMPFILE $TEXTFILE

The script runs on a 1-minute cron, the exporter reads the file on every scrape, and the Prometheus server sees the metric as a counter (for transition count) or a gauge (for time-since-last-transition).

Validation discipline

Every VRRP telemetry metric must be validated before it becomes an alert. The validation discipline:

1. Confirm the keepalived daemon is running. The operator runs systemctl status keepalived and confirms the daemon is active.
2. Confirm the VRRP instance is loaded. The operator runs show vrrp summary and confirms the instance is present.
3. Confirm the master/backup state is correct. The operator runs show vrrp and confirms the expected master is master and the expected backup is backup.
4. Confirm the gNMI subscription is receiving updates. The operator runs the gNMI client and confirms updates are arriving at the expected cadence.
5. Confirm the textfile collector script is producing valid output. The operator runs the script manually and inspects the output file.
6. Confirm the failover is detected. The operator triggers a failover (e.g., shut down the master) and confirms the transition count increments and the time-since-last-transition resets to 0.

Production failure modes

The VRRP telemetry failure modes the operator encounters:

• Silent failover. The keepalived daemon transitions from master to backup and back, but the operator does not have a transition log. Fix: enable the keepalived daemon’s --log-detail flag and ingest the log into a time-series database.
• Gratuitous ARP delay. The new master sends the gratuitous ARP late, so the upstream router continues to send traffic to the old master. Fix: configure the delay and preempt settings on the VRRP instance; the new master should send the gratuitous ARP within 1 second of the transition.
• Preemption flapping. The backup has a higher priority and preempts the master, but the preemption is not stable. Fix: configure preempt_delay 60 to give the network time to stabilise before the preemption.
• Tracked interface false positive. The tracked interface flaps, causing the priority to decrement and the router to give up master status. Fix: configure the tracked interface’s weight to a value that does not trigger a failover on a transient flap.
• Split-brain. Both routers are in Master state simultaneously. Fix: configure the VRRP virtual IP on both routers with the same priority and the same virtual_router_id; the router with the higher priority should win, but if both routers are in Master state, the network is split-brain and the operator must investigate the underlying network issue.
• VRRP on a virtual interface. VRRP on a VTI (Virtual Tunnel Interface) or a bond interface can be unreliable if the underlying interface transitions are not propagated. Fix: configure the VRRP tracking on the underlying physical interface, not on the virtual interface.

Rollback

VRRP telemetry changes are typically configuration-only, but the impact can be cross-cutting. The rollback discipline:

• gNMI, SNMP, textfile collector — all are in the VyOS tree or in the operator’s filesystem. The rollback is rollback N and commit (for the tree changes) or rm (for the script files).
• The custom textfile collector script — installed in /usr/local/bin. The rollback is to remove the script and the cron entry.
• Keepalived log rotation — the keepalived log can grow large; the operator must configure log rotation (/etc/logrotate.d/keepalived).

For every change, use commit-confirm:

configure
# ... make the change ...
commit-confirm 5
# If the change has unintended consequences, the auto-rollback
# fires after 5 minutes and the previous configuration is restored.

Production discipline

Cross-course references

• Part XL (vyos-xl-01-vrrp-concept) covers the VRRP protocol that the telemetry layer is built on.
• Part XL (vyos-xl-05-vrrp-tracking) covers the tracked interface state that the telemetry layer monitors.
• The Observability course covers the consumer side: Prometheus, Grafana, alerting on VRRP state deviation.
• The Ansible course’s XLII-Ansible-BeyondLinux covers the automation hand-off (rolling out the textfile collector to a fleet via a single playbook).
• The Linux course’s XXII-Linux-NetTroubleshoot covers the underlying file descriptor and TCP session primitives that gNMI uses.

Quiz