Skip to main content
RunBook Academy

← All break/fix scenarios in VyOS

advancedvyos-interface~30 min

VLAN Trunk Misconfigured

Reported symptoms

  • ●Wireless access points at the Northgate campus have dropped to standalone mode and cannot reach the controller in the DC, so roaming fails and two SSIDs have disappeared
  • ●The nightly replication job from the campus NAS to the DC failed on Saturday and Sunday with connection timeouts, and has been raised as a storage fault
  • ●The camera NVR is recording locally but has not written anything off-site since Saturday morning
  • ●The NOC cannot SSH to the router on its management address, but the same router answers on its loopback and every monitoring check against it is green
  • ●Devices on the management VLAN can ping each other perfectly well, which is why the first responder ruled out a VLAN problem in the first two minutes
  • ●Desktops on the data VLAN and phones on the voice VLAN are completely unaffected, over the same cable, through the same switch port
  • ●The number of affected devices grew through Monday morning rather than appearing all at once, which reads like a failing component rather than a configuration change

Evidence

  • · `show interfaces ethernet eth0` shows the trunk up with no errors and packet counters advancing in both directions
  • · `show interfaces ethernet eth0 vif 130` shows the sub-interface up, correctly addressed, with a description that matches the design
  • · `show ip route` contains all three connected routes, including 10.20.30.0/24 on eth0.130, with the FIB markers set
  • · `ip link show` lists eth0.10@eth0, eth0.20@eth0 and eth0.130@eth0 - the site VLAN table lists 10, 20 and 30
  • · `ip -s link show eth0.130` shows TX packets advancing and an RX packet counter that has not moved since the interface was created
  • · `tcpdump -i eth0 -e -n vlan 130` captures nothing at all over a two-minute run
  • · `tcpdump -i eth0 -e -n vlan 30` captures a steady stream of ARP requests for 10.20.30.1 that nobody answers
  • · `show dhcp server leases` shows no lease issued in the management pool since Saturday 02:10, while the data and voice pools are busy
  • · `show system commit` records exactly one commit, timestamped Saturday 02:10, against a planned NIC replacement
  • · `show ip ospf neighbor` shows the core adjacency on eth1 in Full, which is why every device-level monitoring check reads healthy
Diagnosis and resolutionclick to reveal

Root cause

The management VLAN sub-interface was rebuilt as `vif 130` where the site uses VLAN 30. Both are valid VLAN IDs, so the commit succeeded, the kernel created a netdev named eth0.130, the address landed on it, and a connected route for 10.20.30.0/24 was installed and selected. Every status command on the router therefore reports a healthy interface, because every one of them is reporting on an interface that is genuinely healthy - it is simply attached to a VLAN that nothing else on the campus uses. The failure is bidirectional and breaks at two different boxes for two different reasons. Frames the router emits leave the trunk tagged 130, and the switch drops them because 130 is not in the trunk's allowed VLAN list. Frames the switch sends tagged 30 arrive on the parent interface and are discarded by the 802.1Q driver, because there is no eth0.30 netdev to deliver them to. What makes the incident last a weekend is that a VLAN sub-interface has no peer and negotiates nothing: unlike an OSPF adjacency or an LACP bundle, there is no far end that can disagree with it and no state machine that can stop short. Its link state simply follows the parent, so `up` is a statement about the local box and nothing else. The uneven symptom set is the estate showing which traffic needs the router. Host-to-host traffic inside VLAN 30 never reaches the router and worked throughout, which is what convinced the first responder the VLAN was fine. Everything that leaves the VLAN - the access points reaching their controller, the NAS reaching the DC, the NVR reaching its off-site target - depends on a default gateway that stopped existing. And the apparent spread through Monday morning was the DHCP lease timer, not a progressing fault: renewal is unicast to a server address that is no longer reachable from the VLAN, so each device kept its address until expiry and then lost it.

Remediation

The correction is one VLAN number on one router, and the work around it is what stops the fix from creating a second incident. Read the trunk's allowed VLAN list off the switch before typing anything - the switch is the authority on what the VLAN actually is, and the wiki table that lists two campuses in adjacent rows is what caused this. Then delete the wrong sub-interface, create the right one with the same address and a description, and move every object that named the old interface by name along with it. On a 1.5 router that means at least the firewall interface group the forward-filter jump keys on: a sub-interface that is not in the group is not covered by the management policy, and depending on how the base chain is written it will either be silently dropped or silently unfiltered, and both are worse than the outage you are repairing. The DHCP server needs no interface change, because VyOS derives the listener from the subnets rather than from an interface binding - which is exactly why it never complained: the 10.20.30.0/24 subnet did match an interface address, just one on the wrong VLAN. Read the whole change with `compare` before committing, and commit under `commit-confirm`. The confirm window is not for the VLAN edit, which cannot lock you out because you are reachable over the core rather than over this VLAN; it is for the firewall group edit, which touches a chain that the two working VLANs also traverse. Fixing immediately is the defensible choice here and worth recording as a decision rather than an assumption: the change does not disturb VLAN 10 or 20, the affected services are degraded to the point of data loss for the off-site camera recordings, and there is no established-session cost to weigh because nothing on the VLAN currently has a working path off it. If you do hold, the hold needs a named owner, an end time, and a note on all four tickets so the storage and wireless teams stop investigating faults they do not have.

Verification

Verify on the wire and from the VLAN, because every check that runs on the router alone passed while the site was down. Start with the netdev name - `ip link show` must list eth0.30@eth0 and must not list eth0.130 - and then prove that frames are moving: `tcpdump -i eth0 -e -n vlan 30` shows tagged traffic and, more specifically, shows the ARP requests for 10.20.30.1 now drawing replies. The counter check is the one worth keeping: `ip -s link show eth0.30` must show an RX packet count that is advancing, not merely non-zero, because a frozen RX counter on a sub-interface whose parent is busy is the single cheapest signature of this entire fault class and is scriptable across every sub-interface on the estate. Then leave the router. From a host on VLAN 30, ping the gateway, then reach something on the far side of the core; the ticket was never about router state. Confirm DHCP has resumed with `show dhcp server leases`, looking for fresh timestamps in the management pool rather than for the pool's mere existence, and check `show dhcp server statistics` for counters that move. Confirm the policy followed the interface: `show firewall group` lists eth0.30, and the forward-filter rule that jumps to the management rule set is incrementing. Finally confirm that nothing else moved - VLAN 10 and 20 counters unchanged, the OSPF adjacency on eth1 never reset, `show ip route` otherwise identical - and get the service owners to close their own tickets from their own side: the controller listing its access points, the replication job completing, the NVR writing off-site again.

Prevention

Make traffic, not link state, the acceptance criterion for an interface rebuild. A `vif` sub-interface comes up because its parent is up; there is no peer to disagree with it, so `up` proves only that the command committed. The check that would have caught this at 02:15 is one line - read the RX packet counter on every sub-interface and require that all of them are advancing - and it belongs in the build record next to the `show ip route` output that was captured instead. Alert on the same thing continuously: a sub-interface with zero received packets over a window is nearly always either a VLAN-id mistake on the router or a pruned VLAN on the switch, and the alert costs nothing and catches both. Validate from the VLAN under test rather than from whichever VLAN the engineer's laptop is patched into; the Saturday sign-off pinged the gateway successfully from the data VLAN and could not have detected this fault. Generate the interface stanza from a single machine-readable VLAN table rather than retyping it from a wiki page that lists two campuses in adjacent rows, and put a `description` on every sub-interface so a diff review has something to compare against a name. Keep one list of everything that refers to an interface by name - firewall interface groups, monitoring targets, capture filters - so that moving an interface moves them with it. And retire the reading that started the incident: devices on a VLAN reaching each other is a statement about the switch, never about the router, and it eliminates nothing.

Reported symptoms

It is Monday morning at the Northgate campus and there are four tickets open, against four different teams, and not one of them says the word VLAN.

The wireless team has the loudest one. Access points across the building have dropped to standalone mode. They cannot reach the controller in the DC, roaming between them fails, and the two SSIDs that are defined centrally have simply disappeared.

The storage team has one from the weekend. The nightly replication job from the campus NAS to the DC failed on Saturday night and again on Sunday night, both times with connection timeouts. It is being worked as a storage fault because that is who owns the job.

Physical security has one that nobody has prioritised. The camera NVR is recording locally, so nothing looks broken on the console, but it has not written anything to the off-site target since Saturday morning. That is the ticket with a retention deadline behind it.

And the NOC has one that contradicts itself. Nobody can SSH to the router on its management address. The same router answers on its loopback, every monitoring check against it is green, and the OSPF adjacency to the core has not flapped once all weekend. The router is simultaneously up and unreachable, depending on who you ask.

Two more facts are in play and nobody has connected them.

Devices on the management VLAN can ping each other. The NAS sees the NVR, the NVR sees the access points, everything on that subnet answers everything else on that subnet. That is what let the first responder rule out a VLAN problem inside two minutes.

And the fault appeared to spread. On Saturday it was the NAS and the cameras. By Monday morning it was most of the access points, and the count kept climbing through the morning. A fault that widens on its own hour by hour reads like failing hardware, not like a change.

The estate

rtr-dist-2 is the distribution router for the campus, a VyOS 1.5 LTS box in a router-on-a-stick configuration.

  • eth0 is the trunk to the access switch stack sw-acc-1. Three VLANs cross it: 10 (DATA), 20 (VOICE), 30 (MGMT).
  • eth0.10 holds 10.20.10.1/24, eth0.20 holds 10.20.20.1/24, and the management sub-interface holds 10.20.30.1/24.
  • eth1 is the uplink to the routed core, in OSPF area 0. Every campus prefix is advertised from here, and the loopback 10.255.0.2/32 is the address the NOC’s monitoring actually polls.
  • The router is the DHCP server for all three VLANs, and the default gateway for all three.

The management VLAN carries the things nobody thinks about until they stop: the access points, the NAS, the NVR, the console server, and the router’s own management address.

One change happened. On Saturday at 02:10 the onboard NIC that carried eth0 was replaced under a planned maintenance window, and because the interface name changed, the sub-interface block was retyped by hand. The engineer worked from the network wiki, whose VLAN table lists Northgate and the Eastfield campus in adjacent rows. Eastfield’s management VLAN is 130.

The handover at 02:40 records: all interfaces up, all connected routes present, core adjacency Full, gateway ping successful from a test laptop. Every one of those statements is true.

Evidence provided

Read-only / Safethe trunk is healthy and busy in both directions
vyos@rtr-dist-2:~$ show interfaces ethernet eth0
eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
  link/ether 52:54:00:aa:11:02 brd ff:ff:ff:ff:ff:ff
  RX:  bytes  packets  errors  dropped  overrun  mcast
  41822901744  38104112   0       0        0     902144
  TX:  bytes  packets  errors  dropped  carrier  collisions
  39118220913  35990041   0       0        0        0

Illustrative output

Read-only / Safethree sub-interfaces, all up; read the names against the site VLAN table
vyos@rtr-dist-2:~$ ip link show
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
4: eth0.10@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
5: eth0.20@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
6: eth0.130@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

Illustrative output

Read-only / Safethe management prefix is present, selected, and installed in the FIB
vyos@rtr-dist-2:~$ show ip route connected
C>* 10.20.10.0/24 is directly connected, eth0.10, 2d09h12m
C>* 10.20.20.0/24 is directly connected, eth0.20, 2d09h12m
C>* 10.20.30.0/24 is directly connected, eth0.130, 2d09h12m
C>* 10.0.0.8/30 is directly connected, eth1, 2d09h12m

Illustrative output

Read-only / Safethirty thousand packets sent, none ever received
vyos@rtr-dist-2:~$ ip -s link show eth0.130
6: eth0.130@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
  link/ether 52:54:00:aa:11:02 brd ff:ff:ff:ff:ff:ff
  RX: bytes  packets  errors  dropped  missed  mcast
  0          0        0       0        0       0
  TX: bytes  packets  errors  dropped  carrier  collisions
  1904688    30328    0       0        0        0

Illustrative output

Read-only / Safethe campus is asking for its gateway, on the wire, and nothing replies
vyos@rtr-dist-2:~$ tcpdump -i eth0 -e -n vlan 30 -c 5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:14:02.118 b4:fb:e4:21:9c:07 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), vlan 30, ethertype ARP, Request who-has 10.20.30.1 tell 10.20.30.41, length 46
09:14:03.121 b4:fb:e4:21:9c:07 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), vlan 30, ethertype ARP, Request who-has 10.20.30.1 tell 10.20.30.41, length 46
09:14:04.126 b4:fb:e4:21:9c:07 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), vlan 30, ethertype ARP, Request who-has 10.20.30.1 tell 10.20.30.41, length 46

Illustrative output

The same capture with vlan 130 in place of vlan 30 returns nothing over a two-minute run. show dhcp server leases shows no lease issued in the management pool since Saturday 02:10, while the data and voice pools are busy. show system commit records one commit, Saturday at 02:10.

Work the evidence before reading on

Everything on this router is up, and the router has no complaint to make about anything.

  1. Three connected routes are present, selected and installed. What does a connected route prove about the far end of the cable? Be precise: name what has to be true for a connected route to appear, and what does not.
  2. ip link show prints the netdev names. Read them against the site VLAN table before you read anything else on the box. This step is free and it is the whole incident.
  3. tcpdump -e -n vlan 130 returns nothing. There are at least two different worlds that produce that result. Name them, and name the single command that separates them.
  4. Hosts on the management VLAN reach each other and reach nothing else. Which device does intra-VLAN traffic touch, and which device does inter-VLAN traffic touch? What does the answer eliminate?
  5. VLAN 10 and VLAN 20 are perfect, over the same cable, through the same switch port, on the same trunk. What does that do to every hypothesis about the physical link, the NIC replacement, and the trunk as a whole?
  6. The blast radius grew hour by hour on Monday morning. What in this estate has a timer measured in hours, and what does it do when it expires?

Before continuing: a VLAN sub-interface has no peer. Work out what that means for the value of every show command on this router, then decide where the evidence has to come from instead.

Root cause

1. A sub-interface is a filter, and this one was set to the wrong channel

The management sub-interface was created as vif 130. The site uses VLAN 30.

Both are legal VLAN IDs - the range is 1 to 4094 - so nothing rejected it. The commit created a kernel netdev named eth0.130, attached the address to it, and the address produced a connected route which FRR selected and installed. Every subsequent status command reported a healthy interface, and every one of them was right. The interface is healthy. It is attached to a VLAN that nothing else on the campus uses.

The failure is bidirectional, and it breaks at a different box in each direction:

  • Outbound. Frames the router emits on eth0.130 leave the trunk tagged 130. The switch drops them, because 130 is not in the trunk’s allowed VLAN list.
  • Inbound. Frames the switch sends tagged 30 arrive on eth0. The 802.1Q driver looks for a netdev registered for VLAN 30, finds none, and discards them. That is why the RX counter on eth0.130 has never moved: nothing is tagged 130, so nothing is ever delivered to it.

The capture is the proof of both halves at once. The ARP requests for 10.20.30.1 are on the wire, correctly tagged 30, arriving at the router’s port - and the router never answers them, because as far as its IP stack is concerned those frames were never addressed to any interface it owns.

2. Why every check on the router passed

This is the part worth carrying beyond VLANs.

A VLAN sub-interface negotiates nothing. There is no peer, no adjacency, no state machine, no far end that can disagree. Its link state simply follows its parent, so eth0.130 is up for exactly one reason: eth0 is up. There is no state in which a sub-interface reports a VLAN-id mismatch, because there is nothing it could compare against.

Compare that with the protocols on the same router. An OSPF adjacency that disagrees with its neighbour about an area or a timer stops in a specific state and tells you so. An LACP bundle whose partner is misconfigured refuses to aggregate and logs it. Those subsystems can report a mismatch because they are talking to somebody.

So the handover checklist was not lazy. It was asking questions that this fault class is structurally incapable of answering. “All interfaces up” and “all connected routes present” are both true on a router with every VLAN id wrong.

3. Why the VLAN looked alive

Intra-VLAN traffic never touches the router. A host on 10.20.30.0/24 ARPs for another host on 10.20.30.0/24, the switch forwards the frame within VLAN 30, and the two talk directly. That path was working all weekend and is still working now.

Only two things on that VLAN need the router: anything leaving the subnet, and DHCP. Both were dead from 02:10.

So “we can ping each other, the VLAN is fine” is a true statement about the switch and an empty one about the router. It eliminated nothing, and it cost the incident about thirty hours.

4. The spread was a lease timer

The fault did not widen. The consequences of it arrived on a schedule.

DHCP renewal is unicast to the server’s address, which is on the router, which is unreachable from the VLAN. Renewal at T1 failed silently on every device, and each device kept using its existing address until the lease actually expired - at which point it lost the address entirely and started broadcasting DISCOVERs that also went nowhere.

So devices fell off in the order their leases ran out, spread across Monday morning, which is precisely the shape of a failing component and precisely not the shape of a Saturday change. The timer was doing the pacing.

Resolution

  1. Read the trunk configuration off the switch before typing anything on the router. The allowed VLAN list on sw-acc-1 is the authority on what the management VLAN actually is; the wiki page that lists two campuses in adjacent rows is what caused this, and consulting it again is not a check.
  2. Decide explicitly whether you are fixing now or holding, and write the decision down. Fixing now is the defensible choice here: the edit does not touch VLAN 10 or 20, nothing on the management VLAN currently has a working path off it so there are no established sessions to disturb, and the camera off-site recording gap has a retention deadline behind it.
  3. Annotate all four tickets before you start. The storage and wireless teams are investigating faults they do not have, and that continues until somebody tells them otherwise.
  4. Stage the change without committing: delete the wrong sub-interface, create the right one with the same address, and give it a description.
  5. Move everything that named the old interface by name. On a 1.5 router the firewall rule sets are reached by a jump from the base chain, keyed on the inbound interface or on an interface group, so a sub-interface that is not in the group is not covered by the policy. Leaving that behind converts one outage into a quieter one.
  6. Leave the DHCP server alone. VyOS derives the listener from the subnets rather than from an interface binding, which is exactly why it never complained: the declared subnet did match an interface address, just one on the wrong VLAN. Once the address is on eth0.30 the same subnet matches the same way.
  7. Read the whole change with compare before you commit it. A second transposed digit here costs another weekend, and the diff is the last cheap place to see it.
  8. Commit with commit-confirm. The confirm window is not protecting you from the VLAN edit - you are reachable over the core, not over this VLAN - it is protecting the two working VLANs from the firewall group edit, which touches a chain their traffic also traverses.
  9. Watch the wire, not the configuration, before you confirm. Run the VLAN 30 capture again and wait for the ARP requests to start drawing replies.
  10. Save once traffic is flowing, and only then close the confirm window.
# On rtr-dist-2, in configuration mode. Confirm the VLAN id against the
# switch trunk before pasting anything.
delete interfaces ethernet eth0 vif 130
set interfaces ethernet eth0 vif 30 address '10.20.30.1/24'
set interfaces ethernet eth0 vif 30 description 'NORTHGATE-MGMT'
delete firewall group interface-group MGMT-IF interface eth0.130
set firewall group interface-group MGMT-IF interface eth0.30
compare
commit-confirm 5

Verification

  1. The kernel names the netdev you expect. ip link show lists eth0.30@eth0 and does not list eth0.130. This is the first check because it is the fault, stated directly.
  2. Frames are arriving. tcpdump -i eth0 -e -n vlan 30 shows tagged traffic, and specifically shows the ARP requests for 10.20.30.1 now drawing replies. The capture is the only evidence in this list that is independent of what the router believes about itself.
  3. The receive counter is advancing. ip -s link show eth0.30 twice, a minute apart, with a higher RX packet count the second time. Advancing, not merely non-zero: a counter that moved once during the commit and stopped is not a working interface.
  4. The test runs from the VLAN, not from the router. On a host on 10.20.30.0/24, ping the gateway, then reach something on the far side of the core. Nobody filed a ticket about router state.
  5. DHCP is issuing again. show dhcp server leases shows leases with fresh timestamps in the management pool, and show dhcp server statistics shows counters moving. The pool existing was never the question.
  6. The policy followed the interface. show firewall group lists eth0.30, and the forward-filter rule that jumps to the management rule set is incrementing. An interface that is quietly outside every rule set is the second defect this change could have introduced.
  7. Nothing else moved. VLAN 10 and 20 counters are still advancing, the OSPF adjacency on eth1 did not reset, and the rest of show ip route is unchanged.
  8. The service owners close their own tickets from their own side: the controller lists its access points, the replication job completes, the NVR writes off-site. Devices whose leases expired will recover on their own DISCOVER; a handful may need a bounce, and how each vendor behaves after an address loss is not something this runbook can promise.

Prevention

  • Make traffic the acceptance criterion for an interface rebuild, not link state. A vif comes up because its parent is up. Requiring an advancing RX counter on every sub-interface is one line, and it belongs in the build record next to the show ip route output that got captured instead.
  • Alert on a sub-interface with zero received packets over a window. It is cheap, it is scriptable from the same counters, and it catches this entire fault class - the router-side VLAN id mistake and the switch-side pruning that looks identical from the router.
  • Validate from the VLAN under test. The Saturday sign-off pinged a gateway successfully and could not have found this, because the laptop was on the one segment that was fine.
  • Generate the interface stanza from one machine-readable VLAN table. A wiki page listing two campuses in adjacent rows is not a source of truth, it is a transposition waiting for a 02:10 maintenance window.
  • Put a description on every sub-interface. A diff review can catch eth0.130 described as NORTHGATE-MGMT. It cannot catch a sub-interface with no description at all, because there is nothing to contradict.
  • Keep one list of everything that names an interface. Firewall interface groups, monitoring targets, capture filters, scripts. Moving an interface has to move them too, and the list is the only thing that makes that reliable.
  • Retire the reading that started this. Hosts on a VLAN reaching each other is a statement about the switch. It says nothing about the router and it eliminates nothing.