Objective
By the end of this lab you will have built a three-VLAN trunk between two VyOS 1.5 routers, proved from a capture which frames carry an 802.1Q tag and which do not, broken the trunk with a one-digit VLAN-ID mismatch and found it from evidence rather than by re-reading your own configuration, and then moved the whole trunk onto a bond without changing an IP address. That last part is what matters in production: a bond is supposed to be a transparent substitution for a cable, and this is where you find out whether you believe that.
Architecture
Two VyOS routers, four NICs, and one shared isolated segment. The shared segment is the important design choice: in production a bond terminates on a single switch, so both member links land in the same broadcast domain. One hypervisor bridge with all four member NICs on it models exactly that.
r1 r2
+---------------------+ +---------------------+
| eth0 mgmt | | eth0 mgmt |
| eth1 trunk / bond | | eth1 trunk / bond |
| eth2 bond member | | eth2 bond member |
+----+-----------+----+ +----+-----------+----+
| | | |
+-----------+------------+-----------+-----------+
|
isolated bridge "LAB-TRUNK"
stands in for a managed switch port-channel
VLAN 10 data 192.0.2.0/24 r1 .1 r2 .2
VLAN 20 voice 198.51.100.0/24 r1 .1 r2 .2
VLAN 30 mgmt 203.0.113.0/24 r1 .1 r2 .2
r2 is a VyOS box standing in for the switch on the far end of the trunk. Where its behaviour differs from a real managed switch — and it does, in two places that matter — the lab says so rather than pretending the substitution is free.
| Interface | Role | r1 | r2 |
|---|---|---|---|
| eth0 | management, untouched | your bridge | your bridge |
| eth1 | trunk (Tasks 1-5), bond member (Task 6 on) | LAB-TRUNK | LAB-TRUNK |
| eth2 | idle (Tasks 1-5), bond member (Task 6 on) | LAB-TRUNK | LAB-TRUNK |
Requirements
- A hypervisor with roughly 4 GiB of free RAM and 20 GiB of free disk. Each router needs 1 GiB of RAM and 8 GiB of disk as a floor.
- The VyOS 1.5 LTS ISO. Every command is written against the 1.5 tree.
- One isolated layer-2 bridge with no physical port and no host IP address,
carrying all four member NICs. On Proxmox that is a
vmbrwith nobridge-portsline, and the per-NIC Proxmox firewall must be off on every routed interface, pervyos-iv-02-vm-deployment. virtioNICs.e1000works, but the driver name appears in your evidence.- Console access to both routers, per the callout above.
Scenario
The site has one cable between the router and the access switch and three broadcast domains that have to share it: data, voice, and a management VLAN the network team insists on keeping separate. You are going to carry all three over one trunk, and then — because the site is getting a second cable next quarter — rebuild the trunk on a bond so that the migration is a change-window exercise rather than a redesign.
Tasks
Configuration blocks are written for the [edit] prompt. Where one opens with
configure and you are already at [edit], skip that line; blocks opening
with set or delete continue the session you have open. bash blocks are
operational mode or the hypervisor host, as their comments say.
Task 1 — Baseline, and an untagged frame to compare against
Capture the starting state on both routers before you type anything. Cleanup compares against these files.
JOURNAL="$HOME/lab05"
mkdir -p "$JOURNAL"
show configuration commands > "$JOURNAL/pre-lab-config.txt"
ip -brief link show > "$JOURNAL/pre-lab-links.txt"
ip -brief address show > "$JOURNAL/pre-lab-addresses.txt"
cat "$JOURNAL/pre-lab-links.txt"
Now build a plain untagged link so you have something to compare the tagged case against. On r1:
configure
set system host-name r1
set interfaces ethernet eth1 address 192.0.2.1/24
set interfaces ethernet eth1 description 'LAB-TRUNK untagged baseline'
commit
On r2:
configure
set system host-name r2
set interfaces ethernet eth1 address 192.0.2.2/24
set interfaces ethernet eth1 description 'LAB-TRUNK untagged baseline'
commit
Start a capture on r2 and ping from r1:
### On r2
tcpdump -ni eth1 -e -c 4 icmp
### On r1
ping -c 4 192.0.2.2
Look at the capture line. It has an Ethernet header — source MAC, destination
MAC, ethertype IPv4 — and no VLAN field at all. Save it. That absence is the
control for everything in Task 3.
Then remove the untagged address on both routers, because the same subnet is about to belong to VLAN 10 and having it in two places is confusing rather than instructive:
delete interfaces ethernet eth1 address 192.0.2.1/24
commit
Task 2 — Turn the link into a trunk
vif N is the whole mechanism. There is no mode trunk directive in VyOS and
no top-level vlan tree; creating sub-interfaces is how a port becomes a
trunk.
On r1:
configure
set interfaces ethernet eth1 description 'LAB-TRUNK to r2, VLANs 10,20,30'
set interfaces ethernet eth1 vif 10 address 192.0.2.1/24
set interfaces ethernet eth1 vif 10 description 'DATA'
set interfaces ethernet eth1 vif 20 address 198.51.100.1/24
set interfaces ethernet eth1 vif 20 description 'VOICE'
set interfaces ethernet eth1 vif 30 address 203.0.113.1/24
set interfaces ethernet eth1 vif 30 description 'MGMT'
commit
save
On r2, the same three VLANs with .2 addresses:
configure
set interfaces ethernet eth1 description 'LAB-TRUNK to r1, VLANs 10,20,30'
set interfaces ethernet eth1 vif 10 address 192.0.2.2/24
set interfaces ethernet eth1 vif 10 description 'DATA'
set interfaces ethernet eth1 vif 20 address 198.51.100.2/24
set interfaces ethernet eth1 vif 20 description 'VOICE'
set interfaces ethernet eth1 vif 30 address 203.0.113.2/24
set interfaces ethernet eth1 vif 30 description 'MGMT'
commit
save
Read the kernel mapping. VyOS shows you eth1 vif 10; the kernel shows you
something with a different name, and every diagnostic tool below the VyOS CLI
uses the kernel’s name:
$ ip link show3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 52:54:00:aa:00:01 brd ff:ff:ff:ff:ff:ff
5: eth1.10@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 52:54:00:aa:00:01 brd ff:ff:ff:ff:ff:ff
6: eth1.20@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 52:54:00:aa:00:01 brd ff:ff:ff:ff:ff:ff
7: eth1.30@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 52:54:00:aa:00:01 brd ff:ff:ff:ff:ff:ffIllustrative output
Note the MAC address. All four devices share the parent’s MAC — the sub-interfaces are not separate stations on the wire, they are the same station speaking three languages. That single fact explains a large fraction of VLAN confusion: nothing about a sub-interface is separate except the tag and the routing table entry.
Prove each VLAN carries traffic:
ping -c 2 192.0.2.2
ping -c 2 198.51.100.2
ping -c 2 203.0.113.2
show interfaces ethernet eth1 vif 10
All three should answer. If one does not, stop and use the flow in Task 4 before configuring anything else — a VLAN that fails now will fail more confusingly once the bond is underneath it.
Task 3 — Prove the tag on the wire
The -e flag prints the link-layer header, which is where the 802.1Q tag
lives. Without it, tcpdump shows you an ordinary IP packet and you have no way
to tell which VLAN carried it.
Run a capture on the parent interface on r2 — not on the sub-interface, because the sub-interface sees frames after the tag has been stripped:
### On r2
tcpdump -ni eth1 -e -c 4 vlan 20
### On r1
ping -c 4 198.51.100.2
$ tcpdump -ni eth1 -e -c 1 vlan 2052:54:00:aa:00:01 > 52:54:00:bb:00:01, ethertype 802.1Q (0x8100), length 102: vlan 20, p 0, ethertype IPv4 (0x0800), 198.51.100.1 > 198.51.100.2: ICMP echo request, id 1234, seq 1, length 64Illustrative output
Put that line next to the Task 1 capture. Same MACs, same wire, one extra field, and that field is the entire mechanism.
Now prove the separation, which is the claim people make loosest. With the same ping running, capture on the wrong sub-interface:
### On r2
timeout 10 tcpdump -ni eth1.10 icmp
Nothing arrives. VLAN 20 frames are delivered to eth1.20 and nowhere else,
because the kernel demultiplexes on the tag before the packet reaches any
sub-interface. That is stronger than “they are on different subnets”: they are
on different interfaces, and a firewall rule bound to eth1.10 will never see
a VLAN 20 packet.
Task 4 — Break it, then find it from evidence
Change one digit on r2 so its VLAN 20 sub-interface becomes VLAN 21:
On r2:
configure
delete interfaces ethernet eth1 vif 20
set interfaces ethernet eth1 vif 21 address 198.51.100.2/24
set interfaces ethernet eth1 vif 21 description 'VOICE'
commit
On r1, ping -c 3 198.51.100.2 now fails. Resist opening r2’s configuration.
Work the flow from vyos-viii-05-vlan-troubleshoot in order, and write down
what each step rules out.
### Step 1 - is the link up at all?
show interfaces ethernet eth1
### Step 2 - are tagged frames for this VLAN arriving?
tcpdump -ni eth1 -e -c 4 vlan 20
### Step 3 - is the local sub-interface up and addressed?
ip link show eth1.20
ip -brief address show eth1.20
### Step 4 - is there a route?
show ip route 198.51.100.0/24
Run the same Step 2 capture on r2 while r1 pings. Here is the finding that
names the fault: on r1 the capture shows echo requests leaving with
vlan 20; on r2 the capture on the parent also shows them arriving with
vlan 20. The frames cross the wire correctly. They are simply delivered to
nothing, because r2 has no eth1.20 for the kernel to demultiplex them into.
That is the signature to memorise. Tagged frames present on the parent and absent on every sub-interface means the VLAN ID exists on the wire and not in the configuration — a trunk mismatch, not a cabling or routing problem. On a real switch the same signature means the port’s allowed-VLAN list does not include the tag.
Confirm and fix:
### On r2
ip link show | grep 'eth1\.'
On r2:
delete interfaces ethernet eth1 vif 21
set interfaces ethernet eth1 vif 20 address 198.51.100.2/24
set interfaces ethernet eth1 vif 20 description 'VOICE'
commit
save
Re-run the ping from r1. It answers.
Task 5 — Measure what the tag costs, do not assume it
The 802.1Q header is four bytes. Whether they come out of your usable payload depends on where you measure and on what the intervening equipment counts as its MTU, and that differs between platforms often enough that the only trustworthy method is measurement on the path you actually have.
From r1, across VLAN 10:
PEER=192.0.2.2
for SIZE in 1464 1468 1472 1473 1476; do
printf '%s bytes: ' "$SIZE"
if ping -M do -s "$SIZE" -c 1 -W 2 "$PEER" > /dev/null 2>&1; then
echo pass
else
echo fail
fi
done
-M do sets the Don’t Fragment bit, so nothing on the path is allowed to
split the packet; -s is the ICMP payload size, to which the kernel adds 8
bytes of ICMP header and 20 of IPv4 header. A payload of 1472 therefore makes
a 1500-byte IP packet.
Record the largest size that passes. Then run the first failing size again without redirecting the output, and read the message, because there are two completely different failures here and they look identical in a pass/fail loop:
PEER=192.0.2.2
ping -M do -s 1473 -c 1 -W 2 "$PEER"
A local error naming the message size means you exceeded the MTU of your own sub-interface and the packet never left the box. Loss with no error means the packet left and something on the path dropped it — the far end, the bridge, or a device between them. The first is a local configuration fact; the second is a path fact, and confusing them is how MTU tickets end up assigned to the wrong team.
To push further, raise both the parent and the hypervisor bridge above 1500 and repeat. Raising only the router’s MTU measures the bridge, not the router, and produces a confidently wrong answer.
Task 6 — Move the trunk onto a bond
Nothing about the VLAN configuration changes here except the device it hangs from. That is the promise of a bond, and this task is where you check it.
A bond member cannot carry an address or a sub-interface, so the three vif
nodes come off eth1 first. (eth1 has held no address of its own since Task
1; if yours still does, delete it in the same commit — VyOS rejects a delete
of a node that is not there, so only delete what show configuration commands | match 'eth1' says exists.) Do the whole thing in one commit on each router:
the intermediate state, with no trunk anywhere, is not one to leave a box in.
On r1:
configure
delete interfaces ethernet eth1 vif 10
delete interfaces ethernet eth1 vif 20
delete interfaces ethernet eth1 vif 30
set interfaces bonding bond0 mode 'active-backup'
set interfaces bonding bond0 description 'LAB-TRUNK to r2'
set interfaces bonding bond0 member interface eth1
set interfaces bonding bond0 member interface eth2
set interfaces bonding bond0 vif 10 address 192.0.2.1/24
set interfaces bonding bond0 vif 10 description 'DATA'
set interfaces bonding bond0 vif 20 address 198.51.100.1/24
set interfaces bonding bond0 vif 20 description 'VOICE'
set interfaces bonding bond0 vif 30 address 203.0.113.1/24
set interfaces bonding bond0 vif 30 description 'MGMT'
commit
save
On r2, the same with .2 addresses:
configure
delete interfaces ethernet eth1 vif 10
delete interfaces ethernet eth1 vif 20
delete interfaces ethernet eth1 vif 30
set interfaces bonding bond0 mode 'active-backup'
set interfaces bonding bond0 description 'LAB-TRUNK to r1'
set interfaces bonding bond0 member interface eth1
set interfaces bonding bond0 member interface eth2
set interfaces bonding bond0 vif 10 address 192.0.2.2/24
set interfaces bonding bond0 vif 10 description 'DATA'
set interfaces bonding bond0 vif 20 address 198.51.100.2/24
set interfaces bonding bond0 vif 20 description 'VOICE'
set interfaces bonding bond0 vif 30 address 203.0.113.2/24
set interfaces bonding bond0 vif 30 description 'MGMT'
commit
save
The kernel now shows bond0 with bond0.10, bond0.20, bond0.30 hanging
from it, and eth1 and eth2 carrying the SLAVE flag:
ip link show bond0
ip link show | grep -E 'bond0|eth1|eth2'
show interfaces bonding
Every ping from Task 2 must still work, unchanged, with the same addresses:
ping -c 2 192.0.2.2
ping -c 2 198.51.100.2
ping -c 2 203.0.113.2
Task 7 — Prove failover, with traffic running
Start a continuous ping on r1 in one session and watch the bond in another, then take the active member down.
### Session 1 on r1 - leave this running
ping -i 0.2 192.0.2.2
### Session 2 on r1 - which member is active right now?
sudo cat /proc/net/bonding/bond0
The file names the mode, the currently active slave, and each member’s MII Status and Link Failure Count. Copy it into the journal before you break
anything; a failover with no “before” to compare against is an anecdote.
Now down the active member. If the file reports eth1 as the active slave,
disable eth1:
On r1:
configure
set interfaces ethernet eth1 disable
commit
$ sudo cat /proc/net/bonding/bond0Ethernet Channel Bonding Driver: v6.6
Bonding Mode: fault-tolerance (active-backup)
Currently Active Slave: eth2
MII Status: up
Slave Interface: eth1
MII Status: down
Link Failure Count: 1
Slave Interface: eth2
MII Status: up
Link Failure Count: 0Illustrative output
Go back to the ping and count the lost sequence numbers. That gap is your failover time; record the number, not an adjective.
Restore, and confirm the count does not go back to zero:
On r1:
delete interfaces ethernet eth1 disable
commit
save
sudo cat /proc/net/bonding/bond0
Link Failure Count: 1 persists on eth1. Outside an incident that is the
most useful field in the file: a member with a non-zero and growing count is
a flapping link, invisible to every reachability test because the bond is
doing its job of hiding it. Sample this file from monitoring, not only from an
SSH session during an outage.
Task 8 — Try 802.3ad and read the diagnosis
This task is here for what it teaches when it fails. On r1 only, switch the mode:
configure
set interfaces bonding bond0 mode '802.3ad'
set interfaces bonding bond0 lacp-rate 'fast'
commit
sudo cat /proc/net/bonding/bond0
Read the LACP section rather than the reachability. What matters is whether
each member reports a non-zero Aggregator ID, and what the partner state
says — an aggregation that never forms shows a partner that is not answering,
not a local syntax problem. On a lab built over a software bridge with no LAG
configured, expect it not to form; on a real switch with a matching
port-channel, expect it to.
Record what you saw. Then decide, and write the decision down: keep 802.3ad
and fix the far end, or revert to active-backup because the far end will
never be a LAG. Both are legitimate; shipping an 802.3ad bond into a peer that
cannot aggregate is not.
set interfaces bonding bond0 mode 'active-backup'
delete interfaces bonding bond0 lacp-rate
commit
save
Validation
ip link showon both routers listsbond0plusbond0.10,bond0.20andbond0.30, and showseth1andeth2with theSLAVEflag.pingsucceeds across all three VLAN subnets from r1 to r2, using the same addresses that worked before the bond existed.- Your journal holds two tcpdump lines from the same wire: one with no VLAN
field, one with
ethertype 802.1Qandvlan 20. - A capture on
eth1.10while VLAN 20 traffic is flowing shows nothing. - The MTU loop has a recorded largest-passing size, and you can state whether the first failing size failed locally or on the path, quoting the message.
/proc/net/bonding/bond0on r1 showsLink Failure Count: 1on the member you downed, andCurrently Active Slavenaming the other member at the moment of the failure.- You can state, in one sentence each, why
active-backupneeded nothing from r2 and why802.3addid.
Expected Outcome
Both routers end with the trunk on a bond and the VLAN addressing untouched since Task 2:
interfaces {
bonding bond0 {
description "LAB-TRUNK to r2"
member {
interface eth1
interface eth2
}
mode active-backup
vif 10 {
address 192.0.2.1/24
description "DATA"
}
vif 20 {
address 198.51.100.1/24
description "VOICE"
}
vif 30 {
address 203.0.113.1/24
description "MGMT"
}
}
ethernet eth1 {
description "LAB-TRUNK to r2, VLANs 10,20,30"
}
ethernet eth2 {
}
}
The three VLANs reach across the bond, one member can be removed without dropping more than a fraction of a second of traffic, and you have evidence for every claim in that sentence.
Troubleshooting
A vif commit is rejected because the interface has an address. A bond
member cannot hold an address or a sub-interface. Delete both in the commit
that creates the bond, as Task 6 does.
One VLAN fails and the others are fine. A per-VLAN fact is a VLAN-ID or sub-interface problem, not a link problem. Run Task 4’s comparison: capture on the parent at both ends and see whether the tag crosses the wire.
Everything fails and ip link show says the sub-interfaces are up. Up
means configured, not connected. Check the parent’s LOWER_UP flag, then the
bridge on the hypervisor.
/proc/net/bonding/bond0 does not exist. The bond was never created. A
bonding node with no member interface commits and produces a device with
nothing in it — check show interfaces bonding for a member list first.
Failover works but the ping never recovers. The peer’s forwarding entry for the bond MAC still points at the old port. On a real switch that resolves as soon as the new member transmits; if it does not, the two members are not in the same broadcast domain — on this lab, different bridges.
Cleanup
Cleanup restores both routers to the Task 1 capture, not merely to “no bond”.
Step 1. Find out what is actually there first — VyOS rejects a delete of a
node that does not exist, and a cleanup that aborts halfway is worse than
none:
show configuration commands | match 'interfaces bonding'
show configuration commands | match 'interfaces ethernet eth1'
show configuration commands | match 'interfaces ethernet eth2'
Then delete exactly what that listed. A reader who finished the lab has the
bond and a leftover description on eth1:
configure
delete interfaces bonding bond0
delete interfaces ethernet eth1 description
commit
save
A reader who stopped after Task 5 has the three vif nodes on eth1 instead
and no bond, so the deletes are these:
configure
delete interfaces ethernet eth1 vif 10
delete interfaces ethernet eth1 vif 20
delete interfaces ethernet eth1 vif 30
delete interfaces ethernet eth1 description
commit
save
Step 2. Prove it, on each router:
JOURNAL="$HOME/lab05"
show configuration commands > "$JOURNAL/post-lab-config.txt"
ip -brief link show > "$JOURNAL/post-lab-links.txt"
diff -u "$JOURNAL/pre-lab-config.txt" "$JOURNAL/post-lab-config.txt" \
&& echo 'CONFIG RESTORED'
diff -u "$JOURNAL/pre-lab-links.txt" "$JOURNAL/post-lab-links.txt" \
&& echo 'LINKS RESTORED'
A non-empty link diff usually means a bond0 device is still present. Check
ip link show bond0 before deleting anything by hand: a device the
configuration no longer describes is a commit that did not complete, and is
worth reading show log over rather than papering over.
Step 3. If the routers exist only for this lab, destroy them.
### On the Proxmox host, once per router
VMID=201
qm stop "$VMID"
qm destroy "$VMID"
Production notes
Adding a VLAN to an existing trunk is additive on the router and not on the switch, whose allowed-VLAN list is usually owned by another team. Sequence the switch side first: a tag arriving at a router with no matching sub-interface is discarded harmlessly, while a router sending a tag the switch does not allow looks, from the router, exactly like a broken cable.
Removing a VLAN is the dangerous direction — delete interfaces ethernet eth1 vif 20 takes down a subnet’s gateway in one commit with no warning. Use
commit-confirm (see vyos-lab-03-commit-rollback), and check
show interfaces ethernet eth1 vif 20 for counters that are still moving
before deciding the VLAN is unused.
Migrating a live trunk onto a bond — Task 6 — is a genuine outage on a
single-homed link, because there is no moment when the addresses exist on both
eth1 and bond0. The mitigation is to build the bond with only the second
cable as a member, move the addressing to it while the first cable still
carries the old trunk, then add the first cable. Two windows and one brief
overlap, which is why the second cable gets pulled before the change.
Holding applies here too. A bond running on one member because the other flaps is degraded, not down, and leaving it there with an owner and an end time on the ticket often beats chasing a mode change at 02:00.
What You Learned
vif Nis the whole trunk mechanism. There is no mode to set, and the kernel names the resulteth1.10.- Sub-interfaces share the parent’s MAC. One station speaking several languages, not several stations — the tag is all that separates them.
- Where you capture decides what you see. The tag is on the parent and already gone on the sub-interface.
- A VLAN-ID mismatch produces no error anywhere. You found it by comparing captures at two ends, which is the only method that works.
- The tag overhead is measured, not recited. This course states the rule
two incompatible ways;
ping -M dosettles it for your path in seconds. - A bond is transparent to the layer above and not to the peer below. The VLANs moved without touching an address, and the mode you may use is decided by whoever owns the far end.