VLAN routing — moving packets between VLANs on the same router

A VyOS box with multiple VLAN sub-interfaces on a single trunk port can route traffic between those VLANs without an external router. This is the router-on-a-stick pattern: a single physical link carrying multiple VLANs, with the router providing the L3 forwarding between them.

The pattern is easy to configure and easy to get subtly wrong, because the thing that makes it work — one Linux netdev per VLAN — is also the thing that decides which firewall rules can see the traffic.

Router-on-a-stick

flowchart LR
  V10[VLAN 10<br/>192.0.2.0/24] --> S[Switch]
  V20[VLAN 20<br/>203.0.113.0/24] --> S
  V30[VLAN 30<br/>198.51.100.0/24] --> S
  S -- trunk --> E[eth0<br/>VyOS]
  E --> VIF10[eth0.10<br/>192.0.2.1]
  E --> VIF20[eth0.20<br/>203.0.113.1]
  E --> VIF30[eth0.30<br/>198.51.100.1]

Each VLAN has its own subnet. The VyOS box has a sub-interface on each subnet and routes between them.

Configuration

configure
set interfaces ethernet eth0 vif 10 address '192.0.2.1/24'
set interfaces ethernet eth0 vif 20 address '203.0.113.1/24'
set interfaces ethernet eth0 vif 30 address '198.51.100.1/24'
commit
save

The vif tree is unchanged between 1.3 and 1.5 — this is one of the few interface families 1.4 did not restructure. Each vif creates a real kernel netdev named eth0.10, eth0.20, eth0.30, and each address produces a connected route.

Inter-VLAN routing

$ show ip route

S>* 0.0.0.0/0 [1/0] via 192.0.2.254, eth0.10, weight 1, 00:04:12
C>* 192.0.2.0/24 is directly connected, eth0.10, 00:04:12
L>* 192.0.2.1/32 is directly connected, eth0.10, 00:04:12
C>* 198.51.100.0/24 is directly connected, eth0.30, 00:04:12
C>* 203.0.113.0/24 is directly connected, eth0.20, 00:04:12

The markers matter more than the numbers. > means the route is installed in the FIB and * means FRR selected it; a connected route that appears without them is a sub-interface that exists in the configuration but is not carrying a link. Confirm the netdev side with show interfaces ethernet eth0 — a vif on a down parent shows the sub-interface down too.

A packet from 192.0.2.10 to 203.0.113.20:

• Arrives on eth0.10 (VLAN 10)
• Routing table lookup: 203.0.113.20 is in 203.0.113.0/24, connected on eth0.20
• Forwarded out eth0.20 with VLAN 20 tag

Where the firewall actually attaches

The per-interface binding older material uses — set interfaces ethernet eth0 vif 10 firewall in name 'VLAN10-IN' — was removed in VyOS 1.4 and does not exist in 1.5. A 1.5 router rejects it at commit. Rule sets are still named the same way, but they are now reached by a jump from one of the base chains: forward filter for traffic passing through the router (which is all inter-VLAN traffic), input filter for traffic addressed to the router itself.

The per-VLAN pattern becomes two halves — the policy, and the dispatch that selects it:

configure
# Half one: the policy for each VLAN
set firewall ipv4 name VLAN10-IN default-action drop
set firewall ipv4 name VLAN10-IN rule 10 action accept
set firewall ipv4 name VLAN10-IN rule 10 state established
set firewall ipv4 name VLAN10-IN rule 20 action accept
set firewall ipv4 name VLAN10-IN rule 20 state related
set firewall ipv4 name VLAN10-IN rule 30 action accept
set firewall ipv4 name VLAN10-IN rule 30 destination address '203.0.113.0/24'

set firewall ipv4 name VLAN20-IN default-action drop
set firewall ipv4 name VLAN20-IN rule 10 action accept
set firewall ipv4 name VLAN20-IN rule 10 state established
set firewall ipv4 name VLAN20-IN rule 20 action accept
set firewall ipv4 name VLAN20-IN rule 20 state related

# Half two: the dispatch, keyed on the sub-interface the frame landed on
set firewall ipv4 forward filter default-action drop
set firewall ipv4 forward filter rule 10 inbound-interface name eth0.10
set firewall ipv4 forward filter rule 10 action jump
set firewall ipv4 forward filter rule 10 jump-target VLAN10-IN
set firewall ipv4 forward filter rule 20 inbound-interface name eth0.20
set firewall ipv4 forward filter rule 20 action jump
set firewall ipv4 forward filter rule 20 jump-target VLAN20-IN
commit
save

Two things about this shape are worth holding on to.

The dispatch key is the sub-interface, not “the VLAN”. VyOS has no firewall match for a VLAN ID. It matches interfaces, interface groups, addresses, ports and state. In a router-on-a-stick that distinction costs you nothing, because there is exactly one sub-interface per VLAN — but it means the rule set is only as accurate as the interface list it names. When a VLAN gains a second sub-interface, keep the list in one place:

set firewall group interface-group VLAN10-IF interface eth0.10
set firewall group interface-group VLAN10-IF interface eth1.10
set firewall ipv4 forward filter rule 10 inbound-interface group VLAN10-IF

The established/related rules are not boilerplate. A per-VLAN policy that permits the request and forgets the reply produces a router that forwards perfectly in one direction and looks, to everyone below you, exactly like a routing fault. The alternative to repeating them in every rule set is to lift state handling out once:

set firewall global-options state-policy established action accept
set firewall global-options state-policy related action accept
set firewall global-options state-policy invalid action drop

Pick one mechanism and use it consistently. A configuration that has both is not wrong, but the next operator has to read two places to work out why a packet was accepted.

Asymmetric routing

flowchart LR
  A[VLAN 10 host] --> R[VyOS]
  R --> B[VLAN 20 host]
  B --> R
  R --> A

In a simple router-on-a-stick, the routing is symmetric: traffic from VLAN 10 to VLAN 20 goes via the VyOS, and traffic from VLAN 20 to VLAN 10 returns via the same VyOS. Conntrack sees both directions of every flow, and the stateful rules work.

If the network has multiple routers or multiple paths between VLANs, the routing can be asymmetric: the request goes through this router and the reply goes through another one. Conntrack on this box then sees a reply for a flow it has no entry for, classifies it as invalid, and the state rules never accept it. The remedies, in the order you should prefer them:

• Make the routing symmetric. Prefer one path with routing policy, or stop the second router advertising the same prefixes. This is the only fix that leaves you with a working stateful firewall.
• Write the affected rules statelessly — match on addresses and ports in both directions and accept without consulting state. You lose the protection state was giving you, on those flows.
• Put both directions on the same box. If the asymmetry is between two VyOS routers you control, conntrack synchronisation (set service conntrack-sync) shares the table between them so either box recognises the flow.

There is no knob that makes a stateful rule tolerate a half-seen flow. state invalid can be accepted rather than dropped, but that is a decision to stop filtering those packets, not a repair.

How the result is validated

show ip route
show interfaces ethernet eth0
show firewall
ping 203.0.113.20 interface eth0.10
traceroute 203.0.113.20

The first shows the connected routes and the FIB markers; the second shows the parent and its sub-interfaces; the third shows which rule sets exist and their counters; the fourth sources a probe from the VLAN 10 address so the reply has to come back through the VLAN 10 path; the fifth shows the path.

The counters in show firewall are the part operators skip and then need. A dispatch rule with a zero packet count is a rule nothing is reaching — usually because an earlier rule in the base chain already matched, or because the interface name is wrong.

How it fails

The production failure modes the engineer must recognise:

• Missing sub-interface on a VLAN. A packet from VLAN 10 to VLAN 30 has no route because eth0.30 is not configured. The switch delivers the tagged frame, the parent netdev has nothing to hand it to, and the kernel drops it silently.
• Rule set attached to nothing. set firewall ipv4 name VLAN20-IN commits cleanly with no dispatch rule pointing at it. The policy exists, is visible in show firewall, and filters nothing. This is the commonest 1.3-to-1.5 migration failure, because the removed per-interface binding was the only thing that used to attach it.
• Dispatch rule naming the parent. inbound-interface name eth0 where eth0.10 was meant: the rule never matches, the traffic falls to the base chain default.
• Asymmetric routing. Two routers in the path; traffic goes via router A but returns via router B. Conntrack marks the reply invalid and the state rules drop it.
• Routing loop. Two routers both have the same routes; the packet bounces between them until TTL expires.
• MTU on the sub-interface too small. Inter-VLAN packets larger than the MTU are fragmented or dropped, and a sub-interface cannot exceed its parent’s MTU however you configure it.

Rollback

The recovery from inter-VLAN routing failure:

• Missing sub-interface: set interfaces ethernet eth0 vif 30 address '198.51.100.1/24', then commit and save.
• Unattached rule set: add the forward filter rule with action jump and jump-target, and confirm the counter moves.
• Wrong interface in a dispatch rule: correct the inbound-interface name value; the rule set itself needs no change.
• Asymmetric routing: force one path, or make the affected rules stateless while you fix the topology.

Any of these can be undone with rollback 1 if the change is a recent commit — but a base chain whose default-action is drop will have cut your own session if you are managing the box in band, so use commit-confirm for firewall changes on a remote router.

Production discipline

Cross-course references

The OPNsense course’s XIV-OPNsense-VLAN covers the equivalent configuration on the firewall side. The Linux course’s XXII-Linux-NetTroubleshoot covers the underlying routing troubleshooting. The Observability course’s LX-Observability-NetworkObs covers how to alert on routing changes.

Quiz