Leaking and firewall — dispatch by interface, state, asymmetric paths

A route leak makes a prefix reachable. It says nothing about whether the packets that use it should be allowed. Those are two independent controls, and an estate that has only the first has removed an isolation boundary and replaced it with nothing.

This lesson is the firewall half. It is also, unavoidably, a correction: the binding model this material used to teach — set interfaces ethernet eth1 firewall in name ... — was removed in VyOS 1.4 and does not exist on 1.5. Everything below is the base-chain model that replaced it, plus the part of the story VyOS genuinely does not provide.

Why the firewall matters at a leak

The leak installs a route; the firewall decides whether the packet that uses it is permitted. A leak with a deny-all policy blocks everything; a leak with no policy permits everything. Production needs both, and needs them written together, because the two are revoked separately during an incident: pulling a firewall rule is a smaller, faster, more reversible action than pulling a route that other flows also depend on.

flowchart LR
  H["host 10.10.0.10<br/>vrf mgmt"] --> R1["table 1001<br/>leaked route to 198.51.100.53"]
  R1 --> F1["forward filter<br/>jump to LEAK-MGMT-TO-SHARED"]
  F1 -->|"accept udp/53"| S1["DNS 198.51.100.53"]
  S1 --> R2["table 254<br/>reverse leak to 10.10.0.0/24"]
  R2 --> F2["forward filter<br/>which ruleset matches now?"]
  F2 --> H

The question mark on the return path is the subject of half this lesson. The two directions of one flow are two different interface pairs, and the firewall dispatches on interfaces.

The binding model on VyOS 1.5

There are two things to get right, and the second one is where 1.3-era material misleads.

Named rulesets hold the policy. A firewall ipv4 name set is a list of rules plus a default action. It is inert until something sends packets to it.

Base chains dispatch to them. VyOS 1.5 has firewall ipv4 input filter for traffic to the router itself, firewall ipv4 output filter for traffic the router originates, and firewall ipv4 forward filter for transit traffic — which is what every leaked flow is. A rule in a base chain matches, takes action jump, and names a jump-target.

configure
# Which interfaces belong to which routing context, in one place
set firewall group interface-group MGMT-IF interface eth1
set firewall group interface-group TENANT-A-IF interface eth2
set firewall group interface-group SHARED-IF interface eth0

# What may cross from mgmt into the shared segment
set firewall ipv4 name LEAK-MGMT-TO-SHARED default-action drop
set firewall ipv4 name LEAK-MGMT-TO-SHARED default-log
set firewall ipv4 name LEAK-MGMT-TO-SHARED rule 10 action accept
set firewall ipv4 name LEAK-MGMT-TO-SHARED rule 10 description 'mgmt hosts to DNS'
set firewall ipv4 name LEAK-MGMT-TO-SHARED rule 10 source address 10.10.0.0/24
set firewall ipv4 name LEAK-MGMT-TO-SHARED rule 10 destination address 198.51.100.53
set firewall ipv4 name LEAK-MGMT-TO-SHARED rule 10 protocol udp
set firewall ipv4 name LEAK-MGMT-TO-SHARED rule 10 destination port 53
set firewall ipv4 name LEAK-MGMT-TO-SHARED rule 20 action accept
set firewall ipv4 name LEAK-MGMT-TO-SHARED rule 20 description 'mgmt hosts to NTP'
set firewall ipv4 name LEAK-MGMT-TO-SHARED rule 20 source address 10.10.0.0/24
set firewall ipv4 name LEAK-MGMT-TO-SHARED rule 20 destination address 198.51.100.123
set firewall ipv4 name LEAK-MGMT-TO-SHARED rule 20 protocol udp
set firewall ipv4 name LEAK-MGMT-TO-SHARED rule 20 destination port 123

# Send the leaked direction to it
set firewall ipv4 forward filter rule 100 action jump
set firewall ipv4 forward filter rule 100 jump-target LEAK-MGMT-TO-SHARED
set firewall ipv4 forward filter rule 100 inbound-interface group MGMT-IF
set firewall ipv4 forward filter rule 100 outbound-interface group SHARED-IF
commit
save

The dispatch rule is the load-bearing line. Without it, LEAK-MGMT-TO-SHARED exists, appears in show firewall, reviews correctly — and never sees a packet. That failure looks exactly like a permissive firewall, because a policy nothing jumps to is a policy that permits everything by not being consulted.

What VyOS does not give you: a VRF match

The rule an operator wants to write is “permit this traffic in VRF mgmt”. VyOS does not expose it. A firewall rule matches interfaces, interface groups, addresses, address groups, ports, protocols and connection state. VRF membership is a property of an interface, not something a rule can select on.

The supported expression of a per-VRF policy is therefore the list of that VRF’s interfaces, and the reason to keep it in a interface-group is that the list changes: a VLAN sub-interface added to mgmt next quarter must appear in one place, not in every rule that meant “mgmt”.

set firewall group interface-group MGMT-IF interface eth1
set firewall group interface-group MGMT-IF interface eth1.100

The return path, and the trap in it

Here is the flow, as the firewall sees it rather than as the diagram suggests.

The request arrives on eth1 and leaves on eth0. In the forward filter base chain it matches inbound-interface group MGMT-IF, outbound-interface group SHARED-IF, and jumps to LEAK-MGMT-TO-SHARED, which permits it.

The reply arrives on eth0 and leaves on eth1. It matches neither of those conditions. It matches whatever rule in the base chain covers inbound-interface group SHARED-IF — a different ruleset, written by a different change, for a different purpose — or, if no rule matches, the base chain’s own default action.

sequenceDiagram
  autonumber
  participant H as host 10.10.0.10
  participant FWD as forward filter (base chain)
  participant A as LEAK-MGMT-TO-SHARED
  participant B as ruleset for SHARED ingress
  participant S as DNS 198.51.100.53

  H->>FWD: in eth1, out eth0
  FWD->>A: rule 100 matched, jump
  A->>S: rule 10 accept, udp/53
  S->>FWD: reply, in eth0, out eth1
  Note over FWD: rule 100 does not match<br/>this is the other interface pair
  FWD->>B: a different rule, a different jump-target
  Note over B: does this ruleset permit<br/>established traffic?

The fix is not subtle, but it has to be a deliberate decision. Either the return direction gets its own dispatch rule and ruleset, or state handling is lifted out of the individual rulesets so it applies everywhere.

configure
# Option A: an explicit ruleset for the return direction
set firewall ipv4 name LEAK-SHARED-TO-MGMT default-action drop
set firewall ipv4 name LEAK-SHARED-TO-MGMT default-log
set firewall ipv4 name LEAK-SHARED-TO-MGMT rule 10 action accept
set firewall ipv4 name LEAK-SHARED-TO-MGMT rule 10 description 'replies to leaked flows'
set firewall ipv4 name LEAK-SHARED-TO-MGMT rule 10 state established
set firewall ipv4 name LEAK-SHARED-TO-MGMT rule 10 state related
set firewall ipv4 name LEAK-SHARED-TO-MGMT rule 20 action accept
set firewall ipv4 name LEAK-SHARED-TO-MGMT rule 20 description 'monitoring poller into mgmt'
set firewall ipv4 name LEAK-SHARED-TO-MGMT rule 20 source address 198.51.100.90
set firewall ipv4 name LEAK-SHARED-TO-MGMT rule 20 destination address 10.10.0.0/24

set firewall ipv4 forward filter rule 110 action jump
set firewall ipv4 forward filter rule 110 jump-target LEAK-SHARED-TO-MGMT
set firewall ipv4 forward filter rule 110 inbound-interface group SHARED-IF
set firewall ipv4 forward filter rule 110 outbound-interface group MGMT-IF
commit
save

configure
# Option B: one state policy for the whole box, so no ruleset has to
# remember to permit the reply of a flow it already permitted
set firewall global-options state-policy established action accept
set firewall global-options state-policy related action accept
set firewall global-options state-policy invalid action drop
commit
save

Option B is the one most estates end up with, and it is worth understanding what it costs. A global state policy accepts established traffic before any named ruleset is consulted, which means a ruleset can no longer deny the return direction of a flow it permitted outbound. That is usually the intent. When it is not — a leak where the request should be permitted and the reply should be rate-limited or logged — the policy has to stay in the rulesets, and both directions have to be written.

Rule 20 in option A is the case that shows why the return direction is a policy question and not just plumbing: the monitoring poller initiates into mgmt. That is a new connection in the reverse direction, it is not covered by any state rule, and it exists only because someone decided it should.

Connection tracking across a leak

The commands that separate the three cases:

show ip route vrf mgmt
show conntrack table ipv4
show firewall
show firewall ipv4 name LEAK-SHARED-TO-MGMT

and, when the answer has to come from the kernel rather than from VyOS’s rendering of it:

ip route show table 254
conntrack -L
nft list ruleset

Read show conntrack table ipv4 for whether the flow is there at all and what state it is in. An entry in ESTABLISHED with a reply direction recorded means the packets are crossing and something is dropping them after tracking. No entry at all means the request never reached the forwarding path, which points back at routing.

Making leaked traffic identifiable

Leaked flows are the ones you will most want to find in a log, because they are the ones that cross a boundary somebody drew on purpose. Three things make that easy, and one thing that older material recommends is not available.

Name the ruleset after the direction of the leak. LEAK-MGMT-TO-SHARED and LEAK-SHARED-TO-MGMT are not decoration: the ruleset name is carried into the generated nftables chain, so it is what you read in nft list ruleset, and the ruleset is the unit you stop dispatching to during an incident.

Describe every rule. description costs one line and is the difference between a rule an incoming engineer can evaluate and one they have to reverse-engineer from the addresses.

Log deliberately, at both ends of the ruleset. default-log on the named ruleset records what the policy rejected; log on an accept rule records what it permitted, which is the one you want when the question is “is this leak being used at all”.

set firewall ipv4 name LEAK-MGMT-TO-SHARED rule 10 log
set firewall ipv4 name LEAK-MGMT-TO-SHARED rule 10 log-options level info

VyOS prefixes each logged packet with the ruleset and rule that matched. Confirm the exact prefix format on your own box before you write a log parser or an alert against it — it is a rendering detail, not a documented interface.

Stateful or stateless at a leak boundary

Stateful filtering tracks connections; stateless filtering evaluates each packet on its own. At a leak boundary stateful is almost always right:

• Nearly every shared service is request/reply. Tracking the request and matching the reply as established is one rule instead of two rulesets that have to agree.
• Consumer hosts generate many short-lived flows — DNS queries, NTP polls, RADIUS exchanges — and the tracking table is built for exactly that churn.
• The alternative is writing the reply’s 5-tuple by hand, which means permitting source port 53 inbound from the whole shared segment. That is a wider hole than the one you were trying to control.

Stateless is the right choice in narrow cases, and they are worth naming so the choice stays deliberate:

• One-way telemetry. A collector the consumer pushes to and never hears back from. No state is needed and none should be created.
• Very high packet rates where tracking is the bottleneck. Tracking every packet of a flow that will never need a stateful decision costs table entries. This is a measured decision, made after show conntrack table ipv4 shows the table under pressure.

Conntrack exhaustion is not a third case. A full table is a fault to be diagnosed — usually one host opening flows faster than they expire — not a reason to redesign the policy around it.

How it fails

• The named ruleset has no dispatch rule. It exists, it reads correctly, and no base chain jumps to it. Everything is permitted and the configuration looks strict. Check show firewall for the base chain, not just for the named set.
• Only the outbound direction was dispatched. The request matches, the reply falls through to a different ruleset or to the base chain’s default action. This is the leak-specific failure and it looks like an intermittent service problem, because whichever flows a global state policy happens to cover keep working.
• 1.3 binding syntax. set interfaces ethernet eth1 firewall in name ... is rejected on 1.5. Translating it to a base-chain rule and forgetting that in and out were two separate bindings is the follow-on error.
• A rule matching the wrong interface. The dispatch names eth0 where eth1 was meant. The leak works and the policy never sees a packet, so the symptom is silence rather than a drop.
• An interface group that drifted. A VLAN sub-interface joined the VRF and nobody added it to MGMT-IF. Part of the VRF is policed and part is not, which is the hardest version of this to spot because most traffic behaves.
• INVALID traffic dropped by the global state policy after a path change. A flow whose request went through another router arrives here as a reply with no matching entry. Correct behaviour, confusing symptom.
• Overlapping tenant address space with no NAT. Two VRFs both using 10.0.0.0/24 share one conntrack namespace. Diagnose this before it ships, because retrofitting NAT at a leak boundary changes every rule that referenced an address.

Rollback

configure
# Stop dispatching to a ruleset without deleting the policy itself
delete firewall ipv4 forward filter rule 100

# Remove a single rule from a ruleset
delete firewall ipv4 name LEAK-MGMT-TO-SHARED rule 20

# Remove a ruleset entirely - delete the dispatch rules first, or the
# commit is rejected for referencing a jump-target that no longer exists
delete firewall ipv4 forward filter rule 110
delete firewall ipv4 name LEAK-SHARED-TO-MGMT

commit
save

• Deleting the dispatch rule is the smaller, more reversible action, and it is what you want during an incident. It leaves the policy in the configuration to be re-enabled with one set.
• Use commit-confirm for any firewall change on a router you are reaching over the network. A base-chain edit can remove your own session.
• After the change, re-test the reply direction of a real flow, not ping. ICMP echo is usually permitted by a rule nobody thinks about, which is why it succeeds on a leak whose actual service traffic is broken.
• rollback N inside configure for anything larger. Confirm what the revision contains before committing it; a firewall rollback that also reverts a leak leaves reachability and policy out of step.

Production discipline

Cross-course references

vyos-xvi-01-leaking-concept and vyos-xvi-02-leaking-config cover the routing half — no firewall change fixes a leak that installed nothing. vyos-xv-05-vrf-troubleshoot covers the diagnostic order across the whole VRF subsystem. The firewall parts XXXVII-VyOS-Firewall cover base chains, rule ordering and state handling in their own right, and XXXVIII covers NAT, which is what an overlapping-address-space design needs at the leak boundary. The Linux course’s XXII-Linux-NetTroubleshoot covers conntrack at the host level.

Quiz