Stateful vs stateless filtering — conntrack, NEW/ESTABLISHED/RELATED, the performance argument

The stateful firewall is the single most important security control on a production router. It is what keeps an Internet attacker out of the inside, and it is what lets the reply to a legitimate outbound request come back without a second rule written to permit it.

On VyOS 1.5 LTS that control is two Linux subsystems wearing a CLI: nf_conntrack, which tracks flows, and nftables, which classifies packets. Every rule written under set firewall ipv4 ... is rendered into an nftables rule, and the ones that mention state consult the conntrack table.

This lesson covers the mechanism: where conntrack sits in the packet path, what the states mean, what stateful filtering actually costs, and the failure that comes from writing a rule without a state clause. The lessons that follow deal with zones, rule ordering, conntrack-aware matches, default-deny posture and diagnostics.

What conntrack is

The connection tracker records every TCP, UDP, ICMP and SCTP flow it sees into an in-memory hash table. Each entry holds:

• The tuple in both directions: source and destination address, source and destination port.
• The protocol.
• The state the flow is in, and the protocol-specific sub-state for TCP.
• A timeout, refreshed as packets arrive.
• The connection mark, used by policy routing and by connmark rules — covered in vyos-xxxvii-04-state-tracking.

Conntrack runs early, in the prerouting and output paths, before the filter rules are evaluated. By the time a packet reaches a rule that says state established, conntrack has already looked the packet up and labelled it, so the rule is reading a verdict rather than computing one.

flowchart LR
  P["Packet arrives on eth0"]
  CT["conntrack lookup<br/>(prerouting)<br/>state assigned"]
  R["Routing decision"]
  I["input filter<br/>(destined for the router)"]
  F["forward filter<br/>(destined elsewhere)"]
  PO["postrouting<br/>NAT, conntrack confirm"]
  E["Packet leaves on eth1"]

  P --> CT
  CT --> R
  R -->|"local"| I
  R -->|"forwarded"| F
  I --> PO
  F --> PO
  PO --> E

The conntrack table is the state. Without it every packet is judged alone, which is stateless filtering. With it every packet is judged against what the router already knows about that conversation.

Conntrack is also not free-standing on VyOS: it becomes active once a stateful firewall or NAT is configured. A router with neither is not tracking anything, and show conntrack table ipv4 on such a box is empty rather than broken.

The states

• NEW — the first packet of a flow that conntrack has no entry for.
• ESTABLISHED — a packet belonging to a flow conntrack has already seen traffic in both directions for.
• RELATED — a packet that is not part of an existing flow but is connected to one: an ICMP error referring to a tracked flow, or a data connection a protocol helper has been told to expect.
• INVALID — a packet conntrack cannot associate with any flow and cannot make sense of on its own.

stateDiagram-v2
  [*] --> NEW: first packet, no entry
  NEW --> ESTABLISHED: reply observed
  NEW --> INVALID: unusable, no reply, protocol nonsense
  ESTABLISHED --> ESTABLISHED: further data
  ESTABLISHED --> RELATED: ICMP error, or a helper-expected flow
  RELATED --> ESTABLISHED: the related flow gets its own reply
  ESTABLISHED --> [*]: teardown, then timeout
  INVALID --> [*]: dropped or timed out

The state machine is per protocol. TCP is the richest, tracking the handshake, the data phase and the teardown separately. UDP has no handshake to observe, so a flow becomes ESTABLISHED once a reply is seen and stays there until its timeout. ICMP is the simplest: a request, then its matching reply.

The 1.5 ruleset shape

Two things about VyOS 1.5 catch people carrying 1.3-era habits, and they compound: a named ruleset does nothing until a base chain jumps to it, and a rule with no state clause is not stateful.

configure

set firewall ipv4 name WAN-IN default-action 'drop'
set firewall ipv4 name WAN-IN default-log

set firewall ipv4 name WAN-IN rule 10 action 'accept'
set firewall ipv4 name WAN-IN rule 10 description 'return traffic for flows we started'
set firewall ipv4 name WAN-IN rule 10 state 'established'
set firewall ipv4 name WAN-IN rule 10 state 'related'

set firewall ipv4 name WAN-IN rule 20 action 'drop'
set firewall ipv4 name WAN-IN rule 20 description 'out-of-state noise'
set firewall ipv4 name WAN-IN rule 20 state 'invalid'

set firewall ipv4 name WAN-IN rule 30 action 'accept'
set firewall ipv4 name WAN-IN rule 30 description 'inbound HTTPS to the router'
set firewall ipv4 name WAN-IN rule 30 protocol 'tcp'
set firewall ipv4 name WAN-IN rule 30 destination port '443'
set firewall ipv4 name WAN-IN rule 30 state 'new'

set firewall ipv4 input filter default-action 'drop'
set firewall ipv4 input filter rule 10 action 'jump'
set firewall ipv4 input filter rule 10 jump-target 'WAN-IN'
set firewall ipv4 input filter rule 10 inbound-interface name 'eth0'

commit
save

Read the last block first. WAN-IN is a named ruleset and named rulesets are not attached to anything by themselves — they are jumped to. set firewall ipv4 input filter is the base chain for traffic addressed to the router; forward filter is the base chain for traffic passing through it. The action jump plus jump-target pair is what connects them.

On VyOS 1.3 the attachment was a property of the interface: set interfaces ethernet eth0 firewall in name WAN-IN. That node is gone. A 1.3 runbook pasted onto 1.5 produces a ruleset that commits cleanly, appears in show firewall, counts zero packets on every rule, and filters nothing — which looks exactly like a firewall that is working until somebody checks the counters.

Stateful versus stateless on the wire

A stateless ruleset judges each packet alone. Permitting an outbound HTTPS request means also writing a rule that permits traffic from port 443 back to the ephemeral range — a rule that, by construction, admits anything from any web server to any high port, whether you asked for it or not:

set firewall ipv4 name WAN-IN rule 40 action 'accept'
set firewall ipv4 name WAN-IN rule 40 description 'stateless return - permits far more than intended'
set firewall ipv4 name WAN-IN rule 40 protocol 'tcp'
set firewall ipv4 name WAN-IN rule 40 source port '443'
set firewall ipv4 name WAN-IN rule 40 destination port '1024-65535'

The stateful version is rule 10 in the ruleset above: state established and state related, and no port arithmetic at all. It admits exactly the replies to conversations this router already accepted, and nothing else.

The difference is not mainly about rule count. It is that the stateless pair cannot express “the reply to something we permitted”, so it approximates it with a port range — and that approximation is a permanent hole sized to the whole ephemeral space.

Performance: what is true and what is not

The common claim is that an established packet is “fast-pathed” and skips the firewall rules. That is not how nftables works. Every packet traverses the chains it is eligible for, every time. There is no implicit bypass.

What is true is cheaper and less magical. A conntrack state match is a single hash lookup, and a ruleset that puts state established accept at the top resolves the overwhelming majority of packets on the first rule. The saving comes from where you put that rule, not from the kernel deciding to skip work on your behalf. Put the state rule at the bottom of a fifty-rule chain and every data packet walks all fifty.

• First packet of a flow. Conntrack creates an entry, and the packet walks the chain until a rule matches. This is the expensive one, and it is expensive in proportion to how far down the matching rule sits.
• Subsequent packets. Conntrack looks the flow up, and the early state rule matches. Cost: one lookup plus the rules above it.

The memory side is straightforward. Every tracked flow occupies an entry — a few hundred bytes — and a router holding a million concurrent flows is spending a few hundred megabytes of kernel memory on the table alone. That is the real ceiling on a busy edge, not CPU.

Sizing conntrack

set system conntrack table-size 524288
set system conntrack hash-size 65536

When the table fills, the kernel drops the packets that would have created new entries and logs nf_conntrack: table full to the ring buffer. Existing flows keep working. The production signature is unmistakable once you have seen it: the operator’s own SSH session is fine, and every new connection to anything fails. The firewall counters show nothing, because nothing was dropped by a rule.

Timeouts are the other lever, and on VyOS 1.5 they are set through custom rules rather than a global node:

set system conntrack timeout custom ipv4 rule 10 protocol tcp established 7200
set system conntrack timeout custom ipv4 rule 10 destination address '10.0.0.0/8'

The custom form scopes a timeout to matching traffic, which is more useful than it first appears: a workload with many short-lived flows can be given a short established timeout without shortening it for the long-lived sessions elsewhere on the same router.

Operational commands

show firewall
show firewall summary
show firewall ipv4 input filter
show firewall ipv4 input filter rule 10

show conntrack table ipv4
show conntrack statistics

clear firewall name WAN-IN counters

show firewall prints every ruleset with per-rule packet and byte counters, which is the first thing to read after any firewall change. A rule that should be matching and shows zero packets is either unreachable behind an earlier rule, or in a ruleset nothing jumps to.

Clear the counters before a test so the numbers you read afterwards belong to the test. clear firewall name WAN-IN counters resets one ruleset.

show conntrack table ipv4 proves whether a flow is tracked at all. A flow that is in the table with traffic counted in both directions is being forwarded; a flow in the table with packets in one direction only tells you the reply is not arriving, which is a different investigation entirely. show conntrack statistics carries the table occupancy — watch it against table-size, because the gap between them is the number of new connections the router can still accept.

Rollback

compare
commit-confirm 5
confirm

Firewall changes cut access. commit-confirm is not optional discipline here; it is the difference between a mistake and a drive to the datacentre. Commit with a timer, verify from a session you did not have open before the change, and only then confirm.

To back a change out afterwards, delete the specific nodes and commit again — a forward change with a diff you can read. rollback N exists but currently requires a reboot to take effect, so it belongs to the recovery case where you already have console access and nothing else has worked.

One thing to check before removing anything: deleting the last rule that accepts established traffic leaves the ruleset dropping every reply to every conversation, including the one you are typing into. Read compare and ask what the ruleset permits after the change, not just what the change removes.

Production discipline

Cross-course references

• Part XXV (XXV-Linux-Firewall) covers nftables and conntrack from the host side. The VyOS firewall is a front-end for the same kernel subsystems.
• Part XXXVII-02 (XXXVII-VyOS-Firewall) covers zones, base chains and the jump model in full.
• Part XXXVII-04 (XXXVII-VyOS-Firewall) covers connmark, connection limits and the other conntrack-aware matches.
• Part LII (LII-VyOS-Troubleshoot) covers the diagnostic method that uses conntrack to establish where a flow is being lost.

Quiz