State tracking — connection marks, recent, and the limits of the conntrack matches

Simple state matching is the basic firewall pattern: admit the return packets of a flow whose first packet you accepted, and reject everything else. On VyOS 1.5 that is state established and state related as values in their own right — the state new enable spelling from 1.3, with a boolean leaf under each state, was removed in 1.4 and does not commit.

set firewall ipv4 name WAN-IN rule 10 action 'accept'
set firewall ipv4 name WAN-IN rule 10 state 'established'
set firewall ipv4 name WAN-IN rule 10 state 'related'

For most production firewalls this is not enough. The operator wants to classify a flow once and act on that classification later, in both directions; to slow down a source that is behaving badly; and to keep the conntrack table itself from becoming the resource that fails.

This lesson covers what VyOS 1.5 gives you for each of those — connection marks, recent, limit, and the conntrack timeout rules — and, just as important, one thing it does not give you at all, so you do not spend an evening looking for it.

Connection marks: classify once, match later

The conntrack entry carries a 32-bit mark of its own, separate from the mark on any individual packet. Because it lives on the connection, it is visible to every packet of that connection in both directions — which is the whole reason to use it.

# Classify: guest VLAN web traffic gets connection mark 100
set firewall ipv4 name LAN-OUT rule 10 action 'accept'
set firewall ipv4 name LAN-OUT rule 10 description 'Classify guest web traffic'
set firewall ipv4 name LAN-OUT rule 10 source address '192.168.100.0/24'
set firewall ipv4 name LAN-OUT rule 10 protocol 'tcp'
set firewall ipv4 name LAN-OUT rule 10 destination port '80,443'
set firewall ipv4 name LAN-OUT rule 10 state 'new'
set firewall ipv4 name LAN-OUT rule 10 set connection-mark '100'

# Act on the classification, anywhere later
set firewall ipv4 name WAN-OUT rule 10 action 'accept'
set firewall ipv4 name WAN-OUT rule 10 description 'Guest web, rate controlled'
set firewall ipv4 name WAN-OUT rule 10 connection-mark '100'
set firewall ipv4 name WAN-OUT rule 10 limit rate '200/second'
set firewall ipv4 name WAN-OUT rule 10 limit burst '400'

Two node names carry the whole idea and they are easy to confuse:

• set connection-mark <value> writes the mark onto the conntrack entry. It is an action, and it sits under the rule’s set node alongside set dscp, set mark, set tcp-mss and set ttl.
• connection-mark <value> matches it. It is a matcher, at the same level as protocol or state.

Write the mark on the first packet — hence state new on the classifying rule — and every later packet of that connection, including the replies, matches without being re-classified. That is the property a packet mark does not have.

The one this lesson cannot give you: concurrent connections per source

A firewall design that says “no single source address may hold more than 50 connections to this server” is a reasonable design, and it is the first thing operators go looking for after state matching. It is worth being direct:

VyOS 1.5 does not expose a per-source concurrent connection limit. There is no connection-limit node, no connlimit, and no connection limit mask under a firewall rule. The rule’s matchers include connection-mark and connection-status, and its rate controls are limit and recent — all of which count events over time, not connections held at once. Those are different questions, and no combination of the available nodes answers the second one.

The underlying nftables engine can do it, with ct count. VyOS has no CLI for that, so reaching it means writing raw nftables outside the configuration tree — unmanaged by commit, unrestored after a reboot, and invisible to everyone who reads the configuration afterwards. That is a worse outcome than not having the feature, and it is not what this course will teach you to do.

What to do instead, in the order to consider it:

Limit the rate of new connections, which is usually the actual threat. A scanner or a broken client opening thousands of connections is opening them quickly:

set firewall ipv4 name WAN-IN rule 20 action 'accept'
set firewall ipv4 name WAN-IN rule 20 description 'HTTPS, new connections rate limited'
set firewall ipv4 name WAN-IN rule 20 protocol 'tcp'
set firewall ipv4 name WAN-IN rule 20 destination port '443'
set firewall ipv4 name WAN-IN rule 20 state 'new'
set firewall ipv4 name WAN-IN rule 20 limit rate '50/second'
set firewall ipv4 name WAN-IN rule 20 limit burst '100'

Note what limit is and is not: it is a rate on the rule, not per source. It protects the service from the aggregate, and it will also throttle legitimate traffic during a flood. Use it where the service is more fragile than the link.

Limit per source with recent, covered next, which is per-source but still counts events in a window rather than concurrency.

Cap half-open connections, which is the SYN-flood case and is a conntrack setting rather than a firewall rule:

set system conntrack tcp half-open-connections '1000'

Enforce concurrency where concurrency is known. The server holds the connections, so the server counts them best — limit_conn in nginx, MaxConnPerIP equivalents in other daemons, or a load balancer in front. A router counting sockets on behalf of a service it cannot see is the wrong layer for this control even where the feature exists.

Recent: per-source rate tracking

recent tracks source addresses and matches the ones that have been seen too often, too recently. On VyOS 1.5 it is exactly two nodes:

set firewall ipv4 name WAN-IN rule <n> recent count <1-255>
set firewall ipv4 name WAN-IN rule <n> recent time [second|minute|hour]

Read recent time carefully, because it is the node this lesson used to get wrong. It takes a unit, not a number of seconds. recent time minute means “within the last minute”. recent time 60 is not a longer way of writing that; it is rejected. The pair reads as a sentence: this many, within this period.

The production pattern for SSH is two rules — one that drops the sources exceeding the threshold, one that accepts the rest — with the drop first:

set firewall ipv4 name WAN-IN rule 10 action 'drop'
set firewall ipv4 name WAN-IN rule 10 description 'SSH brute force'
set firewall ipv4 name WAN-IN rule 10 protocol 'tcp'
set firewall ipv4 name WAN-IN rule 10 destination port '22'
set firewall ipv4 name WAN-IN rule 10 state 'new'
set firewall ipv4 name WAN-IN rule 10 recent count '4'
set firewall ipv4 name WAN-IN rule 10 recent time 'minute'
set firewall ipv4 name WAN-IN rule 10 log

set firewall ipv4 name WAN-IN rule 20 action 'accept'
set firewall ipv4 name WAN-IN rule 20 description 'SSH'
set firewall ipv4 name WAN-IN rule 20 protocol 'tcp'
set firewall ipv4 name WAN-IN rule 20 destination port '22'
set firewall ipv4 name WAN-IN rule 20 state 'new'

A source that opens more than four SSH connections in a minute is dropped for as long as it keeps trying; a human opening one session is never near the threshold. There are no set, update or check modes to configure — the older iptables recent module had those, and VyOS 1.5 renders this onto an nftables dynamic set where recording and matching are one operation. Likewise there is no /proc/net/xt_recent/ to read: that file belongs to the iptables module VyOS no longer uses. The state lives in an nftables set, visible in sudo nft list ruleset.

Timeouts, and where they moved

How long an entry survives without traffic decides two things at once: whether long-idle connections keep working, and how full the table gets. Both are failure modes, in opposite directions.

The nodes changed shape in 1.4. There is no global set system conntrack timeout tcp established on 1.5 — timeouts are now expressed as custom timeout rules with a selector, so a change applies to the traffic you name rather than to everything the router tracks:

set system conntrack timeout custom ipv4 rule 10 description 'Long-lived database sessions'
set system conntrack timeout custom ipv4 rule 10 destination address '10.20.0.0/24'
set system conntrack timeout custom ipv4 rule 10 destination port '5432'
set system conntrack timeout custom ipv4 rule 10 protocol tcp established '86400'

This is more precise than the global knob it replaced, and the precision is the point: raising the established timeout for one database subnet costs a handful of table slots, while raising it globally multiplies every idle connection on the box. Where a global change is genuinely wanted, it is a kernel sysctl (net.netfilter.nf_conntrack_tcp_timeout_established) and should be recorded as such rather than pretended into the configuration tree.

Sizing the table is separate and still global:

set system conntrack table-size '524288'
set system conntrack hash-size '65536'

The failure mode a short timeout produces

An idle TCP connection whose entry has expired is not politely re-admitted. Its next packet is a mid-stream segment with no conntrack entry, so it is invalid rather than new — a state new accept rule will not match it, and a well-built rule set drops it. The application sees a connection that hangs and eventually resets, minutes after the router quietly forgot about it.

flowchart LR
  T0["Connection established<br/>entry created"]
  T1["Application idle<br/>no packets"]
  T2["Timeout expires<br/>entry removed"]
  T3["Application sends<br/>mid-stream segment"]
  T4["No entry: state invalid<br/>not new, not established"]
  T5["Dropped<br/>application hangs, then resets"]

  T0 --> T1 --> T2 --> T3 --> T4 --> T5

That is why “just lower the timeouts” is not a free way to reclaim table space. The candidates for lowering are protocols whose flows genuinely are short — ICMP, DNS over UDP — and the candidate for raising is the specific long-idle application whose sessions keep dying, addressed by its own rule.

Operational commands

show conntrack table ipv4
show firewall statistics
show firewall ipv4 name WAN-IN

The first is the state itself: one row per tracked connection, with the original and reply tuples, the protocol state, the remaining timeout and the connection mark. The second gives per-rule packet and byte counters, which is how you tell a rule that is matching nothing from a rule that is matching and permitting. The third prints one rule set with its rules in evaluation order.

Below the VyOS layer:

sudo conntrack -L -m 100
sudo conntrack -E -e NEW
sudo nft list ruleset
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

conntrack -L -m 100 filters the table to one connection mark — the fastest confirmation that a classification rule is doing its job. conntrack -E streams events live, which is the tool for “connections are being created and immediately destroyed” questions that a snapshot cannot answer. And the count-against-max pair is the number to graph: a table at capacity drops new connections while every existing one keeps working, which produces the memorable support call where the network is “fine for everyone already connected”.

Rollback

# See what the session would change
compare

# Remove one matcher without disturbing the rest of the rule
delete firewall ipv4 name WAN-IN rule 10 recent count
delete firewall ipv4 name WAN-IN rule 10 recent time
commit

Two cautions specific to state tracking.

Removing a classifier leaves its consumers matching nothing. Delete the rule that carries set connection-mark 100 and every rule matching connection-mark 100 still commits cleanly, still appears in the configuration, and now matches nothing at all. If that rule was an accept, traffic falls through to the default action — which on a well-built rule set is a drop. Remove the pair together, or check with show configuration commands | match connection-mark before you commit.

Existing connections keep their marks. Conntrack entries created before the change still carry the old mark until they expire, so the effect of a rollback is partial for as long as those connections live. That is usually what you want during an incident, and it is occasionally confusing: the new rules are correct and the old connections are still behaving as though they were not.

Production discipline

Cross-course references

• Part XXXVII-01 (XXXVII-VyOS-Firewall / stateful vs stateless) covers the conntrack layer these matches rely on.
• Part XXXVI-05 (XXXVI-VyOS-ECMP / fwmark) covers the policy-routing side of the connmark bridge.
• Part XXXIX-04 (XXXIX-VyOS-MultiWAN / policy routing) covers the set policy route side of the connection-mark bridge, where the mark written here selects a table.
• Part XLVIII-03 (XLVIII-VyOS-Logging / firewall logs) covers the logging of state-tracking events.

Quiz