Firewall Conntrack Table Full

Reported symptoms

The complaints started on Tuesday and nobody could make them repeat on demand.

Users on the DC-East office network say the internet “works, then stops, then works again”. It goes wrong somewhere around 09:10, clears by about 09:50, and comes back after lunch. Reloading a page helps sometimes and does nothing other times. Nobody can produce a failing example while someone is watching.

The router looks fine, and it looks fine in every way an operator normally checks:

• CPU around 22%, no interface errors, no interface drops, no queue buildup.
• BGP and OSPF stable, no adjacency changes, nothing from FRR in the logs.
• The firewall’s accept rules are advancing at their usual rate and the catch-all deny counter is no busier than on a quiet day. On that evidence the on-call engineer ruled the firewall out, which was a reasonable conclusion from a real observation.
• show log firewall shows nothing unusual in the affected windows.

Two things in the incident channel should have been the whole diagnosis and were read as noise instead.

First, two engineers disagreed about whether the router was reachable. One had an SSH session already open and found it completely responsive. The other, opening a new session at the same moment, got nothing until the client gave up. They spent ten minutes deciding one of them had a local network problem.

Second, a blackbox probe that opens a new TCP connection to an external endpoint had been flapping for two days. It was muted last month for being noisy.

Nothing has been changed on the router. The only change on the estate that week is a vulnerability scanner the security team rolled out on Monday.

Evidence provided

$ cat /proc/sys/net/netfilter/nf_conntrack_count /proc/sys/net/netfilter/nf_conntrack_max

262144
262144

$ sudo dmesg -T | grep nf_conntrack | tail -3

[Tue Aug 18 09:11:44 2026] nf_conntrack: nf_conntrack: table full, dropping packet
[Tue Aug 18 09:11:44 2026] nf_conntrack: nf_conntrack: table full, dropping packet
[Tue Aug 18 09:11:45 2026] nf_conntrack: nf_conntrack: table full, dropping packet

$ sudo conntrack -S

cpu=0 found=0 invalid=214 insert=0 insert_failed=8814 drop=8814 early_drop=0 error=0 search_restart=1902
cpu=1 found=0 invalid=190 insert=0 insert_failed=9022 drop=9022 early_drop=0 error=0 search_restart=1877

$ sudo conntrack -L 2>/dev/null | grep -c SYN_SENT

241377

$ sudo conntrack -L 2>/dev/null | awk '/^tcp/{print $5}' | sort | uniq -c | sort -rn | head -4

239104 src=10.30.0.40
 1183 src=10.30.0.61
  908 src=10.30.0.62
  644 src=203.0.113.61

$ show firewall

Rule     Action   Packets     Bytes   Description
----     ------   -------     -----   -----------
10       accept   3184402     441M    Established/related
20       drop     37          2.9K    Invalid
30       accept   1204        88K     SSH from jump host
9999     drop     14022       841K    Default deny

$ show interfaces

Work the evidence before reading on

The hardest part of this incident is that a correct observation sent the investigation the wrong way. Work out which one.

1. An SSH session opened before 09:10 is responsive throughout the window, and a new one cannot be established. What is different about the two from the router’s point of view, and which subsystem knows that difference?
2. The firewall counters are normal and the catch-all deny is flat. If the packets are being discarded, and the firewall is not the thing discarding them, what else in the packet path is in a position to?
3. 241,377 of 262,144 entries are in SYN_SENT. What kind of traffic produces a half-open flow, in bulk, that never progresses — and what does it tell you that 91% of them share one source?
4. The windows start and end at the same times on consecutive weekdays. No router change was made. What kind of cause has a schedule?

Before continuing: at what point in the packet path is a packet dropped, if the drop leaves no trace in any firewall counter and no line in the firewall log?

Root cause

1. A full table drops the packets that would create entries

Connection tracking is a fixed-size hash table in kernel memory. When it is full, the kernel does not evict, does not queue and does not admit the packet untracked — it drops the packets that would have created new entries.

That produces a very specific failure shape, and the shape is the diagnosis. A flow that already has an entry is matched, fast-pathed and entirely unaffected. A flow that does not — a new SSH session, a new page load, a new probe — is discarded. Everything that was working keeps working, and nothing new can start.

This is why two engineers on the same router disagreed about whether it was up. One was asking a question the table already had an answer to, and the other was asking for a new entry there was no room to create.

2. The drop happens before the firewall, so the firewall cannot report it

Conntrack sits ahead of the filter chains in the packet path. A packet discarded for want of a table slot never reaches a rule, so it increments no rule counter and produces no log line, including from the default-deny rule and its logging.

The on-call engineer looked at the firewall counters, saw nothing anomalous, and concluded the firewall was not involved. That observation was completely correct and the conclusion drawn from it was not: the firewall really was not dropping this traffic, and the subsystem that was sits one hop earlier and keeps its accounting somewhere else entirely.

conntrack -S is where that accounting lives. insert_failed and drop are the only counters on the box that move when this happens, and nothing routinely graphs them.

3. What filled the table was traffic that is never meant to complete

The scanner deployed on Monday sweeps the estate on weekday mornings and again after lunch. Every probe it sends to a port that does not answer leaves a flow parked in SYN_SENT until that state times out, and the default is generous because it is chosen to survive a slow handshake rather than to survive a sweep.

At a few thousand probes a second, that parks hundreds of thousands of entries which will never progress and never be cleaned up early. The windows end when the sweep does and the parked entries age out over the following couple of minutes, which is exactly the recovery pattern the users described.

User traffic was never the problem. The table was consumed by flows that had no intention of becoming connections.

4. Underneath all of it, a number nobody owned

The table size was set once, against a flow count measured at the time, and that measurement was correct when it was taken. Nothing re-measured it when the estate acquired a workload whose entire purpose is to open connections that do not complete.

That is the part worth carrying away, because it is not specific to conntrack. A capacity figure derived from a measurement is only as current as the measurement, and nothing about it announces when it has gone stale.

Resolution

1. Identify the consumer before touching any setting. Grouping the conntrack table by source takes one command and under a minute, and it decides everything that follows: one host holding 91% of the table is a completely different problem from a table that filled evenly.
2. Restore service. Raising the table size with set system conntrack table-size takes effect on commit and is reversible, which makes it the right immediate action — recorded in the incident as a hold, with an owner and a date, not as the resolution.
3. Take the driver to its owner. Pausing, rate-limiting or rescoping the sweep so that it does not cross the stateful edge is the cheapest real fix available, and it is a change in the tool owned by the security team rather than in your router. A message in a channel is not an owner.
4. Consider the timeout that is parking the entries, and price it honestly. Shortening the state that half-open flows sit in frees the table faster, and it also means a legitimate connection to a slow or briefly unreachable destination can lose its state before the retransmission that would have completed it.
5. Add the structural control: a per-source connection cap, so that no single host can consume the table whatever it is doing. Size the limit and the mask from measured behaviour — a guessed cap will eventually be hit by a proxy or a mail relay that is behaving perfectly, and that failure will look exactly like this one.
6. Re-measure the baseline once the estate is quiet, and set the table size from that measurement plus room for an abnormal day, rather than leaving whatever number the incident happened to end on.

Verification

1. Verify across a whole scan window, not on a quiet afternoon. Every check in this list passes at 16:00 on a Friday and none of them mean anything there.
2. The count peaks with headroom. Through the window, nf_conntrack_count must stay visibly below nf_conntrack_max rather than pinning at it.
3. The failure counters stay flat. sudo conntrack -S must show insert_failed and drop unchanged from the start of the window to the end, and sudo dmesg -T must produce no new table-full lines.
4. Reproduce the exact thing that failed. Open a new SSH session from the jump host in the middle of a window. Confirming that an existing session still works reproduces the symptom that misled everybody, not the fault.
5. Unmute the blackbox probe that opens a new TCP connection, and require it green across a full window. It was the only check that ever saw this, and it is the check that tells you whether the fix held next month.
6. If a per-source cap was added, prove it fires by exceeding it deliberately from a test host, and prove it does not fire for legitimate high-fan-out hosts by watching them through a busy period. An untested cap is an outage waiting for a mail relay.

Prevention

• Alert on the ratio of nf_conntrack_count to nf_conntrack_max, not on the count. The count is meaningless without the maximum, and the ratio needs to fire while there is still headroom to act — around three quarters is early enough to be useful and late enough not to be noise.
• Give this failure its own signal. The firewall counters, the interface counters and the CPU graph are all blind to it by construction. The kernel ring buffer and conntrack -S are where it is visible; forward the kernel log to central syslog and graph the two counters.
• Size the table from a measured peak with room for an abnormal day, and re-measure whenever the estate acquires a workload that changes the shape of its traffic. A capacity number is only as current as the measurement behind it.
• Set the protocol timeouts to what the traffic actually does. The defaults are chosen for a general-purpose host, not for an edge router carrying an estate’s flow table.
• Cap per-source consumption structurally. A per-source connection limit turns “the estate is down” into “one host is being refused”, which is the difference between an incident and a ticket.
• Decide who may open flows through the stateful edge before the tool that opens them is deployed. A scanner sweeping through a stateful device is a capacity decision, and the person who owns the scanner is usually not the person who finds out.
• Treat a flapping check as unread signal rather than as noise. The one probe that detected this was muted a month before the incident, for behaving exactly as it should have.