Break/Fix: 2% of connections time out and the server log has no record of them

Reported symptoms

• Roughly 2% of requests to the API fail. The client reports a connection timeout, not a connection refused and not an HTTP error.
• The API access log has no entry for the failed requests. Not a 500, not a 499, nothing. As far as the API is concerned they never happened.
• Failures cluster between 08:00 and 19:00 and vanish overnight.
• A retry succeeds immediately, so the client library’s retry policy hides most of it — which is why this ran for three weeks before anyone filed a ticket.
• Every host-level metric is boring: CPU 22%, memory 40%, no disk pressure, listen backlog empty, no socket accept queue overflow.
• The network team sees no loss on any link and no errors on any port.

Evidence provided

$ sudo dmesg -T | grep -i conntrack | tail -3
[Tue Aug 11 09:41:07 2026] nf_conntrack: table full, dropping packet
[Tue Aug 11 09:41:12 2026] nf_conntrack: table full, dropping packet
[Tue Aug 11 09:43:55 2026] nf_conntrack: table full, dropping packet

$ cat /proc/sys/net/netfilter/nf_conntrack_count /proc/sys/net/netfilter/nf_conntrack_max
262144
262144

$ sudo conntrack -S | head -4
cpu=0   found=41028 invalid=118 insert_failed=9944 drop=9944 early_drop=0 error=0 search_restart=88121
cpu=1   found=39877 invalid=104 insert_failed=9718 drop=9718 early_drop=0 error=0 search_restart=86440
cpu=2   found=40551 invalid=97  insert_failed=9803 drop=9803 early_drop=0 error=0 search_restart=87002
cpu=3   found=40194 invalid=111 insert_failed=9871 drop=9871 early_drop=0 error=0 search_restart=87719

$ sudo conntrack -L 2>/dev/null | awk '{print $4}' | sort | uniq -c | sort -rn | head -4
 248911 TIME_WAIT
  11402 ESTABLISHED
   1188 SYN_SENT
    643 CLOSE_WAIT

$ sudo conntrack -L 2>/dev/null | grep -c 'dport=9100'
247803

$ ss -s
Total: 14118
TCP:   9042 (estab 8811, closed 118, orphaned 0, timewait 118)

$ sudo nft list ruleset | grep -n 'ct '
14:		ct state established,related accept

$ sysctl net.netfilter.nf_conntrack_tcp_timeout_time_wait net.netfilter.nf_conntrack_tcp_timeout_established
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_established = 432000

Work the evidence before reading on

Five things to reconcile:

1. nf_conntrack_count equals nf_conntrack_max exactly. That is not a coincidence and it is not a rounding artefact.
2. ss -s reports 14,118 sockets on this host. conntrack -L reports 262,144 entries. Where are the other quarter of a million?
3. 247,803 of those entries have dport=9100. The API listens on 8443.
4. The API log has no record of the failed requests, and the client saw a timeout rather than a refusal. Work out at which point in the TCP handshake a packet has to be lost to produce exactly that pair of observations.
5. nf_conntrack_tcp_timeout_established is 432000 seconds. Convert it to days before continuing.

Root cause

1. A dropped SYN produces a timeout and no log entry

The distinction the ticket buried is the most diagnostic fact in it:

• Connection refused — the SYN arrived, nothing was listening, the kernel sent a RST. Fast, and the client knows immediately.
• Connection timeout — the SYN was never answered. The client retransmits it on the kernel’s SYN backoff schedule and eventually gives up.

A timeout with an empty server log means the SYN did not reach the listening socket. The packet was dropped below the application, and on this host the only thing below the application capable of dropping a SYN is netfilter.

2. Connection tracking is on, on a host nobody firewalls

The ruleset contains exactly one stateful rule:

ct state established,related accept

That single rule is enough to load nf_conntrack. And here is the part that surprises people: once the module is loaded and a hook is registered, every packet traversing the host is tracked, not only the packets that the stateful rule matches. Connection tracking is not a per-rule feature; it is a subsystem the rule switches on.

So a host with no NAT, no real filtering policy, and one habitual copy-pasted accept rule has a finite, shared, unmonitored table sitting in the path of all its traffic.

3. The arithmetic

The occupancy of the conntrack table is, to a good approximation:

entries  =  new connections per second  x  how long an entry is kept

The metrics scraper polls 3,000 endpoints on port 9100, once per second, each on a fresh TCP connection that it closes cleanly. A cleanly closed connection leaves a conntrack entry in TIME_WAIT, held for nf_conntrack_tcp_timeout_time_wait — 120 seconds by default.

3,000/s  x  120 s  =  360,000 entries

against nf_conntrack_max = 262144. The table is structurally too small for the workload, and it saturates every morning when scraping ramps up.

Note that ss -s does not show these. The sockets themselves were reaped long ago; conntrack keeps its own record for its own timeout, and the two have nothing to do with each other. Looking at socket counts to estimate conntrack usage is the most common wrong turn here.

4. Why the API is the victim and the scraper is fine

When the table is full, the kernel drops the packet that would have created a new entry. Existing entries are untouched, so established connections carry on perfectly — which matches the report exactly.

The scraper’s connections mostly survive because it is generating so many that it wins on volume, and because its own retries are invisible. The API’s much smaller share of new connections is subject to the same lottery, and it loses about 2% of them.

The service that fails and the service that fills the table share nothing except one kernel table. No amount of investigating the API can find this.

Resolution

1. Confirm the drops are current, not historical. Take two sudo conntrack -S samples five minutes apart and compare insert_failed. A counter that is high but static is a past event; one that is climbing is your incident
2. Find out what is in the table before changing its size. Grouping by destination port, by state, and by source tells you whether you have a sizing problem or a leak:
3. `` sudo conntrack -L 2>/dev/null | awk '{print $4}' | sort | uniq -c | sort -rn | head ``
4. Buy headroom immediately. Raise the maximum and the hash bucket count together — a larger table with the original bucket count means longer hash chains and more CPU per lookup, which trades a drop for a slowdown:
5. `` sudo sysctl -w net.netfilter.nf_conntrack_max=1048576 echo 262144 | sudo tee /sys/module/nf_conntrack/parameters/hashsize ``
6. Check the memory cost you just accepted. Roughly 300 bytes per entry, so a million entries is about 300 MB of unswappable kernel memory. On a host with headroom this is fine; on a memory-constrained one it is a different incident
7. Remove the demand rather than only accommodating it. The scraper traffic needs no stateful filtering, so exempt it from tracking entirely in the raw table:
8. `` table ip raw { chain prerouting { type filter hook prerouting priority raw; policy accept; tcp dport 9100 notrack } chain output { type filter hook output priority raw; policy accept; tcp sport 9100 notrack } } ``
9. Both directions matter. A notrack on one hook only means the reply still creates an entry, and you have halved the problem rather than removed it
10. Reconsider whether this host needs conntrack at all. If the only stateful rule is a blanket established/related accept, with no NAT and no policy that depends on state, removing it removes the entire failure class. Do this deliberately and with the security team, not casually
11. Persist every sysctl. A sysctl -w value is gone at the next reboot and the incident returns looking brand new:
12. `` printf 'net.netfilter.nf_conntrack_max = 1048576\n' | sudo tee /etc/sysctl.d/60-conntrack.conf sudo sysctl --system ``
13. The hashsize is a module parameter, not a sysctl. Persist it in /etc/modprobe.d/ so it applies when the module loads at boot

Verification

1. The table has headroom at peak. Sample cat /proc/sys/net/netfilter/nf_conntrack_count repeatedly during the busiest hour and confirm it sits well below the maximum. Measuring at 04:00 proves nothing
2. Drops have stopped, measured as a delta. Record sudo conntrack -S, wait five minutes at peak, record again, and confirm insert_failed and drop are unchanged. This is the check that can fail, and the absolute values will stay high from the incident — only the delta is meaningful
3. The kernel has stopped complaining. sudo dmesg -T | grep -i conntrack shows no line newer than the change. The message is rate-limited, so absence over a short window is weak evidence; check across a full peak period
4. **notrack is actually taking effect.** sudo conntrack -L 2>/dev/null | grep -c dport=9100 should fall to near zero. If it does not, the rule is on the wrong hook, the wrong table, or the wrong priority
5. A real client succeeds. Run several hundred sequential connects from an affected client during peak and confirm 100% success. At a 2% failure rate, ten attempts have an 82% chance of showing nothing at all
6. Memory is still healthy. free -m and grep -i slab /proc/meminfo after the table grew. A larger table is memory you have committed permanently
7. It survives a reboot. Reboot and re-check both nf_conntrack_max and the hashsize. The sysctl and the module parameter persist in different places and it is easy to fix one and forget the other
8. Monitoring now sees it. Add the utilisation ratio to the dashboard and confirm it reports a plausible value, so the next approach to the ceiling is visible before it is an outage

Prevention

• Alert on utilisation, expressed as count divided by max, with a threshold around 80%. An alert on “table full” fires after the outage has begun. This one is two divisions and a comparison and it would have caught the problem weeks earlier.
• Know which hosts track connections and why. Installing Docker, podman or kube-proxy inserts NAT rules and loads nf_conntrack on hosts nobody has ever thought of as firewalls, complete with a default-sized table.
• Use notrack for high-volume traffic that needs no state: health checks, metrics scrapes, backup streams, log shipping. Every entry you do not create is one you never have to size for.
• Review the long timeouts once, deliberately, rather than inheriting five days by accident.
• When an intermittent failure has no server-side log entry, work downwards rather than sideways. The absence of a log line is information: it locates the drop below the application, which is a short list.
• Keep net.ipv4.tcp_keepalive_time shorter than the shortest conntrack or firewall idle timeout on the path, so long-lived idle connections refresh state instead of being silently forgotten.