Runbook: Troubleshoot a firewall that is dropping traffic

Most firewall work in production is not construction. It is diagnosis: a service was reachable yesterday and is not today, and the ruleset has three hundred lines that four people wrote. This runbook finds the rule that drops the packet.

The companion runbook linux-runbook-recover-from-firewall-lockout covers the case where the firewall has locked you out. This one covers the case where it has locked a client out, and you still have a shell.

When to use this runbook

• A client times out connecting to a service that is listening and healthy.
• The service works from the host itself (curl localhost) but not from the network.
• The service works from one network and not another.
• IPv4 works and IPv6 does not, or the reverse.
• The service started failing after a firewall reload, a deploy, a package update, or a container start.

Step 1: Prove where the packet stops

Do not read rules yet. Narrow the failure first, because each test eliminates a whole class of cause:

# On the server: is anything listening, and on which address?
ss -tlnp '( sport = :443 )'

# From the server itself - bypasses the input chain entirely
curl -sS -m 5 -o /dev/null -w '%{http_code}\n' https://127.0.0.1/

# From another host on the SAME subnet - no routing, no WAN firewall
nc -vz -w 5 <server-ip> 443

# From the real client network
nc -vz -w 5 <server-ip> 443

Read the results:

A timeout and a connection reset mean different things. A timeout is a silent drop, which is what a firewall drop verdict produces. A reset usually means something answered: either the service refused, or a rule used reject with tcp reset.

Step 2: Read the live ruleset, not the file

The ruleset that filters packets is the one in the kernel. The file on disk is a hope.

# -a prints handles, which you need to delete a specific rule
sudo nft -a list ruleset

# What does the boot file say?
sudo nft -f /etc/nftables.conf --check   # syntax only, applies nothing

# Diff live against disk
diff <(sudo nft list ruleset) <(sudo nft -f /etc/nftables.conf --check --echo 2>/dev/null) \
&& echo "live and disk agree"

Two failure modes show up here constantly:

Duplicated rules. A ruleset file without a leading flush ruleset that has been loaded twice leaves two copies of every chain. The first matching copy wins, and the copy you are editing may be the second one.

Live drift. Someone made a runtime change during an incident and never persisted it, or persisted a change and never reloaded. Either way, editing the file changes nothing until a reload - and the reload will also apply everything else that has drifted.

# Count how many times a chain has been declared
sudo nft list ruleset | grep -c 'chain input'

Step 3: Trace the packet

nft monitor trace shows every rule a packet is evaluated against and the verdict that ends it. This is the tool that turns a three-hundred-line ruleset into one line.

# Mark the traffic you care about - narrow it hard
sudo nft add rule inet filter prerouting \
ip saddr 203.0.113.10 tcp dport 443 meta nftrace set 1

# In a second terminal
sudo nft monitor trace

Then reproduce the failure from the client. The trace names the table, chain, rule handle and verdict:

trace id 3f2a inet filter input packet: iif "eth0" ip saddr 203.0.113.10 ...
trace id 3f2a inet filter input rule ct state established,related accept (verdict continue)
trace id 3f2a inet filter input verdict drop

A verdict of drop with no rule named means the chain policy dropped it: no rule matched, and the default is drop. That is a missing allow, not a wrong deny.

Remove the trace rule when finished - it has a real cost under load:

sudo nft -a list chain inet filter prerouting   # find the handle
sudo nft delete rule inet filter prerouting handle <n>

Step 4: Distinguish a policy drop from a state drop

A stateful ruleset accepts established,related early. If conntrack has no entry for the flow, or has one in an unexpected state, the packet falls through to the policy and is dropped - even though every rule is correct.

# Watch the flow appear (or not)
sudo conntrack -E -p tcp --dport 443

# Is the table full? A full table drops new flows silently.
sudo conntrack -C
sysctl net.netfilter.nf_conntrack_max
dmesg | grep -i 'conntrack table full'

Three signatures worth recognising:

• Nothing in conntrack -E: the packet never reached the state engine. It is being dropped before, or it never arrived - go back to Step 1.
• [UNREPLIED] entries that never complete: the SYN arrived and the reply did not come back. Asymmetric routing or a return-path filter, not an input rule.
• Table at nf_conntrack_max: capacity, not policy. The symptom is intermittent and load-correlated, which is why it gets misdiagnosed as a rule problem.

Step 5: Find the other writers

nft list ruleset is authoritative only if nothing else is programming netfilter. On a real host, several things are.

# Is a container runtime publishing ports?
sudo nft list table ip nat | grep -i docker
sudo iptables -t nat -L DOCKER -n 2>/dev/null
docker ps --format '{{.Names}}\t{{.Ports}}'

# Legacy and nft backends can both be populated
sudo iptables-legacy -L -n -v 2>/dev/null | head
sudo iptables-nft -L -n -v 2>/dev/null | head

# firewalld direct rules bypass the zone model
sudo firewall-cmd --direct --get-all-rules

# ufw is a wrapper - read what it generated, not what you typed
sudo ufw status verbose

Step 6: Check IPv6 separately

inet tables filter both families. ip and ip6 tables filter one each. A ruleset that mixes them almost always has a family whose policy nobody tested.

sudo nft list table ip6 filter
ss -tln | grep ':443'          # is the service even bound on v6?
nc -6 -vz -w 5 <server-v6> 443
getent ahosts service.example.com   # is the client resolving to AAAA first?

The specific trap: dropping all ICMPv6. IPv6 requires Neighbour Discovery and Packet Too Big, both ICMPv6. Drop them and connectivity degrades in ways that look nothing like a firewall: neighbours go stale, and large responses blackhole while the handshake succeeds.

# Minimum ICMPv6 that must be permitted
sudo nft add rule inet filter input meta l4proto ipv6-icmp \
icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, \
              parameter-problem, echo-request, nd-neighbor-solicit, \
              nd-neighbor-advert, nd-router-advert } accept

Step 7: Fix, verify, then persist

Apply the narrowest change that works, at runtime only:

sudo nft insert rule inet filter input \
ip saddr 203.0.113.0/24 tcp dport 443 accept

Verify from the real client, from outside the host:

# From the client
nc -vz -w 5 <server-ip> 443

# From a management host - what is actually exposed?
nmap -Pn -p 22,80,443,3306 <server-ip>
nmap -6 -Pn -p 22,80,443 <server-v6>

Only then persist, and reload once so that live and disk are known to agree:

sudo cp /etc/nftables.conf /var/backups/nftables.conf.$(date +%F-%H%M)
sudo nft list ruleset | sudo tee /etc/nftables.conf >/dev/null
sudo nft -f /etc/nftables.conf --check
sudo systemctl reload nftables

Four scenarios worth practising

1. Live differs from disk. Load a ruleset file twice without flush ruleset, then delete a drop rule from the file and reload. Traffic is still dropped, because the duplicate chain still holds a copy. Find it with nft -a list ruleset and the chain count from Step 2.

2. A shadowed allow. Append an accept for a network after an existing drop for a wider prefix. nft list ruleset shows the allow present and correct, and the traffic is still dropped. Only the trace in Step 3 shows which rule ends the evaluation. Order matters; presence does not.

3. A container that bypasses ufw. Run a container with -p 8080:80 on a host with ufw default deny incoming. ufw status reports the port closed; nmap from outside reports it open. Reconcile with Step 5.

4. IPv6 fails, IPv4 works. Drop all ICMPv6 in the ip6 table. The handshake succeeds and large responses hang. Diagnose with Step 6 and the MTU symptoms in linux-mtu-and-jumbo-frames.

Each one is solved by evidence rather than by re-reading the file: nft -a list ruleset, nft monitor trace, conntrack -E, and an external scan.

Escalation

Escalate when:

• The trace shows the packet never reaching the host. The loss is upstream - network ACL, cloud security group, or routing.
• A container runtime is publishing ports that contradict the host policy. That is a platform decision, not a firewall fix.
• The host is a cluster node and the affected port is corosync, DRBD, or a fence device. Changing rules there can trigger fencing; agree the change first.

Bring: the failing 5-tuple, the trace output, the live-vs-disk diff, and the external scan.