NAT troubleshooting — hairpin failure, port exhaustion, asymmetric routing, conntrack flush

NAT troubleshooting is the discipline of finding the break in the chain: is the NAT rule firing? Is the conntrack entry being created? Is the firewall permitting the traffic? Is the routing correct?

The conntrack table is the source of truth. Every NAT translation is recorded in the conntrack table; every return packet is matched against the conntrack entry. If the conntrack entry is correct, the NAT is working. If the conntrack entry is stale, the NAT is broken.

This lesson covers the diagnostic method, the conntrack table as the source of truth, the hairpin NAT failure, port exhaustion, asymmetric routing, and the conntrack flush procedure.

The diagnostic method

The NAT diagnostic method is a five-step walk:

1. Verify the NAT rule — is the rule in the configuration? Is it at the correct hook? Does it match the traffic?
2. Verify the conntrack entry — is the entry created? Does it have the correct translation? Is it in the ESTABLISHED state?
3. Verify the firewall rule — does the firewall see the post-NAT traffic? Is the firewall permitting it?
4. Verify the routing — is the destination reachable? Is the return path correct?
5. Flush conntrack — is the conntrack entry stale?

flowchart TB
  S1["Step 1: Verify the NAT rule<br/>show nat source<br/>show nat destination"]
  S2["Step 2: Verify the conntrack entry<br/>conntrack -L<br/>conntrack -E"]
  S3["Step 3: Verify the firewall rule<br/>show firewall name<br/>show log firewall"]
  S4["Step 4: Verify the routing<br/>show ip route<br/>tcpdump on each interface"]
  S5["Step 5: Flush conntrack<br/>conntrack -F<br/>retest"]
  FIX["Diagnosis complete<br/>fix and re-test"]

  S1 --> S2
  S2 --> S3
  S3 --> S4
  S4 --> S5
  S5 --> FIX

The five steps are sequential. Skipping a step leads to a misdiagnosis.

Tool 1: the conntrack table

The conntrack table is the source of truth for NAT. Every NAT translation is recorded; every return packet is matched against the entry.

# Show all conntrack entries
sudo conntrack -L

# Show entries from a specific source
sudo conntrack -L -s 192.168.1.100

# Show entries to a specific destination
sudo conntrack -L -d 203.0.113.10

# Live event stream
sudo conntrack -E

# Show the table size and current count
cat /proc/sys/net/netfilter/nf_conntrack_max
cat /proc/sys/net/netfilter/nf_conntrack_count

# Flush the entire conntrack table
sudo conntrack -F

# Delete a specific entry
sudo conntrack -D -s 192.168.1.100 -d 203.0.113.10

A typical entry for SNAT’d outbound traffic:

tcp 6 300 ESTABLISHED src=192.168.1.100 dst=198.51.100.5
  sport=52012 dport=443 packets=12 bytes=1844 [ASSURED]
  src=198.51.100.5 dst=203.0.113.10 sport=443 dport=52012
  packets=8 bytes=4096

The fields tell the operator:

• Original direction: src=192.168.1.100 (pre-SNAT source), dst=198.51.100.5 (original destination).
• Reply direction: src=198.51.100.5 (original destination), dst=203.0.113.10 (post-SNAT source).

If the entry is missing, the NAT is not happening. If the entry has the wrong translation, the NAT rule is wrong. If the entry is stale, the rule was changed without a conntrack flush.

Tool 2: the NAT rule rendering

The NAT rules are rendered into nftables. The operator inspects the rendered rules to verify the configuration is correct.

# Show the rendered NAT rules
sudo nft list chain inet nat prerouting
sudo nft list chain inet nat postrouting

# Show the full ruleset
sudo nft list ruleset

# Live trace
sudo nft monitor

A typical rendering of a DNAT rule:

table inet nat {
  chain prerouting {
    type nat hook prerouting priority -100; policy accept;
    tcp dport 443 dnat to 192.168.1.100:443
  }
}

A typical rendering of a SNAT rule:

table inet nat {
  chain postrouting {
    type nat hook postrouting priority 100; policy accept;
    ip saddr 192.168.1.0/24 oifname "eth0" snat to 203.0.113.10
  }
}

The operator verifies:

• The rule is in the correct chain (prerouting for DNAT, postrouting for SNAT).
• The match conditions are correct (protocol, source, destination, port).
• The translation address is correct.

Tool 3: packet capture

The packet capture proves whether the packet reaches the router and whether it has been translated.

# Capture on the WAN interface (pre-NAT)
sudo tcpdump -ni eth0 'host 203.0.113.10 and port 443'

# Capture on the LAN interface (post-NAT for inbound)
sudo tcpdump -ni eth1 'host 192.168.1.100 and port 443'

# Capture with verbose output
sudo tcpdump -ni eth0 -vv 'host 203.0.113.10 and port 443'

# Capture and write to a file
sudo tcpdump -ni eth0 -w /tmp/nat-capture.pcap

For inbound DNAT traffic:

• Pre-NAT packet (on eth0): dst=203.0.113.10:443.
• Post-NAT packet (on eth1): dst=192.168.1.100:443.

For outbound SNAT traffic:

• Pre-NAT packet (on eth1): src=192.168.1.100.
• Post-NAT packet (on eth0): src=203.0.113.10.

The packet capture proves the NAT is happening. If the pre-NAT packet arrives but the post-NAT packet does not, the NAT rule is not firing (or the firewall is dropping the post-NAT packet).

The hairpin failure

The hairpin NAT failure is the most common production NAT problem. The scenario: a LAN host tries to reach the public IP of a service exposed via port forwarding.

The diagnostic method:

1. Verify the DNAT rule — is the rule in place?
2. Verify the hairpin SNAT rule — is the SNAT rule for LAN-originated traffic in place?
3. Verify the conntrack entry — does the entry have both NATs applied?
4. Test from the LAN — does the LAN client succeed?

The conntrack entry for hairpin traffic should have both NATs:

tcp 6 300 ESTABLISHED src=192.168.1.50 dst=203.0.113.10
  sport=52012 dport=443 packets=12 bytes=1844 [ASSURED]
  src=192.168.1.100 dst=192.168.1.1 sport=443 dport=52012
  packets=8 bytes=4096

If the reply direction has dst=192.168.1.50 (not 192.168.1.1), the hairpin SNAT is not happening. The fix: add the hairpin SNAT rule.

Port exhaustion

Port exhaustion is the failure mode for SNAT’d traffic when the number of concurrent flows exceeds the available ephemeral ports. The Linux kernel uses ports 32768-60999 by default (28,232 ports per IP).

The diagnostic:

1. Verify the conntrack count — is it approaching the table max?
2. Verify the per-source flow count — is a single source opening too many flows?
3. Verify the per-IP port usage — is a single public IP running out of ports?

The error in the kernel log: nf_conntrack: nf_nat: no unique tuple found (the kernel cannot find a free port for SNAT).

The fix: use multiple public IPs (each IP has its own port range); reduce the conntrack table size to limit the number of concurrent flows; identify and rate-limit the source that is opening too many flows.

Asymmetric routing

Asymmetric routing is the failure mode where the forward path goes through the router but the return path does not. The conntrack entry is created on the forward path; the return packet has no conntrack entry on its path.

The diagnostic:

1. Verify the conntrack entry — is it created?
2. Verify the return path — does the return traffic go through the router?
3. Capture on the suspected bypass interface — confirm the return traffic bypasses the router.

The fix: ensure the return path goes through the router (the host’s default gateway is the router; the upstream router does not bypass the conntrack router).

The conntrack flush procedure

The conntrack flush is the universal fix for stale entries. The operator uses it:

# Flush the entire conntrack table
sudo conntrack -F

# Flush a specific source's entries
sudo conntrack -D -s 192.168.1.100

# Flush a specific destination's entries
sudo conntrack -D -d 203.0.113.10

The flush procedure:

1. Identify the affected flow(s).
2. Flush the relevant entries (entire table or specific entries).
3. Verify the flush with sudo conntrack -L.
4. Re-test the NAT.

The discipline: every NAT change is followed by a conntrack flush.

Operational commands

The NAT troubleshooting uses the following operational commands:

# NAT rules
show nat source
show nat destination

# Conntrack
sudo conntrack -L
sudo conntrack -E
sudo conntrack -F

# Firewall
show firewall name WAN-IN
show firewall name LAN-FW
show log firewall

# nftables
sudo nft list ruleset
sudo nft monitor

# Packet capture
sudo tcpdump -ni eth0
sudo tcpdump -ni eth1
sudo tcpdump -w /tmp/capture.pcap

Rollback

The NAT diagnostic changes (e.g., trace rules) are rolled back the same way as any VyOS configuration change:

# Revert to a previous revision
rollback 1
commit
save

# Or remove the trace rule
sudo nft delete rule inet filter WAN-IN handle <handle>

# Flush conntrack after the change
sudo conntrack -F

The discipline: trace rules are temporary; the operator must remove them after diagnosis.

Production discipline

Cross-course references

• Part XXXVIII-01 (XXXVIII-VyOS-NAT / SNAT vs DNAT concept) covers the NAT primitives.
• Part XXXVIII-02 (XXXVIII-VyOS-NAT / masquerade) covers the dynamic SNAT pattern.
• Part XXXVIII-03 (XXXVIII-VyOS-NAT / port forwarding) covers the DNAT pattern.
• Part XXXVII-06 (XXXVII-VyOS-Firewall / troubleshoot) covers the firewall diagnostic method.

Quiz