NAT troubleshooting — finding the translation that did not happen

NAT failures are trickier than rule failures because they have two layers: the translation rule and the firewall rule that matches the translated packet. The packet arrives with its original source and destination; PF applies the NAT translation; the post-translation packet is evaluated against the filter ruleset. If the NAT rule did not match, no translation happens and the packet hits the filter ruleset with its original addresses. If the NAT rule matched but the post-translation filter rule did not, the packet is dropped after translation.

This lesson covers the structured methodology for NAT failures, the evidence each layer produces, and the most common NAT mistakes in production estates.

The methodology

The NAT troubleshooting methodology has six steps, executed in order:

1. Confirm the symptom with capture. tcpdump on both sides of the translation (LAN and WAN for outbound; WAN and LAN for port-forwards).
2. Determine the NAT direction. Outbound (LAN-to-WAN) or inbound (WAN-to-LAN port-forward).
3. Check the NAT ruleset. pfctl -s nat for the rule that should match.
4. Check the translated packet’s filter rule. For port-forwards, the filter rule uses the post-DNAT destination. For outbound NAT, the filter rule uses the translated source.
5. Check state. pfctl -s state | grep for the flow with the translated addresses.
6. Form the cause hypothesis. One sentence. “Port-forward rule is present but the post-DNAT destination does not match any filter rule.”

The methodology moves from the wire to the NAT ruleset to the filter ruleset. Each step is a checkpoint; if the flow is present at one step, continue. If absent, the failure is between the previous step and this one.

Step 1: confirm the symptom with capture

The first step is to confirm the packet is on the wire on both sides of the translation. For an outbound NAT failure, capture on LAN and WAN. For an inbound (port-forward) failure, capture on WAN and LAN.

$ tcpdump -nei igb0 -c 5 host 192.0.2.50 and port 443

tcpdump: listening on igb0, link-type EN10MB
14:22:11.123456 aa:bb:cc:11:22:33 > 66:77:88:99:aa:bb, IPv4, length 74: 192.0.2.50.51820 > 203.0.113.50.443: S 1823018421:1823018421(0) win 65535
14:22:11.124567 66:77:88:99:aa:bb > aa:bb:cc:11:22:33, IPv4, length 74: 203.0.113.50.443 > 192.0.2.50.51820: S 3748291000:3748291000(0) ack 1823018422 win 65535

Step 2: determine the NAT direction

The direction determines the rest of the investigation:

• Outbound NAT. LAN-to-WAN. The packet’s source is the LAN host; the translation changes the source to the WAN IP. Failure modes: missing outbound NAT rule, asymmetric NAT (different WAN IPs for inbound and outbound), NAT pool exhaustion.
• Inbound NAT (port-forward). WAN-to-LAN. The packet’s destination is the WAN IP; the translation changes the destination to the internal host. Failure modes: missing rdr rule, post-DNAT filter rule missing, internal host unreachable, port collision.

For outbound NAT, the investigation centres on pfctl -s nat | grep nat on and the outbound NAT rule list. For inbound NAT, the investigation centres on pfctl -s nat | grep rdr and the associated filter rule.

Step 3: read the NAT ruleset

pfctl -s nat shows the compiled NAT ruleset.

rdr on igb1 inet proto tcp from any to 198.51.100.1 port = 443 -> 192.0.2.10 port 443
nat on igb1 inet from 192.0.2.0/24 to any -> (igb1) round-robin

Reading the ruleset:

• RDR rule present, source/destination matches. The translation should happen for port-forwards. If the packet does not arrive at the internal host, the rule may not be matching — check the rule’s source and destination.
• RDR rule present, but the rule is for a different destination IP. The translation applies to a different WAN IP. The packet goes to the WAN IP but the rule is on a different IP. Fix: ensure the rule’s destination matches the actual WAN IP.
• Outbound NAT rule present, source matches the LAN subnet. Translation should happen on egress. Confirm the rule’s source and interface.
• Outbound NAT rule present, but the rule is manual with a different translation target. The translation may use a specific IP or pool, not the interface IP. Confirm the translation target.
• No matching NAT rule. The translation is missing. Add it.

The NAT ruleset is the authoritative source for translations. The GUI’s NAT page may show rules in a different order than the compiled ruleset.

Step 4: check the post-translation filter rule

For inbound NAT (port-forwards), the post-DNAT destination is what the filter rule matches. The rule that allows the traffic references the LAN IP, not the WAN IP.

@75 pass in quick on igb1 inet proto tcp from any to 192.0.2.10 port = 443

The rule references 192.0.2.10 (the LAN host), not 198.51.100.1 (the WAN IP). The operator looking for 198.51.100.1 in the filter ruleset will not find the rule.

For outbound NAT, the filter rule on the egress interface matches the translated source. The rule on the WAN side typically uses any for the source (because the translated source is the WAN IP, and the rule is meant to allow all egress). The filter rule is usually not the cause of an outbound NAT failure.

Step 5: check state

pfctl -s state shows the state for the translated flow.

all tcp 198.51.100.1:443 <- 203.0.113.50:51820       ESTABLISHED:ESTABLISHED
all tcp 198.51.100.1:443 -> 192.0.2.10:443           ESTABLISHED:ESTABLISHED

Reading the state:

• State present with WAN IP and LAN IP. Translation worked, state created. The flow is established.
• State present with WAN IP only. Half-state; the translation was applied to one direction only. Investigate the NAT rule’s symmetric flag.
• No state for the translated flow. Translation did not happen or the filter rule did not match. Investigate the NAT ruleset and the filter ruleset.

For outbound NAT, the state shows the LAN IP and the WAN IP:

all tcp 192.0.2.50:51820 -> 198.51.100.1:443       ESTABLISHED:ESTABLISHED
all tcp 198.51.100.1:443 <- 192.0.2.50:51820       ESTABLISHED:ESTABLISHED

The state always shows the post-translation addresses.

Step 6: form the cause hypothesis

With capture, NAT ruleset, filter ruleset, and state in hand, the operator forms a one-sentence hypothesis. The hypothesis names the NAT rule, the expected match, and the actual match.

Examples:

• “Port-forward RDR rule for 198.51.100.1:443 to 192.0.2.10:443 is present. Post-DNAT filter rule on igb1 for 192.0.2.10:443 is missing. The flow is dropped at the filter step after translation.” (Missing filter rule.)
• “Outbound NAT rule for 192.0.2.0/24 to any uses manual mode with translation target 198.51.100.5, but the WAN interface is 198.51.100.1. The packet is being translated to a non-existent IP and dropped upstream.” (Wrong translation target.)
• “Port-forward RDR rule is present but the destination port on the external side is 8443, not 443. The client connects to 443; the rule does not match.” (Port mismatch.)

The hypothesis is testable. The test is the evidence that supports or refutes it.

The most common NAT mistakes

Six mistakes account for most NAT failures in production estates:

1. Missing outbound NAT rule. The interface has automatic outbound NAT disabled (set to manual) but no manual rule covers the source. The packet leaves without translation and is dropped by the upstream provider’s uRPF check.
2. Missing port-forward RDR rule. The operator set up the firewall rule for the LAN host but forgot the RDR rule on the WAN. The packet arrives on the WAN but is not translated.
3. Asymmetric outbound NAT. Multiple WAN interfaces with separate outbound NAT pools. A flow goes out WAN-A but returns on WAN-B. The return is dropped because no state exists on WAN-B.
4. NAT reflection loop. A LAN client tries to reach the LAN host via the WAN IP. NAT reflection translates the destination back to the LAN IP, but the source remains the LAN IP. The packet goes out and back to the LAN host; the LAN host sees a connection from itself, which most applications reject.
5. 1:1 NAT address pool mismatch. A 1:1 NAT rule maps a public IP to a LAN host, but the public IP is not configured as a virtual IP on the WAN interface. The packet arrives on the WAN, the rdr matches, but the firewall does not own the destination IP and drops the packet.
6. Port collision. Two RDR rules use the same WAN IP and external port for different LAN hosts. The first rule wins; the second never matches.

UnderTheHood: how PF applies NAT and filter

PF applies NAT and filter in a specific order. The operator who knows the order diagnoses NAT failures faster than the operator who reads the rulesets independently.

$ pfctl -s nat -s rules | head -30

nat on igb1 inet from 192.0.2.0/24 to any -> (igb1) round-robin
rdr on igb1 inet proto tcp from any to 198.51.100.1 port = 443 -> 192.0.2.10 port 443
pass in quick on igb1 inet proto tcp from any to 192.0.2.10 port = 443
pass in quick on igb0 inet all
pass out all

PF evaluation for an inbound packet:

1. PF receives the packet on igb1 with destination 198.51.100.1:443.
2. PF evaluates the NAT ruleset. The RDR rule matches; PF translates the destination to 192.0.2.10:443.
3. PF evaluates the filter ruleset against the translated packet. The filter rule that matches the translated destination (192.0.2.10:443) is consulted.
4. If the filter rule is pass, state is created and the packet is forwarded to 192.0.2.10.
5. If no filter rule matches, the packet is dropped.

The NAT ruleset and the filter ruleset are evaluated sequentially. A NAT rule that matches but a filter rule that does not match produces a “translated but dropped” failure.

Summary

• NAT failures have two layers: the NAT rule and the post-NAT filter rule. Both must be checked.
• Confirm the symptom with capture on both sides of the translation. Confirm the NAT direction. Read the NAT ruleset. Read the post-NAT filter rule. Check state.
• Six mistakes account for most NAT failures: missing outbound NAT, missing RDR, asymmetric NAT, NAT reflection loops, 1:1 NAT pool mismatch, port collision.
• NAT reflection is a workaround for clients that cannot use split-horizon DNS. The correct fix is split-horizon DNS.
• PF evaluates NAT rules before filter rules. The filter ruleset is written in terms of post-translation addresses.