NAT anti-patterns — broad translations, ICMP breakage, MTU, and the auto-rule override trap

NAT is a frequent source of subtle production failures. The firewall works, the routing is correct, the rules are in place — and yet some flows fail, or fail intermittently, or work for some users and not others. The root cause is usually a NAT configuration mistake that breaks something the operator did not think to test: ICMP errors that never arrive, MTU black holes, or a translation that is broader than it should be.

This lesson covers the recurring NAT anti-patterns. It is the culmination of the outbound NAT sequence — the things an experienced operator looks for when reviewing a NAT ruleset before signing off on a production deployment.

The broad-translation anti-pattern

The broad-translation anti-pattern is a manual outbound NAT rule that translates a CIDR larger than the subnet it intends to cover. The classic case: an operator types 192.0.2.0/16 when they meant 192.0.2.0/24, or 10.0.0.0/8 when they meant 10.10.0.0/24.

The symptom is silent until something collides. The translated subnet now covers addresses that should not be translated: subnets from a different region, subnets that should retain their original source for compliance, subnets that are unreachable from the WAN. The remote host replies to an unexpected source IP, and the reply goes nowhere.

The fix is discipline: every outbound NAT rule has a specific source subnet, and the subnet is verified against the routing table before the rule is applied.

$ pfctl -s nat | grep 'nat on'

nat on igb1 inet from 10.0.0.0/8 to any -> (igb1:0)
nat on igb1 inet from 192.168.0.0/16 to any -> (igb1:0)

The ICMP breakage anti-pattern

ICMP error messages — destination unreachable, time exceeded, redirect — reference the IP header of the packet that caused the error. If PF translates the source of a TCP packet and the remote host later sends an ICMP error referencing the original (pre-NAT) source, PF has to match the error to the state to rewrite the ICMP payload back to the post-NAT source.

This matching is reliable for outbound traffic where PF created state on the original flow. It fails in three cases:

• ICMP errors for unsolicited packets. A UDP probe to a closed port produces an ICMP unreachable. If PF has no state for the probe (because the probe did not match a permitting rule), the ICMP error has no state to attach to and is dropped.
• ICMP errors after state timeout. The original flow timed out; PF removed the state. An ICMP error arriving after the state is gone has nothing to match and is dropped.
• ICMP errors when translation changed. If the NAT rule changed between the original flow and the ICMP error, PF cannot reverse-translate the error correctly.

The visible symptom: traceroute shows * * * after the first hop. The firewall is dropping ICMP time-exceeded messages. The operator checks the firewall logs and sees no record of the drops — ICMP errors with no state are dropped silently.

The fix is two-fold:

• Allow ICMP error messages on the WAN interface. A common pattern is a floating rule that permits ICMP errors to the firewall’s WAN IP, with state matching by payload (PF supports this for ICMP error messages).
• Use scrub to normalise packets. The scrub anchor rewrites IP header fields and helps PF match ICMP errors to state.

The MTU black-hole anti-pattern

NAT does not change packet sizes, but it interacts with MTU in subtle ways. The classic MTU black hole:

1. A LAN client sends a 1500-byte packet (full Ethernet MTU).
2. The packet traverses a tunnel (VPN, IPsec, GRE) that has a smaller MTU — say 1400 bytes.
3. The tunnel requires fragmentation. The router at the tunnel endpoint cannot fragment because the DF (do-not-fragment) bit is set, which is the default for most modern clients.
4. The router sends an ICMP “fragmentation needed” error back to the source. The error is for the original packet.
5. PF does not have state for the error because the error arrived on the WAN and the original flow left on the LAN. The error is dropped.
6. The client never learns that the path MTU is smaller. It keeps sending 1500-byte packets. They vanish.

The fix is path MTU discovery (PMTUD, RFC 1191). The client reduces its MTU based on the ICMP error, and subsequent packets fit the tunnel. For PMTUD to work, the ICMP errors must reach the client — which requires the firewall to allow ICMP errors on the WAN interface and to match them to the originating flow.

NAT also affects PMTUD indirectly: when the source is translated, the ICMP error references the post-NAT source, and the client has to use the post-NAT source for the next packet. This works because the source IP is consistent within the flow.

The anti-pattern: operators disable ICMP on the WAN interface because they read that ICMP is “dangerous”. Disabling ICMP errors also disables PMTUD, and the result is the MTU black hole. The right pattern is to allow ICMP errors to the firewall on the WAN while blocking ICMP echo to the firewall on the WAN.

The auto-rule override anti-pattern

In Hybrid mode, the operator adds manual rules on top of the auto rules. A recurring mistake: an operator reads the auto rules, decides they are wrong, and tries to “override” them by adding a manual rule that conflicts with the auto rule.

The result: PF evaluates both rules. The first match wins. If the manual rule is placed above the auto rule (the GUI default), the manual rule wins for matching flows. If the manual rule is placed below, the auto rule wins.

The anti-pattern manifests in two ways:

• Manual rule is meant to override but placed below. The operator expects the manual rule to take effect; the auto rule matches first; the manual rule never fires. The operator does not realise the placement matters.
• Manual rule is meant to be additive but conflicts. Two rules for the same source with different translation targets. Whichever is placed first wins. The other is dead code.

The fix is to switch to Manual mode when the auto rules are genuinely wrong. Hybrid is for adding rules on top; Manual is for taking control. Operators who find themselves fighting the auto rules in Hybrid mode should switch to Manual.

$ pfctl -s nat

nat on igb1 inet from 10.99.0.0/24 to any -> 198.51.100.5
nat on igb1 inet from 192.0.2.0/24 to any -> (igb1:0)
nat on igb1 inet from 10.10.0.0/24 to any -> (igb1:0)

The source-not-routed anti-pattern

An outbound NAT rule translates a source subnet to the WAN IP, but the firewall has no route for the original source subnet back to the WAN. The result: PF translates the source, sends the packet out the WAN, the remote host replies to the WAN IP, the reply arrives on the WAN, and the firewall has no state entry (because state was created against the wrong route) — or worse, the firewall has state but cannot deliver because the original source is unreachable.

This is uncommon but happens in complex topologies where the firewall has overlapping routes or where a source subnet is moved between interfaces without updating the NAT rules.

The fix is to verify the routing table and the NAT ruleset together. netstat -rn shows what the firewall will route; the NAT rules show what will be translated. They must agree on every internal subnet.

The audit checklist

A production NAT ruleset audit has seven checks:

1. Every internal subnet has an explicit translation rule. No implicit coverage from auto rules alone — verify the auto rules actually cover every subnet.
2. Every translation target is a valid WAN IP. No typos, no alias addresses that are not configured, no references to interfaces that do not exist.
3. The translation targets match the egress interfaces. A rule translating to WAN-A’s IP must use WAN-A as the egress interface. A mismatch breaks the return path.
4. Multi-WAN failover is handled deliberately. Either the NAT target follows the route, or the failover is disabled, or the upstream handles failover.
5. ICMP errors are allowed where needed. The WAN rules allow ICMP error messages; PMTUD can function.
6. The ruleset has descriptions. Every manual rule has an owner and a ticket reference.
7. The ruleset is in version control. The generated ruleset, exported after every config apply, is committed to a repository. Drift between the GUI and the running ruleset is detectable.

Recovery from a NAT misconfiguration

The recovery sequence when a NAT change breaks production traffic:

1. Recognise. tcpdump on the WAN shows the wrong source IP, or users report broken flows.
2. Roll back. Restore the previous configuration from a backup. OPNsense keeps configuration history under System → Configuration → History. Restore the last known good version.
3. Verify. pfctl -s nat and pfctl -s rules show the restored ruleset. Test traffic from each affected subnet.
4. Diagnose. Compare the failed ruleset to the restored one. Identify the rule that caused the failure.
5. Fix forward. Make the corrected change in a development environment. Test thoroughly. Apply to production.

The discipline: every NAT change is made in a way that can be rolled back. Configuration history is the safety net. Operators who cannot roll back a NAT change should not be making NAT changes in production.

Summary

• Broad translations (wrong CIDR) silently break return paths for subnets that should not be translated.
• ICMP error handling is required for traceroute and PMTUD. Disabling ICMP on the WAN disables both.
• MTU black holes occur when ICMP fragmentation-needed errors do not reach the source. PMTUD requires the firewall to allow ICMP errors to the WAN IP.
• Hybrid mode is not for overriding the auto rules; it is for adding rules on top. To replace the auto rules, switch to Manual.
• Every NAT change in production must have a tested rollback. Capture the ruleset before the change.