Port forward troubleshooting — packet flow verification end-to-end

A port forward fails. The operator’s job is to find the failure point in the packet path. The path has five stages: ingress on WAN, rdr translation, filter evaluation, routing, and egress on LAN. Each stage is a possible failure point. The troubleshooting methodology moves from the ingress side toward the egress side, capturing at each interface, until the flow stops appearing.

This lesson covers the structured methodology, the tools at each stage, and the five most common failure points.

The methodology

The port forward troubleshooting methodology has six steps, executed in order:

1. Confirm the symptom. Is the port forward actually failing? From an external host, attempt the connection. curl -v https://198.51.100.1 shows the failure mode.
2. Capture on the WAN. tcpdump -ni igb1 host 203.0.113.99 and port 443. Does the packet arrive?
3. Check the rdr rule. pfctl -s nat | grep rdr. Is the rule present and correct?
4. Check the firewall rule. pfctl -s rules | grep pass. Is there a pass rule that matches?
5. Capture on the LAN. tcpdump -ni igb0 host 192.0.2.10 and port 443. Does the rewritten packet leave on the LAN?
6. Check state. pfctl -ss | grep 203.0.113.99. Is state being created and matched?

The methodology moves from the WAN side toward the LAN side. Each step is a checkpoint; if the flow is present, continue to the next step. If the flow is missing, the failure is between the previous step and this one.

$ tcpdump -ni igb1 -c 5 host 203.0.113.99 and port 443

tcpdump: listening on igb1, link-type EN10MB
13:42:11.842819 203.0.113.99.51820 > 198.51.100.1.443: S 1823018421:1823018421(0) win 65535
13:42:12.844281 203.0.113.99.51820 > 198.51.100.1.443: S 1823018421:1823018421(0) win 65535
13:42:14.846123 203.0.113.99.51820 > 198.51.100.1.443: S 1823018421:1823018421(0) win 65535
13:42:18.849812 203.0.113.99.51820 > 198.51.100.1.443: S 1823018421:1823018421(0) win 65535
13:42:26.853214 203.0.113.99.51820 > 198.51.100.1.443: S 1823018421:1823018421(0) win 65535

Stage 1: confirming the symptom

Before diving into packet captures, confirm that the port forward is actually failing from the client’s perspective. The operator runs:

curl -v --max-time 10 https://198.51.100.1

The output shows:

• Connection timed out. The TCP handshake did not complete.
• Connection refused. The server actively rejected the connection (a RST arrived).
• SSL handshake failed. The TCP handshake succeeded but the TLS handshake failed.

Each failure mode points to a different stage of the path. Connection timeout = the firewall is not forwarding the packet or the server is not responding. Connection refused = a RST arrived from somewhere (often the server, indicating the port is closed on the server). SSL handshake failed = TCP works, TLS fails (certificate issue, SNI mismatch, or the server speaks plain HTTP).

Stage 2: capture on the WAN

If the connection is timing out, the next step is to capture on the WAN interface. The operator runs:

tcpdump -ni igb1 -c 10 host 203.0.113.99 and port 443

The capture shows the client’s SYN packets. If the capture is empty, the client’s packets are not reaching the WAN — the failure is upstream of the firewall (routing, ISP, or the client itself is misconfigured).

If the capture shows the SYN but no SYN-ACK, the firewall is not forwarding the reply. The next step is to check the rdr rule and the firewall rule.

If the capture shows the SYN, the SYN-ACK, and then a RST or FIN, the TCP handshake succeeded; the failure is in TLS or application-layer.

$ pfctl -s nat | grep rdr

rdr on igb1 inet proto tcp from any to any port = 443 -> 192.0.2.10 port 443

Stage 3: check the rdr rule

The operator checks the rdr rule:

pfctl -s nat | grep rdr

The output shows the rdr rule. The operator verifies:

• Interface matches the WAN.
• Protocol matches the service (TCP, UDP).
• External port matches the port the client is connecting to.
• Internal IP matches the server’s address.
• Internal port matches the port the server is listening on.

If any field is wrong, the rdr rule is not rewriting the packet correctly. Fix the port forward in the GUI.

If the rdr rule is correct but the WAN capture shows no forwarded traffic, the next step is the firewall rule.

Stage 4: check the firewall rule

The operator checks the firewall rule:

pfctl -s rules | grep -E 'pass|block' | grep 443

The output shows the firewall rules that match port 443. The operator verifies:

• A pass rule exists on the WAN interface.
• The rule’s destination is the post-DNAT IP (the internal IP).
• The rule’s destination port is 443.
• The rule is above any block rule that might match the same flow.

If no pass rule exists, the implicit deny drops the packet. The fix is to enable the associated rule or write a manual one.

If a pass rule exists but the WAN capture still shows no forwarded traffic, the next step is the LAN capture.

Stage 5: capture on the LAN

The operator captures on the LAN interface:

tcpdump -ni igb0 -c 10 host 192.0.2.10 and port 443

The capture should show the rewritten packet (destination 192.0.2.10:443, source still 203.0.113.99:51234). If the capture is empty, the firewall is not forwarding the rewritten packet — the failure is between the firewall rule and the LAN (routing, or the server’s interface is wrong).

If the capture shows the rewritten packet arriving at the server, the next step is to check the server’s reply.

The operator also captures on the LAN for the return path:

tcpdump -ni igb0 -c 10 src host 192.0.2.10 and port 443

If the server is sending a SYN-ACK, the capture shows it. If the capture is empty, the server is not responding — the failure is on the server (service down, firewall on the server, wrong port).

Stage 6: check state

The operator checks PF state:

pfctl -ss | grep 203.0.113.99

If state is being created, the firewall is processing the flow. The state entry should show both views (pre- and post-DNAT).

If state is not being created, the firewall rule is not matching — the failure is at the filter stage.

The five common failure points

Through many production deployments, the same five failure points account for the majority of port forward tickets.

Failure 1: no associated rule

The rdr rule is configured but the associated firewall rule is disabled, and no manual rule exists. The rdr rule rewrites the destination, but no firewall rule permits the rewritten packet.

The diagnostic: pfctl -s rules | grep 192.0.2.10 shows no pass rule for the internal IP. The WAN capture shows the SYN arriving; the LAN capture shows nothing leaving.

The fix: enable the associated rule or write a manual one.

Failure 2: wrong internal IP or port

The rdr rule points to the wrong internal IP or port. The packet is forwarded to the wrong host, which is not listening on the expected port.

The diagnostic: the LAN capture shows the rewritten packet arriving at the internal IP. The internal host does not respond (no SYN-ACK). Verify the host’s listening port with netstat or ss.

The fix: update the rdr rule with the correct internal IP and port.

Failure 3: server’s default gateway is wrong

The server’s default gateway is not the firewall. The server receives the SYN, replies with a SYN-ACK, but the SYN-ACK leaves via the wrong gateway — never reaching the firewall.

The diagnostic: the LAN capture shows the SYN arriving at the server but no SYN-ACK leaving. netstat -rn on the server shows the default gateway is some other router.

The fix: change the server’s default gateway to the firewall’s LAN IP.

Failure 4: upstream ISP blocks the port

The WAN ISP blocks inbound traffic to port 443 (or whatever port the operator is forwarding). The SYN never reaches the firewall.

The diagnostic: the WAN capture shows nothing arriving. The ISP blocks the traffic upstream.

The fix: contact the ISP or change the port. Many ISPs block port 25 (SMTP); some block port 80 or 443 for residential connections.

Failure 5: the firewall itself does not have the public IP

The firewall’s WAN interface does not have the public IP that the client is connecting to. The SYN arrives at the WAN IP but the kernel drops it because no interface has that IP.

The diagnostic: ifconfig igb1 does not show the public IP. The operator may have configured the port forward before configuring the virtual IP, or the virtual IP is on a different interface.

The fix: add the public IP as a virtual IP on the WAN interface, then verify the port forward.

The tcpdump one-liners

Five tcpdump one-liners cover most port forward troubleshooting:

# WAN: see incoming SYNs
tcpdump -ni igb1 -c 10 'tcp[tcpflags] & tcp-syn != 0 and port 443'

# LAN: see rewritten packets to the server
tcpdump -ni igb0 -c 10 'host 192.0.2.10 and port 443'

# WAN: see SYN-ACKs leaving (the firewall forwarding the reply)
tcpdump -ni igb1 -c 10 'src host 192.0.2.10 and port 443'

# LAN: see the server's SYN-ACK
tcpdump -ni igb0 -c 10 'src host 192.0.2.10 and tcp[tcpflags] & tcp-syn != 0'

# Both interfaces at once: see the flow end-to-end
tcpdump -ni igb1 -c 5 'host 203.0.113.99 and port 443' &
tcpdump -ni igb0 -c 5 'host 192.0.2.10 and port 443'

The one-liners are quick triage tools. For deeper analysis, use Wireshark or a longer capture with timestamp correlation.

Summary

• The troubleshooting methodology moves from WAN capture, to rdr rule check, to firewall rule check, to LAN capture, to state check.
• Each step is a checkpoint; the flow stops at the first checkpoint where it is missing.
• The five common failure points are: no associated rule, wrong internal IP, server’s wrong default gateway, ISP blocking, firewall missing the public IP.
• tcpdump on both interfaces is the primary diagnostic tool; pfctl -s nat and pfctl -s rules confirm the configuration.
• The server’s default gateway is the most overlooked failure point; always verify it.