medium riskservice affecting~45 min

Runbook: Troubleshoot NAT

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The report is converted into a flow before any command is run: source address, destination address, protocol, port, and which end sent the first packet. "NAT is broken" is a ticket; 198.51.100.5 to 203.0.113.10 tcp/443, client-initiated inbound is something you can run commands against, and doing that conversion is the first five minutes of this job.
· The direction is established, because the three NAT shapes have nothing in common diagnostically: outbound source NAT (LAN hosts reaching the Internet), inbound destination NAT (an Internet client reaching a published service), or hairpin (a LAN host reaching the public address of a service that is also on the LAN).
· Whether this ever worked is established, and if so when it stopped. A service that has never been reachable is a missing rule or a missing firewall permit; one that stopped is a change, a WAN address change, or a stale translation, and those have different first commands.
· The scope is established: one service, one source host, or everything behind the router. Everything at once points at the source-NAT rule, the WAN address or the conntrack table itself — not at the service that was reported.
· The change record for the last 72 hours is checked on this router and on the host being translated to, covering NAT rules, firewall rules, interface addressing, and any WAN address change from DHCP or PPPoE.
· Pre-diagnosis state is captured off-box and attached to the ticket: show configuration commands | match nat, show nat source rules, show nat destination rules, show firewall, and show conntrack statistics.
· Out-of-band console access is confirmed before any NAT or firewall change is contemplated. A source-NAT or firewall edit on the interface carrying the operator session removes the operator, and a NAT runbook turns into a firewall change more often than anyone plans for.
· It is agreed before the window that conntrack -F is not in the plan. Flushing the whole table to fix one port forward drops every translation the router is holding, breaks every long-lived session behind it, and is not a diagnostic step. The targeted conntrack -D filter for this service is written down in advance instead.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1State the flow as a 5-tuple with a direction, and name which rule is supposed to translate it. If nobody can say which rule should fire, the most likely answer is that none does, and time spent reading conntrack is spent in the wrong place.
2Decide which of the three shapes you are in — outbound SNAT, inbound DNAT, or hairpin. The hairpin case is the one most often misfiled: it is reported as "the port forward is broken" by a user who is sitting on the same LAN as the server, while every Internet client is fine.
3Read the rules and their counters before reading anything else: show nat source rules and show nat destination rules for what is configured, show nat source statistics and show nat destination statistics for whether the rule is being hit. A rule with a zero counter is not matching the traffic; a rule with a climbing counter is matching, and the fault is downstream of it.
4Read the conntrack entry for the specific flow with sudo conntrack -L narrowed by source or destination. The entry carries two tuples — the packet as the client sent it, and the reply the router expects — and the second one is where the translation is visible.
5Decide from the entry which link broke. No entry at all means the packet never reached conntrack or was dropped before it. An entry with no translation in the reply tuple means the rule did not fire. An entry with the wrong translation means the wrong rule fired. An entry that is correct means the fault is the firewall, the routing, or the far end.
6Check the firewall against the address the firewall actually sees, which is not the same on both directions: for inbound traffic DNAT has already run, so the rule must name the private address; for outbound traffic SNAT has not run yet, so the rule must name the private address too — for the opposite reason. Use the per-rule counters in show firewall ipv4 forward filter as the evidence rather than reading the rule and believing it.
7Prove bytes moved, on both interfaces, with monitor traffic interface eth0 and monitor traffic interface eth1 filtered to the flow. Everything above this step proves intent; only the capture proves the packet arrived and left translated.
8For a hairpin failure, look at the reply tuple: if it shows the server answering directly to the LAN client rather than to the router, the source NAT for LAN-originated traffic is missing and the client is discarding a handshake that arrives from an address it never contacted.
9For a source-NAT failure affecting many hosts at once, check for port exhaustion before checking anything else: compare nf_conntrack_count against nf_conntrack_max, and look for nf_conntrack: nf_nat: no unique tuple found in the kernel log. A single translation address carries roughly 28,000 concurrent flows, and the symptom is legitimate traffic being dropped with every rule correct.
10For an inbound flow whose conntrack entry never reaches an established state, prove the return path rather than assuming it. If the reply leaves the server by a route that does not traverse this router, conntrack never sees it and no NAT configuration on this router can fix it.
11Apply any fix inside configure, with compare read before the commit and commit-confirm 10 used for anything touching the operator path or a rule set that carries production traffic. Confirm only after the verification below has passed.
12Remove only the translations the change invalidated, using a conntrack -D filter that was first run as conntrack -L so its blast radius is visible before it is applied. Never the whole table.
13Re-verify from both sides — an Internet client and a LAN client — and name in the ticket which link of the chain had broken, because that is the part of the incident the next person can use.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The rule that should translate the flow shows a climbing counter in show nat source statistics or show nat destination statistics, not a zero.
✓The conntrack entry for the flow exists, is in an established state, and its reply tuple carries the translation the design intends — the public address for an outbound flow, the private address for an inbound one.
✓For a hairpin flow, the reply tuple names the router LAN address as the destination rather than the LAN client, which is the signature that both translations are being applied.
✓The firewall rule that admits the flow has a non-zero per-rule counter in show firewall ipv4 forward filter, and the default-action counter is not the one absorbing this traffic.
✓A capture on both interfaces shows the packet arriving untranslated on one and leaving translated on the other, in both directions.
✓One real client transaction succeeds from the Internet, and one from a host on the LAN using the public address, because those two paths fail independently and only one of them is usually tested.
✓The conntrack count is well clear of the table maximum, and the kernel log carries no no unique tuple found entries since the change.
✓Any conntrack deletion performed was the targeted form, its filter is recorded in the ticket, and the sessions it removed are named rather than discovered by their owners.
✓The configuration is saved only after the above passes, and the diagnosis in the ticket names the link that broke: rule, translation, firewall, routing, or state.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Reading is free: every step up to the fix in this runbook is read-only, and nothing before the first commit needs rolling back.
↶A fix applied with commit-confirm 10 and found to be wrong needs no action at all — do nothing and the router reverts on its own. That is the cheapest rollback available and the reason the confirm window exists.
↶A committed and confirmed fix is reverted by deleting the specific nodes that were added, with compare read before the commit. Prefer that to a whole-configuration rollback 1, which also discards anything else committed in the same window.
↶Reverting a NAT rule leaves behind the translations it created. Remove them with the same targeted conntrack -D filter used when the rule went in, listed with conntrack -L first, so the revert does not leave flows being translated by a rule that no longer exists.
↶A firewall rule added during the investigation to test a hypothesis is removed in the same session it was added in, and its removal is verified with the per-rule counters rather than assumed.
↶If the operator session is lost mid-change, do not re-commit blind from a second path. Get to the console, read compare and show configuration commands, and establish what the router is actually running before touching it again.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the service owner when the conntrack entry is correct, the firewall counters show the packet admitted, and the capture shows the translated packet leaving toward the server. At that point the router has done its job and the fault is on the host.
· Escalate to the upstream provider when the outbound packet is correctly translated and leaves on the WAN interface, and no reply arrives in a capture on that interface. No configuration on this router can create a reply that is not coming back.
· Escalate to network engineering when the evidence points at asymmetric routing — the forward path traverses this router and the return path does not. That is a topology fault, it will be breaking every stateful device in the path, and it cannot be fixed from one end.
· Escalate to capacity or platform ownership on port exhaustion. Adding translation addresses, or rate-limiting the source that is opening tens of thousands of flows, is a design change with an owner; it is not something to improvise during a page.
· Escalate to security when the source opening enough flows to exhaust the translation address cannot be accounted for, and when a published service is receiving traffic the change record cannot explain.
· Hold rather than improvise when the only fix available is one you cannot bound: a permissive firewall rule to "see if it is the firewall", or a full conntrack flush to clear a suspicion. Record the hold with an owner, an end time, and a plain statement of what stays broken while it holds — that is the information the person deciding actually needs.

“NAT is broken” is a claim about a chain, and the chain has five links. Each one fails differently, each one has a command that reads it, and the whole value of this runbook is refusing to skip to the interesting link.

Link	Command that reads it	What a break here looks like
The rule exists and matches	`show nat source rules`, `show nat destination rules`	Rule present, counter at zero
The translation happened	`sudo conntrack -L`	No entry, or a reply tuple with the untranslated address
The firewall permitted it	`show firewall ipv4 forward filter`	Permit rule at zero, default-action counter climbing
The routing sent it somewhere sensible	`show ip route`, a capture on each interface	Translated packet leaves by the wrong interface, or never leaves
The state survived the change	`sudo conntrack -L` for the specific flow	Existing flows broken, new flows fine — or the reverse

The conntrack table is the middle of that chain and the source of truth for the two links either side of it. It is also the only place where the translation is visible as a fact rather than as an intention.

When this runbook applies

A published service is unreachable from the Internet.
LAN hosts cannot reach the Internet, or some of them cannot.
A LAN host cannot reach a service by its public address while Internet clients can.
A NAT or firewall change was committed and some flows work while others do not.
Traffic was working and stopped at a WAN address change.

When it does not

The route to the destination is missing. NAT rewrites addresses; it does not create reachability. Prove the route first — that is a different runbook and a shorter one.
The service on the target host is down. A translated packet delivered to a closed port produces a refusal that looks a great deal like a firewall drop from the client’s side. Establish that the host is listening before spending an hour on the router.
The firewall is dropping traffic that was never NAT-ed. The symptom overlaps and the evidence does not. If the flow crosses no NAT rule, the blocked-traffic runbook is the right one.

Blast radius

Every step up to the fix is read-only, which makes the sequencing free: there is no reason to skip a step to save time. The risk in a NAT incident is concentrated in two actions, and both of them look small.

`conntrack -F` is not a diagnostic step

It empties the entire connection tracking table. Every translation the router was holding goes with it, so every source-NAT-ed session loses the mapping its packets depend on and breaks mid-stream. Every remaining flow also loses the state that a state established rule was matching on and has to be re-evaluated from nothing. SSH sessions, VPN control channels and database connections are the ones that notice, because they are the ones that are long-lived.

The blast radius is every user behind the router, for one command typed to fix one port forward.

Delete only what changed. conntrack -D takes the same filters as conntrack -L, so the list form is a dry run of the delete:

Read-only / Safelist first — this is the dry run

sudo conntrack -L -p tcp -d 203.0.113.10 --dport 443

Service impact possiblethen delete exactly what the list showed

sudo conntrack -D -p tcp -d 203.0.113.10 --dport 443

Even the targeted form is disruptive to the sessions it matches, so it belongs in the change window with the rest of the work. What it must never be is a habit applied after every commit.

The second action is a permissive firewall rule added “just to see whether it is the firewall”. The per-rule counters answer that question without opening anything, and a rule added under pressure is the one that is still there a year later.

Step 0: turn the report into a flow

A NAT lookup needs a source, a destination, a protocol, a port and a direction. “The website is down” gives you none of them. Establish all five, and then establish which rule is supposed to translate this flow.

That last question is not bureaucracy. If nobody can name the rule, the most likely explanation is that no rule exists for this traffic, and every minute spent reading conntrack is a minute spent in the wrong subsystem.

Step 1: which of the three shapes is this?

Shape	Who is affected	First thing to read
Outbound source NAT	LAN hosts reaching the Internet	`show nat source rules`, and whether the WAN address changed
Inbound destination NAT	Internet clients reaching a published service	`show nat destination rules`, then the forward-chain counters
Hairpin	LAN hosts reaching a LAN service by its public address	The reply tuple in conntrack

The hairpin case is the one most often misfiled. It arrives as “the port forward is broken”, reported by someone sitting on the same LAN as the server, while every Internet client is working perfectly. Asking “where is the person who reported this?” resolves a surprising number of these before any command is run.

Step 2: is the rule there, and is it firing?

Read-only / Safeconfiguration, then the counters

show nat source rules
show nat destination rules
show nat source statistics
show nat destination statistics
show nat source translations
show nat destination translations

A rule that is present with a zero counter is not matching the traffic you care about. That is a different fault from a rule that is matching and translating to the wrong place, and the counters separate the two in one command.

The two match conditions that most often make a rule silently not fire are the interface and the direction. On 1.4 and 1.5 the interface matcher takes a name or a group beneath it, so a source rule reads outbound-interface name eth0 and a destination rule reads inbound-interface name eth0. In a source rule outbound-interface is the interface the packet leaves by; in a destination rule inbound-interface is the interface it arrived on. Naming the LAN interface where the WAN belongs produces a rule that describes the opposite flow, commits cleanly, and never matches anything.

A masquerade rule with no translation address translates nothing

translation address is required, and the keyword masquerade is what makes it dynamic — the kernel substitutes the current primary address of the outbound interface at translation time, which is what lets the rule survive a DHCP lease change.

Leave the clause out and the rule still commits. It matches traffic and translates nothing, so packets leave with their private source addresses intact and nothing comes back.

The failure is silent from the router’s own point of view, because the router’s own traffic is not subject to the rule — a ping from the CLI still works while every host behind the router is dark. Two commands separate this from a routing fault: show nat source rules shows the rule with no translation, and the reply tuple in conntrack shows the private address where the public one should be.

Step 3: read the conntrack entry

Read-only / Safethe flow, not the whole table

# Substitute your own values before running:
LAN_HOST=192.168.1.100
PUBLIC_IP=203.0.113.10

sudo conntrack -L -s "$LAN_HOST"
sudo conntrack -L -d "$PUBLIC_IP"

# Watch a new flow being created, while a client retries
sudo conntrack -E

An entry carries two tuples. The first is the packet as the sender sent it. The second is the reply the router expects to receive, and it is the second one that shows whether a translation was applied — because the reply is addressed to the translated address, not the original.

What you see	State	Where to go
No entry for the flow at all	The packet never reached conntrack	Capture on the ingress interface; the packet is not arriving, or is dropped before NAT
Entry present, reply tuple carries the untranslated address	The rule did not fire	Step 2: the match conditions or the missing translation address
Entry present, reply tuple carries the wrong address	A different rule fired first	Rule ordering; read the whole rule set, not the rule you expected
Entry present and correct, no established state	The reply is not coming back	Step 5: firewall, return path, or the host
Entry present, correct and established, traffic still fails	The fault is past the router	The host, or the client’s own path

sudo conntrack -E is the underused one. It prints the events as they happen, so a client retrying while you watch tells you whether an entry is being created at all — which is a far cleaner signal than searching a table with tens of thousands of rows for a flow that may never have existed.

Why the reply tuple is the only honest evidence of a translation

Conntrack records both directions because the kernel has to reverse the translation on the way back. Without the entry, a reply arriving addressed to the public address would have nothing to tell the kernel that the original destination was a private host.

That is why the reply tuple, and not the rule, is where a translation becomes a fact. The rule states an intention. The reply tuple states what the kernel committed to for this flow, at the moment the flow started — and it does not change afterwards.

Two consequences follow directly, and both are things operators meet.

A NAT rule change does not affect flows that already exist. The entries created under the old rule keep the old translation until they expire, so immediately after a commit the router is running two policies at once: the new one for new flows, the old one for everything already open. This is what “stale entry” actually means, and it is why the targeted conntrack -D is part of a NAT change rather than an afterthought.

A WAN address change breaks existing flows for the same reason. Masquerade picks the interface address at translation time, so entries created before the change still expect replies at the old address. New flows work; the old ones are silently dead. The operator sees a partial outage — new connections succeeding while established sessions hang — which reads like packet loss and is not.

Step 4: the firewall, against the address it actually sees

The filter hooks sit between the two NAT hooks. Destination NAT has already run by the time the forward chain evaluates an inbound packet; source NAT has not yet run when it evaluates an outbound one. The rule of thumb is easy to state and easy to misexplain: you always write the private address, for opposite reasons.

Direction	What the firewall sees	Why
Inbound to a published service	Destination is the private address	DNAT already ran in prerouting
Outbound from a LAN host	Source is the private address	SNAT has not run yet; postrouting is after the filter

An operator who internalises only “the firewall sees the post-NAT address” writes the outbound rule against the public address and is then surprised.

Read-only / Safecounters are the evidence; the rule text is not

show firewall ipv4 forward filter
show firewall ipv4 input filter
show firewall statistics
show firewall

An inverted or mis-addressed rule is hard to spot by reading and trivial to spot by counting: the permit rule sits at zero while the default-action counter climbs. That single observation distinguishes “the rule is missing” from “the rule is present and does not match”, which are the two firewall faults a NAT incident produces, and they have different fixes.

Step 5: prove that bytes moved

Everything above this proves intent. Only a capture proves the packet arrived, and that it left translated.

Read-only / Safeboth interfaces, because a translation spans two

monitor traffic interface eth0 filter 'port 443'
monitor traffic interface eth1 filter 'port 443'

For an inbound published service, the packet on the WAN interface carries the public destination and the packet on the LAN interface carries the private one. For an outbound flow it is the source address that differs between the two captures. If the pre-NAT packet arrives and the post-NAT packet never appears, either the rule did not fire or the firewall dropped the translated packet — and step 4 has already told you which.

If nothing arrives on the WAN interface at all for an inbound service, stop. The router is not the problem, and no amount of NAT configuration will make a packet exist.

Step 6: the four failure modes worth recognising on sight

Hairpin. A LAN client connects to the public address. Destination NAT rewrites the destination to the LAN server, the router forwards it across the same segment, and the server replies directly to the client — because they are neighbours. The client receives a handshake from an address it never contacted and discards it. The signature is in the reply tuple: it names the LAN client where it should name the router’s LAN address. The fix is a source NAT rule for LAN-originated traffic to that service, and it is the second half of a port-forward deployment rather than an optional extra.

Port exhaustion. Source NAT behind a single translation address is bounded by the ephemeral port range, roughly 28,000 concurrent flows for one address. Past that the kernel cannot find a free tuple, and legitimate traffic is dropped with every rule correct.

Read-only / Safetable pressure, then the kernel's own complaint

cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

# The kernel says so plainly when it cannot allocate a tuple
sudo dmesg | grep -i 'no unique tuple'

The remedies are all design changes: more translation addresses, or finding and rate-limiting the source that is opening tens of thousands of flows. None of them is something to invent at three in the morning, which is why this appears in the escalation list rather than the procedure.

Asymmetric routing. The forward path traverses this router and the return path does not. Conntrack never sees the reply, so no entry ever reaches an established state and the stateful rule set drops the traffic — correctly. This is not a NAT fault wearing a NAT costume; it is a topology fault, and it will be breaking every stateful device in the path, not just this one.

A stale translation after a change. New flows use the new rule and existing flows use the old one, so the symptom is “it works for some people”. The targeted conntrack -D for the changed service is the fix, and it belongs in the change plan with its blast radius written down.

What each step costs if you get it wrong

Action	Cost when it is the wrong move
Reading the rule instead of its counter	Concluding a rule fires because it looks correct
Skipping conntrack and going to the firewall	Rewriting firewall rules while the translation never happened
Writing the outbound firewall rule against the public address	A rule that never matches, and a silent drop that reads like a NAT fault
Flushing the whole conntrack table	Every session behind the router breaks, to fix one port forward
Adding a permissive firewall rule to test a theory	A hole that outlives the incident and the person who opened it
Declaring success from one client	The hairpin path stays broken, and it is the path your own staff use
Treating port exhaustion as a rule problem	Hours spent on correct configuration while the real limit is arithmetic

Escalation, and the option to hold

Escalate as soon as the evidence crosses a boundary you do not own, and send the evidence rather than the conclusion. A capture on the WAN interface showing the translated packet leaving and no reply arriving is complete proof that the fault is upstream; a conntrack entry that is correct and established, with the firewall counters showing the packet admitted, is complete proof that the fault is on the host.

Hold rather than improvise when the only available action is one whose blast radius you cannot state. “Try flushing conntrack” and “open the firewall and see” are both available, both often work, and both convert a bounded incident into an unbounded one. Record the hold with an owner and an end time, and be specific about what stays broken while it holds.

Why this matters

NAT diagnosis rewards one habit above all others: reading the reply tuple. The rule tells you what someone intended. The reply tuple tells you what the kernel actually committed to for this flow, and the gap between those two is where nearly every NAT incident lives — a rule that is correct and never matched, a rule that matched and was then dropped by a firewall reading a different address, a translation that was correct yesterday and is still being applied from an entry created before the change.

The second habit is knowing that a translation spans two interfaces, and therefore that half the evidence is on the interface you did not capture. An operator watching only the WAN side of a port forward can see the packet arrive and cannot see whether it left, which is exactly the question.

And the flush is always available. It always seems to help, because it clears the symptom along with everything else the router was holding. An estate can be read by the outages that follow it: one command, typed to fix one service, that dropped every session behind the router while nobody connected the two.