high riskservice affecting~45 min

Runbook: Configure NAT

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The direction is settled in one sentence before any command is typed: traffic leaving the estate is nat source (postrouting); traffic arriving from outside is nat destination (prerouting). A rule written in the wrong direction commits cleanly and fails silently.
· The existing rules are read and their numbers noted with show nat source rules and show nat destination rules. NAT is first-match like the firewall, so a broad rule at a low number shadows anything specific placed below it.
· For SNAT: the translation address is either masquerade (dynamic WAN — DHCP, PPPoE) or a literal address that is actually assigned to this router. A literal address the router does not hold is a black hole for the return traffic.
· For DNAT: the return path is established. The translated host must send its replies back through this router — normally by having it as the default gateway — or the reply bypasses conntrack and the translation is never reversed.
· The matching firewall permit is planned at the same time as the translation, against the POST-translation address, in the correct interface direction. DNAT runs before the filter chains, so a rule naming the public address never matches.
· Whether LAN clients need to reach the public address is decided now. If they do, hairpin SNAT is a second rule and a second test, not an afterthought.
· The exact conntrack -D filter for this change is written into the ticket with its blast radius stated, and conntrack -L has been run with the same filter so the operator can see what the delete would remove. conntrack -F is not in the plan.
· The pre-change configuration is on disk: save /config/pre-change-TICKET.conf, plus show configuration commands | match nat captured to the ticket.
· Current conntrack pressure is known for a SNAT change: compare /proc/sys/net/netfilter/nf_conntrack_count against nf_conntrack_max. A single translation address supports roughly 28,000 concurrent flows before ports run out.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Capture the pre-change baseline: show nat source rules, show nat destination rules, show firewall ipv4 forward filter, and a conntrack -L run with the filter you intend to delete with later. Then save /config/pre-change-TICKET.conf.
2Choose the rule number deliberately against the rules already present, leaving a gap of 10. Confirm nothing above it already matches this traffic.
3Enter configure and write the translation rule — nat source with outbound-interface name, or nat destination with inbound-interface name.
4Write the firewall permit in the same candidate, against the post-translation address and port, with inbound-interface name set to the interface the packet ARRIVED on and outbound-interface name the interface it LEAVES by.
5Add the hairpin SNAT rule if LAN clients must reach the public address.
6Run compare and check three things specifically: the direction, the rule numbers relative to existing rules, and that the firewall rule names the private address rather than the public one.
7Apply with commit-confirm 10, not a bare commit.
8Verify on new flows first, before touching conntrack at all — a brand-new connection from the real client exercises the new rule with no stale state involved.
9Read the counters: the NAT rule counter and the firewall permit counter must both be non-zero, and the forward chain default-action must not be climbing.
10Inspect the conntrack entry for the test flow and confirm the reply tuple carries the translation you configured.
11Only if existing flows must move to the new behaviour, run the targeted conntrack -D from the ticket — the same filter you listed with conntrack -L first. Never conntrack -F.
12Test from every direction that matters: the real external client, and a LAN client via the public address if hairpin was configured.
13Type confirm, then save, then diff show configuration commands | match nat against the pre-change capture.

4 · Verification

Confirm the procedure actually fixed the problem.

✓show nat destination rules (or show nat source rules) shows a non-zero packet counter on the new rule after a real client has connected. A zero counter means the rule is not being reached — wrong direction, wrong interface, or shadowed by a lower-numbered rule.
✓show firewall ipv4 forward filter shows the paired permit rule with a non-zero counter, AND the chain default-action counter not increasing for this flow. Both together are what distinguishes a working permit from an inverted one.
✓show conntrack table ipv4 contains an entry for the test flow whose reply tuple carries the configured translation. This is the authoritative proof that the translation happened, rather than that a rule exists.
✓For DNAT: a connection from a real external client succeeds end to end, not merely a packet arriving at the translated host.
✓For SNAT or masquerade: an internal host reaches an external service and the external side observes the intended source address.
✓For hairpin: a LAN client reaches the service via the PUBLIC address and completes the handshake, not just receives a SYN/ACK.
✓A capture on both interfaces shows the pre-translation packet on one side and the post-translation packet on the other — monitor traffic interface eth0 and monitor traffic interface eth1.
✓Flows that were NOT the subject of this change are still working, confirming no conntrack entries were cleared beyond the documented filter.
✓compare saved is empty after save.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Not yet committed: discard. No translation reached the kernel and no conntrack entry was created under the new rule.
↶Committed under commit-confirm N and the change is wrong: do nothing and let the window expire. The router reverts to the previous configuration by itself.
↶Committed and confirmed: load /config/pre-change-TICKET.conf then commit, or remove the specific rules with delete nat destination rule N / delete nat source rule N followed by commit.
↶The rollback has the same conntrack problem as the change did, in reverse: flows created while the new rule was live still carry the new translation in their conntrack entries and will keep using it. Removing the rule does not remove them.
↶So a rollback is two actions, not one: revert the configuration, then run the SAME targeted conntrack -D filter from the ticket. List it with conntrack -L first, exactly as during the change.
↶Do NOT use rollback N. The VyOS documentation states it applies the revision and triggers a system reboot — which clears every conntrack entry on the router by rebooting it, breaking every session on the box to undo one NAT rule.
↶Re-verify after rollback on the same evidence used for the change: rule counters back to their previous behaviour, and a real client flow following the old path.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the application or host owner when the NAT rule counter and the firewall permit counter are both non-zero, the conntrack entry shows the correct translation, and the service still does not answer. The router has done its job and the remaining problem is on the destination host.
· Escalate to network engineering when the return path does not traverse this router — asymmetric routing defeats conntrack entirely, so the translation is never reversed, and no NAT rule on this box can fix it.
· Escalate to capacity or network engineering on evidence of port exhaustion: nf_conntrack_count approaching nf_conntrack_max, or kernel log entries about no unique tuple being found. The fixes are more translation addresses or fewer flows, both of which are decisions above this runbook.
· Escalate to the security owner when a port forward or 1:1 NAT exposes a host that was previously unreachable from outside. A translation is an exposure decision as much as a routing one.
· Hold rather than proceed if conntrack -F is the only clearing plan anyone can offer, or if the blast radius of the flow deletion has not been established. Record the hold with an owner and an end time, and hand the window back — a NAT change applied without a flow-clearing plan is half-applied by design.

NAT looks like one feature and is three subsystems wearing a trench coat. Almost every NAT change that goes wrong goes wrong at one of the seams between them.

The translation — the nat table. This is what you configure.
The permission — the filter chains. This is what makes the translation usable, and it is a separate rule you have to remember to write.
Connection tracking — which decides when your change takes effect, and holds the only mechanism that can reverse a translation on the way back.

The runbook is built around the three facts that follow from that: the firewall sees the translated address, a NAT rule only affects new flows, and clearing old flows is a destructive act with a blast radius you have to choose deliberately.

When this runbook applies

Use it to add or change a translation on a router that is already carrying traffic: a new port forward, a source NAT for a new subnet, a 1:1 mapping, a masquerade rule for a second WAN, or a change to any of those.

Do not use it for:

A port forward that has stopped working. Nothing about the configuration changed, so re-entering it proves nothing. Diagnose first.
The initial NAT configuration of a router before it carries traffic. The conntrack scaffolding here exists because there are live flows to protect. There aren’t any yet.
A change whose real subject is the firewall policy. If the translation is already correct and the question is who may reach the translated host, that is a firewall change.

Fact one: the firewall sees the translated address

DNAT runs in prerouting, before the packet reaches any filter chain. By the time the forward chain evaluates it, the destination has already been rewritten. So the firewall rule that permits a port forward must name the private address, not the public one the client typed.

Client → 203.0.113.10:443
   prerouting DNAT → destination becomes 192.168.1.100:443
      forward chain sees 192.168.1.100:443   ← write the rule against THIS
         postrouting SNAT (if any)

A permit written against 203.0.113.10 is not rejected and does not warn. It simply never matches, the default action drops the packet, and the port forward is broken by the rule that was written to allow it.

The second half of the same trap is interface direction. In a forward rule, inbound-interface name is the interface the packet arrived on and outbound-interface name is the interface it leaves by. A packet from the internet to a LAN server arrives on WAN and leaves by LAN.

Configuration changetranslation and permission, written together

configure

# The translation
set nat destination rule 10 description 'Port forward HTTPS to web01'
set nat destination rule 10 inbound-interface name 'eth0'
set nat destination rule 10 protocol 'tcp'
set nat destination rule 10 destination port '443'
set nat destination rule 10 translation address '192.168.1.100'
set nat destination rule 10 translation port '443'

# The permission - note the PRIVATE address and the interface pair
set firewall ipv4 forward filter rule 20 description 'Forwarded HTTPS to web01'
set firewall ipv4 forward filter rule 20 action accept
set firewall ipv4 forward filter rule 20 protocol tcp
set firewall ipv4 forward filter rule 20 inbound-interface name eth0
set firewall ipv4 forward filter rule 20 outbound-interface name eth1
set firewall ipv4 forward filter rule 20 destination address 192.168.1.100
set firewall ipv4 forward filter rule 20 destination port 443
set firewall ipv4 forward filter rule 20 state new

compare

An inverted rule is hard to read and easy to count

Reading a forward rule and deciding whether its interface pair is the right way round is genuinely difficult, and reviewers get it wrong. Counting is easy and never ambiguous:

show firewall ipv4 forward filter

An inverted permit sits at zero while the chain’s default-action counter climbs. That is the signature, and it takes one command. Confirm direction by counter, not by argument.

Fact two: a NAT rule only affects new flows

The translation for a flow is recorded in its conntrack entry when the first packet is seen. Every later packet in that flow — and every reply — is translated from the entry, not from the rule.

So the moment after you commit, the router is running two policies at once: new connections follow the new rule, established ones carry on with whatever was recorded when they started. This is not a bug and it is not a race; it is how a stateful router works. But it produces two false results if you do not plan for it:

A false pass. You test with a connection that was already open, it works, and you conclude the new rule is fine. It never touched the new rule.
A false fail. You test with something that reuses an existing flow, it behaves the old way, and you conclude the rule is broken.

The discipline is simple: verify with a brand-new connection from the real client, before touching conntrack at all. That tests the new rule in isolation, with no stale state and nothing cleared.

Fact three: clearing flows is the destructive step

Once the new rule is proven on a new flow, the remaining question is whether the existing flows need to move to it. Sometimes they do — you moved a service to a different host and the old translation now points at nothing.

Destructivelist first, then delete the same set

# ALWAYS list with the filter first. This is what the delete will take.
sudo conntrack -L -p tcp -d 203.0.113.10 --dport 443

# Same filter, now as a delete. Blast radius: exactly these flows.
sudo conntrack -D -p tcp -d 203.0.113.10 --dport 443

# Or every flow to the old translation target, when a service moved host
sudo conntrack -D -p tcp -r 192.168.1.100

Even the targeted form is disruptive to the sessions it matches, so it belongs inside the change window with the rest of the work. What it must never become is a reflex applied after every commit.

Choosing the translation address

For source NAT the choice is between a literal address and masquerade, and it is a question about whether the WAN address can change underneath you.

Configuration changemasquerade for a dynamic WAN address

set nat source rule 100 outbound-interface name 'eth0'
set nat source rule 100 source address '192.168.1.0/24'
set nat source rule 100 translation address 'masquerade'

masquerade uses whichever address the egress interface holds at translation time. On DHCP, PPPoE or mobile broadband it is the only safe choice: a literal address becomes wrong at the next lease renewal, and the symptom is that outbound traffic is translated to an address the router no longer holds, so nothing comes back.

A literal address is fine — and marginally cheaper — when the WAN address is static and genuinely assigned to this router:

set nat source rule 100 translation address '203.0.113.10'

The failure mode to know is that VyOS will accept a translation address the router does not hold. The rule commits, traffic is rewritten to an address that routes nowhere back, and the flow simply dies. Check the address is on an interface before you use it.

Hairpin, and why it is a separate decision

A LAN client that connects to the public address of a LAN server gets its destination rewritten by the DNAT rule and reaches the server — but the server replies directly to the client with its own private source address. The client asked 203.0.113.10 and is answered by 192.168.1.100, so the handshake fails.

The fix is a second translation that makes the server reply via the router:

Configuration changehairpin SNAT

set nat source rule 30 outbound-interface name 'eth1'
set nat source rule 30 source address '192.168.1.0/24'
set nat source rule 30 destination address '192.168.1.100'
set nat source rule 30 destination port '443'
set nat source rule 30 protocol 'tcp'
set nat source rule 30 translation address '192.168.1.1'

This is why “does anyone on the LAN need to reach this by its public name?” is a pre-check and not a later discovery. Without hairpin the service works perfectly from outside and is broken for the staff sitting next to it, and that gets reported as an intermittent fault.

Apply, then verify on evidence

Service impact possiblecommit-confirm, verify, then confirm

commit-confirm 10

run show nat destination rules
run show firewall ipv4 forward filter
run show conntrack table ipv4

# only when the checks below pass:
confirm
save

Three pieces of evidence, and all three have to agree:

The NAT rule counter moved. Zero means the rule was never reached — wrong direction, wrong interface, or a lower-numbered rule matched first.

The firewall permit counter moved and the default-action counter did not. This pair is what separates “the permit works” from “the permit is inverted and the traffic is being dropped at the bottom of the chain”.

The conntrack entry carries the translation. The reply tuple is the authoritative statement of what the router will do to the return packet. A rule can exist and a counter can move while the translation is still not what you intended; the conntrack entry is where you find that out.

Read-only / Safesee the packet on both sides of the translation

monitor traffic interface eth0 filter 'port 443'
monitor traffic interface eth1 filter 'port 443'

For inbound DNAT the WAN capture shows the public destination and the LAN capture shows the private one. If the pre-translation packet arrives and the post-translation packet never appears, the translation is not firing or the firewall is dropping it after translation — and the counters from the previous step tell you which.

Reading the rendered ruleset, when the config and the behaviour disagree

VyOS renders the NAT configuration into an nftables ruleset. When the configuration says one thing and the packets say another, the question is what the kernel actually holds.

sudo nft list ruleset

Find the NAT table in the output and read the prerouting and postrouting chains. A rule in show nat destination rules but absent from the rendered ruleset means the commit did not fully apply. A rule present in both with a zero counter means packets are not reaching the hook, which moves the question back to interfaces and routing.

This is a last-resort diagnostic, not a routine step — reach for it when the three evidence sources above disagree, and escalate with its output rather than stacking another change on top of an unexplained one.

Capacity: the limit nobody plans for

A single SNAT translation address supports roughly 28,000 concurrent flows, because that is how many ephemeral ports there are to allocate. Past that, new flows cannot be translated and are dropped.

cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

The failure is indistinguishable from a network fault at the user end — some connections work, some do not, and it varies by the second. Check the count before adding a large subnet to an existing masquerade rule; the fix is more translation addresses or fewer flows, and both are decisions with an owner.

What each step costs if it is wrong

Step	Cost of getting it wrong
Wrong direction (`source` where `destination` was meant)	Rule commits, never matches the intended traffic, and may match something else. Silent.
Firewall rule against the public address	Port forward silently broken by the rule meant to permit it. Client times out with no log line naming the cause.
Inverted `inbound`/`outbound` interface pair	Same symptom, different cause. Distinguishable only by counters.
Translation address not held by the router	Traffic is rewritten to an address nothing routes back to. Flows die with no error anywhere.
Rule number below an existing broader rule	New rule is dead code. The old behaviour continues and looks like the change did not commit.
`conntrack -F` instead of a filtered delete	Every session on the router breaks mid-stream to fix one flow.
Verifying on an existing connection	The test never touches the new rule. False pass, discovered by users.
No hairpin when LAN clients need the public address	Works from outside, broken from inside. Reported as intermittent.
`rollback N` used as the undo	Reboot, which clears all conntrack by clearing everything.

Escalation and the option to hold

Escalate outward once the router’s own evidence is complete. NAT counter non-zero, firewall counter non-zero, conntrack entry correct, service still silent — that is a destination-host problem and further router changes only add variables.

Hold rather than proceed when nobody can state the blast radius of the flow deletion, or when the only clearing plan offered is conntrack -F on a router carrying production traffic. A NAT change without a flow-clearing plan is half-applied by design, and choosing to stop is a better outcome than discovering the radius at 02:00. Record the hold with an owner and an end time.

Why this matters

The reason NAT changes go wrong out of proportion to their apparent size is that the visible artefact — one translation rule — is only a third of the change. The permission is a second rule in a different subsystem that sees a different address than the one you typed, and the state layer decides whether any of it applies to the traffic already flowing.

That third part is what makes NAT unlike a routing change. A route change takes effect for everything at once; a NAT change takes effect for new flows and leaves the rest running the old policy until someone makes a deliberate, disruptive decision to move them. Operators who have not internalised that reach for conntrack -F because it does make the inconsistency go away — and it does, by breaking every session on the router at once to resolve an inconsistency affecting a handful.

Listing the flows before deleting them costs one command and turns the scariest step in this runbook into a measured one.