Lab: NAT and Port Forwarding

Objective

By the end of this lab, three things will be true on a router you built from nothing: a private LAN reaches an external host through one public address, one TCP service on that LAN is reachable from outside on a different port than it listens on, and a host sitting on the LAN can reach that same service by its public address. Each of the three will be proved from the conntrack table and from packet captures taken on both sides of the router, because “it works” is not evidence and stops being available the moment it does not.

You will then break all three in the three ways that actually happen: a port forward with no firewall permit, a permit written in the wrong direction, and a NAT change applied without a conntrack flush. The third is the interesting one, because the router’s configuration is correct throughout and the traffic is wrong anyway.

Architecture

Four VyOS nodes. One is the router under test; the other three are hosts, run on VyOS purely so the lab needs one ISO rather than three.

                 inet  (stands in for the Internet)
                 eth1 203.0.113.9/24
                 dum0 198.51.100.9/32   <- the "external service"
                         |
                    vmbr96  — WAN segment, 203.0.113.0/24
                         |
                 eth1 203.0.113.10/24   <- the public address
              +---------------------------+
              |           edge            |   the router under test
              +---------------------------+
                 eth2 192.168.10.1/24
                         |
                    vmbr97  — LAN segment, 192.168.10.0/24
                         |
            +------------+------------+
            |                         |
     srv 192.168.10.100        lan 192.168.10.50
     (SSH on tcp/22)           (the inside client)

  every node also has eth0 on vmbr0 — your management LAN, untouched

Three address ranges, three jobs, and the split matters more than it looks. 192.168.10.0/24 is RFC 1918 space: unroutable on the public internet, which is the entire reason NAT exists. 203.0.113.0/24 is the “public” segment — it is an RFC 5737 documentation range, so nothing in this lab can point at a real host. 198.51.100.9/32 sits on a dummy interface on inet and stands in for a service out on the internet; giving it its own address rather than reusing the WAN gateway keeps every capture filter unambiguous about which packet is which.

vmbr96 and vmbr97 are isolated bridges with no physical uplink. vmbr0 carries management only and no step in this lab touches it.

Requirements

• A hypervisor that can run four small VMs. The commands are written for Proxmox VE; any KVM/libvirt host works if you can attach each VM to the right bridges.
• 2 GB RAM for edge, 1 GB each for the other three, and 8 GB of disk each — 5 GB of RAM and 32 GB of disk in total. VyOS 1.5 runs a lab of this size comfortably in 1 GB.
• The VyOS 1.5 LTS ISO uploaded to hypervisor storage. Everything here is written against the VyOS 1.5 configuration tree, where NAT interface matches are spelled outbound-interface name eth0 and inbound-interface name eth0. On 1.3 they are spelled without the name node and these blocks will be rejected.
• Console access to all four VMs — the Proxmox web console or qm terminal. Task 5 installs a default-deny forward policy. It does not filter the management interface and it does not filter traffic to the routers themselves, so your SSH session should survive; keep the console available anyway.
• Roughly two hours. The VM build is about 40 minutes of that.

Scenario

A small office has one public address, 203.0.113.10, on a router that also serves a private LAN. Two things are wanted, and they are usually asked for in the same sentence as though they were one change:

1. Everyone on the LAN should be able to reach the internet.
2. The server at 192.168.10.100 should be reachable from outside for remote administration — but not on port 22, because the office has been told that leaving 22 open collects a permanent background of login attempts.

Those are two different NAT directions, they use different kernel hooks, and they fail in completely different ways. Nothing is configured yet. You will build the starting state, not inherit it.

Tasks

Task 1: Build the starting state

Two isolated bridges first. On the Proxmox host, add two Linux bridges with no ports — no ports is what makes them isolated. This is config text for /etc/network/interfaces, not a command:

auto vmbr96
iface vmbr96 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0

auto vmbr97
iface vmbr97 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0

Apply with ifreload -a, or reboot the node if your host has no ifupdown2. Then create the four VMs. edge needs three NICs; the other three need two:

# Run on the Proxmox host.
# Substitute your own values before running. The ISO volume ID must match
# exactly what `pvesm list local` prints for your upload.
ISO=local:iso/vyos-1.5-lts-amd64.iso
STORE=local-lvm

qm create 260 --name edge --memory 2048 --cores 2 \
  --net0 "virtio,bridge=vmbr0,firewall=0" \
  --net1 "virtio,bridge=vmbr96,firewall=0" \
  --net2 "virtio,bridge=vmbr97,firewall=0" \
  --scsihw virtio-scsi-single --scsi0 "$STORE:8" \
  --ide2 "$ISO,media=cdrom" --boot order=ide2 --ostype l26

qm create 261 --name inet --memory 1024 --cores 1 \
  --net0 "virtio,bridge=vmbr0,firewall=0" \
  --net1 "virtio,bridge=vmbr96,firewall=0" \
  --scsihw virtio-scsi-single --scsi0 "$STORE:8" \
  --ide2 "$ISO,media=cdrom" --boot order=ide2 --ostype l26

for VMID in 262 263; do
  qm create "$VMID" --memory 1024 --cores 1 \
    --net0 "virtio,bridge=vmbr0,firewall=0" \
    --net1 "virtio,bridge=vmbr97,firewall=0" \
    --scsihw virtio-scsi-single --scsi0 "$STORE:8" \
    --ide2 "$ISO,media=cdrom" --boot order=ide2 --ostype l26
done

qm set 262 --name srv
qm set 263 --name lan

for VMID in 260 261 262 263; do qm start "$VMID"; done

firewall=0 on every NIC is load-bearing. Proxmox’s per-interface firewall runs on the host, and leaving it enabled on a routing VM means the host quietly drops transit packets the guest expected to forward. The symptom is indistinguishable from a NAT bug inside VyOS, and you will spend an hour inside the guest looking for it.

Open the console on each VM, log in with the live-boot credentials, run install image, accept the defaults, then move the boot order back to disk and detach the ISO:

# Run on the Proxmox host, after `install image` completes on all four VMs.
for VMID in 260 261 262 263; do
  qm set "$VMID" --boot order=scsi0
  qm set "$VMID" --delete ide2
done

Task 2: Baseline, and watch a router forward a private address

Configure all four nodes. On edge:

configure
set system host-name edge
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth0 description "management - not part of the lab"
set interfaces ethernet eth1 address 203.0.113.10/24
set interfaces ethernet eth1 description "WAN - the public address"
set interfaces ethernet eth2 address 192.168.10.1/24
set interfaces ethernet eth2 description "LAN"
set protocols static route 198.51.100.0/24 next-hop 203.0.113.9
set service ssh
commit
save

On inet — the node that stands in for the internet. Note what it does not get: any route towards 192.168.10.0/24. That absence is the point.

configure
set system host-name inet
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth1 address 203.0.113.9/24
set interfaces ethernet eth1 description "WAN segment"
set interfaces dummy dum0 address 198.51.100.9/32
set interfaces dummy dum0 description "external service address"
set service ssh
commit
save

On srv, the LAN server whose SSH will be forwarded:

configure
set system host-name srv
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth1 address 192.168.10.100/24
set protocols static route 0.0.0.0/0 next-hop 192.168.10.1
set service ssh
commit
save

On lan, the inside client:

configure
set system host-name lan
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth1 address 192.168.10.50/24
set protocols static route 0.0.0.0/0 next-hop 192.168.10.1
set service ssh
commit
save

Save a named baseline on all four nodes. Cleanup restores from it, and a lab you can only run once is not a lab:

save /config/pre-lab.boot

Now the observation that most of this lab rests on. Start a capture on inet, then ping from lan.

$ sudo tcpdump -ni eth1 icmp -c 4

tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
11:02:14.318442 IP 192.168.10.50 > 198.51.100.9: ICMP echo request, id 2211, seq 1, length 64
11:02:15.319884 IP 192.168.10.50 > 198.51.100.9: ICMP echo request, id 2211, seq 2, length 64
11:02:16.321220 IP 192.168.10.50 > 198.51.100.9: ICMP echo request, id 2211, seq 3, length 64

$ ping 198.51.100.9 count 3

PING 198.51.100.9 (198.51.100.9) 56(84) bytes of data.

--- 198.51.100.9 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2039ms

Sit with that pair for a moment, because it contradicts the mental model most people carry into their first NAT change. The packets left. The router forwarded them. It did not drop them for having a private source, because nothing asked it to — a router with no NAT rule and no filter is a router that forwards. The failure is entirely on the return path: inet builds a perfectly good reply and has nowhere to send it. It holds no route towards 192.168.10.0/24, so the reply goes to whatever default its management interface picked up from DHCP, where nothing has ever heard of a LAN behind edge, and it is discarded out of sight.

Write this in your journal. Every NAT failure in the rest of the lab is a variation on it: the forward direction succeeds, and something about the return direction does not.

Task 3: Source translation, and three ways to prove it

On edge, add masquerade for the LAN:

configure
set nat source rule 100 description "LAN to internet"
set nat source rule 100 outbound-interface name eth1
set nat source rule 100 source address 192.168.10.0/24
set nat source rule 100 translation address masquerade
commit-confirm 5

commit-confirm 5 starts a five-minute timer; if you do not type confirm, the router reverts on its own. Nothing here can lock you out — management is on eth0 — but this is the habit that matters on a real edge router, where the NAT you are changing frequently carries the session you are connected over. Type confirm.

translation address masquerade is the difference between a rule that works and a rule that silently does nothing. Masquerade substitutes the current primary address of the outbound interface at translation time, which is what makes it survive a DHCP or PPPoE address change. Leave the translation address clause out entirely and you get a rule that matches traffic and translates nothing — packets leave with their RFC 1918 source intact, exactly as in Task 2, and the router’s own traffic is unaffected so ping from the CLI still works.

Now prove the translation three separate times, because they are three separate claims.

Claim one — the far side sees the public address.

$ sudo tcpdump -ni eth1 icmp -c 4

11:08:41.104223 IP 203.0.113.10 > 198.51.100.9: ICMP echo request, id 2318, seq 1, length 64
11:08:41.104391 IP 198.51.100.9 > 203.0.113.10: ICMP echo reply, id 2318, seq 1, length 64
11:08:42.105772 IP 203.0.113.10 > 198.51.100.9: ICMP echo request, id 2318, seq 2, length 64
11:08:42.105903 IP 198.51.100.9 > 203.0.113.10: ICMP echo reply, id 2318, seq 2, length 64

The source is now 203.0.113.10 and — this is the half that matters — there is a reply. inet has a route to 203.0.113.0/24 because it is directly connected, so the reply is built and sent.

Claim two — the router recorded the translation.

$ sudo conntrack -L -p icmp

icmp     1 29 src=192.168.10.50 dst=198.51.100.9 type=8 code=0 id=2318 packets=3 bytes=252 src=198.51.100.9 dst=203.0.113.10 type=0 code=0 id=2318 packets=3 bytes=252 mark=0 use=1
conntrack v1.4.7 (conntrack-tools): 1 flow entries have been shown.

Read the two halves. The original direction is what the LAN host sent: src=192.168.10.50 dst=198.51.100.9. The reply direction is what the router expects to receive back: src=198.51.100.9 dst=203.0.113.10 — the translated address, not the real one. When a reply arrives, the kernel matches it against reply tuples, finds this entry, and rewrites the destination back to 192.168.10.50.

-p icmp narrows the dump to one protocol, which is the difference between reading one line and paging through a router’s entire flow table. Take the habit now — the same command with -p tcp is what you will use for the port forward, and --dport narrows it further to a single service.

Claim three — the reply reached the original sender.

$ ping 198.51.100.9 count 3

PING 198.51.100.9 (198.51.100.9) 56(84) bytes of data.
64 bytes from 198.51.100.9: icmp_seq=1 ttl=63 time=0.688 ms
64 bytes from 198.51.100.9: icmp_seq=2 ttl=63 time=0.512 ms
64 bytes from 198.51.100.9: icmp_seq=3 ttl=63 time=0.531 ms

--- 198.51.100.9 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2041ms

ttl=63 rather than 64 is worth noticing: one router decremented it. That single byte is often the fastest way to tell “the reply came back through the path I think it did” from “something answered me locally”.

Task 4: Destination translation, on a port that is not the service’s port

Now the inbound half. Forward TCP 2222 on the public address to srv’s SSH on 22 — an address translation and a port translation in one rule:

configure
set nat destination rule 100 description "remote admin to srv"
set nat destination rule 100 inbound-interface name eth1
set nat destination rule 100 protocol tcp
set nat destination rule 100 destination port 2222
set nat destination rule 100 translation address 192.168.10.100
set nat destination rule 100 translation port 22
commit-confirm 5

Confirm. Probe it from inet with VyOS’s TCP reachability test, which completes a handshake rather than sending an ICMP echo:

$ ping tcp 203.0.113.10 port 2222 count 3

And watch what arrives on the LAN segment while the probe runs:

$ sudo tcpdump -ni eth2 'tcp port 22' -c 4

11:19:07.442118 IP 203.0.113.9.51422 > 192.168.10.100.22: Flags [S], seq 2216390411, win 64240, length 0
11:19:07.442604 IP 192.168.10.100.22 > 203.0.113.9.51422: Flags [S.], seq 1846620127, ack 2216390412, win 65160, length 0
11:19:07.442901 IP 203.0.113.9.51422 > 192.168.10.100.22: Flags [.], ack 1, win 502, length 0

Two things in that capture are worth stating out loud.

The destination is 192.168.10.100.22 — both the address and the port were rewritten, in prerouting, before the router decided where to send the packet. That ordering is why DNAT works at all: the routing decision is made against the address the packet will actually be delivered to.

The source is unchanged: 203.0.113.9, the real address of the external client. A port forward translates one direction only. The server sees the client’s real address, which is usually what you want — it is what makes the server’s logs worth reading — and it creates the requirement that makes Task 6 necessary.

$ sudo conntrack -L -p tcp

tcp      6 431999 ESTABLISHED src=203.0.113.9 dst=203.0.113.10 sport=51422 dport=2222 src=192.168.10.100 dst=203.0.113.9 sport=22 dport=51422 [ASSURED] mark=0 use=1
conntrack v1.4.7 (conntrack-tools): 1 flow entries have been shown.

Compare it against the SNAT entry from Task 3. There the reply direction carried the rewritten address; here it is the reply direction that carries the original public destination, because the translation happened on the way in. One table, one format, two directions of NAT — and reading which half changed tells you which rule fired without looking at the rules.

Task 5: The permission DNAT does not grant, and the direction people get wrong

Add the filtering an edge router would actually have. On edge:

configure
set firewall ipv4 forward filter default-action drop
set firewall ipv4 forward filter rule 5 action accept
set firewall ipv4 forward filter rule 5 description "return traffic for established flows"
set firewall ipv4 forward filter rule 5 state established
set firewall ipv4 forward filter rule 5 state related
set firewall ipv4 forward filter rule 10 action accept
set firewall ipv4 forward filter rule 10 description "LAN out to the internet"
set firewall ipv4 forward filter rule 10 inbound-interface name eth2
set firewall ipv4 forward filter rule 10 outbound-interface name eth1
set firewall ipv4 forward filter rule 10 state new
commit-confirm 5

Confirm, then re-run the probe from inet. It fails. Outbound traffic from the LAN still works, because rule 10 permits it and rule 5 carries the replies. The port forward does not, because nothing permits it.

Now write the permit — deliberately wrong first, in the way this is actually got wrong:

configure
set firewall ipv4 forward filter rule 20 action accept
set firewall ipv4 forward filter rule 20 description "remote admin to srv"
set firewall ipv4 forward filter rule 20 inbound-interface name eth2
set firewall ipv4 forward filter rule 20 outbound-interface name eth1
set firewall ipv4 forward filter rule 20 protocol tcp
set firewall ipv4 forward filter rule 20 destination address 192.168.10.100
set firewall ipv4 forward filter rule 20 destination port 22
set firewall ipv4 forward filter rule 20 state new
commit-confirm 5

Confirm, probe from inet again, and it still fails. Everything in that rule reads correctly if you skim it: the right protocol, the right server, the right port. Do not re-read it. Read the counters instead.

$ show firewall ipv4 forward filter

Ruleset Information

ipv4 Firewall "forward filter"

Rule     Action    Protocol    Packets    Bytes    Conditions
-------  --------  ----------  ---------  -------  ------------------------------
5        accept    all         412        38K      state established,related
10       accept    all         96         8.1K     iifname eth2 oifname eth1 state new
20       accept    tcp         0          0        iifname eth2 oifname eth1 daddr 192.168.10.100 dport 22
default  drop                  9          540

Rule 20 has never matched a packet and the default action has. That pair is the diagnosis, and it took no knowledge of what the rule was supposed to do — a permit rule at zero while the default-action counter climbs means the traffic is not shaped the way the rule describes.

It also rules out the other suspect without an argument. If the DNAT rule had stopped matching, the packet would never have reached the forward chain in that shape at all; confirm the translation is still configured and then stop looking at it:

$ show nat destination rules

In a forward rule, inbound-interface is the interface the packet arrived on and outbound-interface is the interface it leaves by. A packet from the outside to a LAN server arrives on eth1 and leaves by eth2. Rule 20 describes the opposite journey. Fix it:

configure
set firewall ipv4 forward filter rule 20 inbound-interface name eth1
set firewall ipv4 forward filter rule 20 outbound-interface name eth2
commit-confirm 5

Confirm and probe again. It succeeds, and rule 20’s counter moves.

Task 6: The hairpin, and why the inside client is the awkward one

From lan, probe the service the way an office laptop would — by the public address, because that is what the bookmark and the DNS record say:

$ ping tcp 203.0.113.10 port 2222 count 3

It fails, and it fails in two distinct ways one after the other. Getting from the first to the second is the whole task.

Stage one — the router answers, and refuses. Capture the LAN segment from edge while the probe runs:

$ sudo tcpdump -ni eth2 'tcp port 22 or tcp port 2222' -c 4

11:31:52.664201 IP 192.168.10.50.44118 > 203.0.113.10.2222: Flags [S], seq 3311902264, win 64240, length 0
11:31:52.664402 IP 203.0.113.10.2222 > 192.168.10.50.44118: Flags [R.], seq 1, ack 3311902265, win 0, length 0

The reset came from 203.0.113.10 — the router’s own public address. No translation happened at all, and the reason is in the rule you wrote in Task 4: inbound-interface name eth1. This packet arrived on eth2, so the rule did not match; the destination stayed as the router’s own address, the routing decision delivered it locally, and nothing was listening on 2222.

That interface match is not a mistake — a destination-NAT rule that matches on every interface is a rule you cannot reason about. The right fix is a second rule, scoped to the inside, that says the same thing about traffic arriving from the LAN:

configure
set nat destination rule 110 description "NAT reflection - inside clients to srv"
set nat destination rule 110 inbound-interface name eth2
set nat destination rule 110 protocol tcp
set nat destination rule 110 destination address 203.0.113.10
set nat destination rule 110 destination port 2222
set nat destination rule 110 translation address 192.168.10.100
set nat destination rule 110 translation port 22

The forward chain needs telling as well, and for the same reason it did in Task 5: this packet enters on eth2 and leaves on eth2, a journey that neither rule 10 nor rule 20 describes. Add it in the same session, so the two halves of one change commit together:

set firewall ipv4 forward filter rule 30 action accept
set firewall ipv4 forward filter rule 30 description "reflected LAN traffic to srv"
set firewall ipv4 forward filter rule 30 inbound-interface name eth2
set firewall ipv4 forward filter rule 30 outbound-interface name eth2
set firewall ipv4 forward filter rule 30 protocol tcp
set firewall ipv4 forward filter rule 30 destination address 192.168.10.100
set firewall ipv4 forward filter rule 30 destination port 22
set firewall ipv4 forward filter rule 30 state new
commit-confirm 5

Stage two — the server answers, and the client refuses. Confirm, probe again, and capture again:

$ sudo tcpdump -ni eth2 'tcp port 22 or tcp port 2222' -c 4

11:34:06.118442 IP 192.168.10.50.44202 > 192.168.10.100.22: Flags [S], seq 2884113071, win 64240, length 0
11:34:06.118904 IP 192.168.10.100.22 > 192.168.10.50.44202: Flags [S.], seq 921144870, ack 2884113072, win 65160, length 0
11:34:06.119102 IP 192.168.10.50.44202 > 192.168.10.100.22: Flags [R], seq 2884113072, win 0, length 0

Trace it. The client sent to 203.0.113.10:2222. Rule 110 matched this time, rewrote the destination to 192.168.10.100:22, and edge routed the packet back out of the interface it arrived on — the “hairpin”.

The server replied, and because the client is on its own subnet it replied directly, with a source of 192.168.10.100. That reply never touched edge, so nothing reversed the translation. The client sent a SYN to 203.0.113.10 and got a SYN/ACK from 192.168.10.100; those do not belong to the same connection, so it answered with a reset. The R in the third line is the client refusing an answer it never asked for.

The fix is to make the reply come back through the router, and the only lever that does that is the source address the server sees. Translate it to the router’s own LAN address:

configure
set nat source rule 110 description "NAT reflection - inside clients to srv"
set nat source rule 110 outbound-interface name eth2
set nat source rule 110 source address 192.168.10.0/24
set nat source rule 110 destination address 192.168.10.100
set nat source rule 110 destination port 22
set nat source rule 110 protocol tcp
set nat source rule 110 translation address 192.168.10.1
commit-confirm 5

The source rule matches destination address 192.168.10.100 and destination port 22 — the post-DNAT values — because source NAT is evaluated in postrouting, after the destination has already been rewritten. Naming 203.0.113.10 and 2222 here matches nothing.

Confirm, probe again from lan, and it succeeds. The capture now shows the server answering 192.168.10.1 — the router — which holds the conntrack entry needed to reverse both translations before handing the packet back to the client with a source of 203.0.113.10.

$ sudo conntrack -L -p tcp --dport 2222

tcp      6 431999 ESTABLISHED src=192.168.10.50 dst=203.0.113.10 sport=44118 dport=2222 src=192.168.10.100 dst=192.168.10.1 sport=22 dport=44118 [ASSURED] mark=0 use=1
conntrack v1.4.7 (conntrack-tools): 1 flow entries have been shown.

That single entry is the whole of hairpin NAT. Original direction: what the client sent, with the public destination. Reply direction: src=192.168.10.100 dst=192.168.10.1 — the server answering the router rather than the client. Both rewrites are recorded together, and the kernel unwinds both on the way back.

Task 7: Change a NAT rule under a live flow

This is the failure with no misconfiguration in it. Start a long ping from lan and leave it running:

$ ping 198.51.100.9 count 120

Now, on edge, take the masquerade rule away while that flow is in flight:

configure
delete nat source rule 100
commit

Look at the running ping. It keeps working. Then start a second ping, from srv this time, to the same destination — and it fails immediately.

The rule set now says one thing and the traffic does two different things, and the conntrack table explains both:

$ sudo conntrack -L -p icmp

icmp     1 29 src=192.168.10.50 dst=198.51.100.9 type=8 code=0 id=2318 packets=61 bytes=5124 src=198.51.100.9 dst=203.0.113.10 type=0 code=0 id=2318 packets=61 bytes=5124 mark=0 use=1
icmp     1 29 src=192.168.10.100 dst=198.51.100.9 type=8 code=0 id=2401 packets=4 bytes=336 src=198.51.100.9 dst=192.168.10.100 type=0 code=0 id=2401 packets=4 bytes=336 mark=0 use=1
conntrack v1.4.7 (conntrack-tools): 2 flow entries have been shown.

The first entry still has dst=203.0.113.10 in its reply direction: it was created while the rule existed, and NAT rules are evaluated once, for the first packet of a flow. The second entry, created after the delete, has dst=192.168.10.100 — no translation, because there is no longer a rule to apply. Nothing is going to revisit the first entry. It will keep being translated by a rule that no longer exists until the flow times out.

Flush it and watch the first ping die:

$ sudo conntrack -F

conntrack v1.4.7 (conntrack-tools): connection tracking table has been emptied.

Now restore the rule and confirm that new flows work again:

configure
set nat source rule 100 description "LAN to internet"
set nat source rule 100 outbound-interface name eth1
set nat source rule 100 source address 192.168.10.0/24
set nat source rule 100 translation address masquerade
commit
save

Production notes — running this as a change

Everything above happened in one sitting on an isolated bridge. The same work on a live edge router is shaped differently, and the difference is worth writing down while the mechanics are fresh.

The window is set by conntrack, not by the commit. The commit itself is sub-second and reversible. The flush is the disruptive step, so the window has to be long enough to cover the flush plus the time for clients to re-establish. For an office edge that is a few minutes at a time when sessions are cheap to lose; for anything carrying long-lived flows — database replication, VPN concentrators, file transfers — the flush is the change, and it needs its own notification.

Order the steps so the risky one is last and reversible. Add the firewall permit before the DNAT rule, not after — the reverse of the order this lab used, which was chosen to make the missing permit visible. A permit for a translation that does not exist yet matches nothing and is harmless; a translation with no permit is a service that appears broken during the window for no reason a monitoring system can explain.

“Hold” is a first-class outcome. If the probe from outside fails after the commit and the counters do not tell you why within a few minutes, the right move is to revert and reschedule rather than to keep adding rules under time pressure. Give that decision an owner and an end time in the change ticket before you start. commit-confirm enforces it mechanically: say nothing and the router reverts itself.

Every rule needs an owner and an expiry. Port forwards outlive the reason they were created: the server is decommissioned, the rule stays, and it translates to an address something else picks up later. The description field is the cheapest place to record what a rule is for and who asked for it.

Validation

Work through each item and confirm the stated result before you call the lab done.

• ping 198.51.100.9 count 3 from lan and from srv both return three replies.
• A capture on inet shows those echo requests with a source of 203.0.113.10, and no packet anywhere in it has a 192.168.10.x source.
• sudo conntrack -L -p icmp on edge shows an entry whose original direction holds a private source and whose reply direction holds 203.0.113.10.
• ping tcp 203.0.113.10 port 2222 count 3 succeeds from inet.
• A capture on edge’s eth2 during that probe shows the destination as 192.168.10.100.22 and the source unchanged as 203.0.113.9.
• show firewall ipv4 forward filter shows a non-zero packet counter on rule 20, and your journal holds the earlier capture of the same command with rule 20 at zero and the default action climbing.
• show nat source rules and show nat destination rules list the rules you configured, with the descriptions you gave them.
• ping tcp 203.0.113.10 port 2222 count 3 succeeds from lan, and the conntrack entry for that flow shows dst=192.168.10.1 in its reply direction.
• Your journal records the Task 2 capture — private source arriving at inet — and one sentence on why the ping still failed.
• Your journal records the Task 7 pair: one flow surviving the deletion of the rule that translates it, and one flow created after the deletion that does not.
• Answer in writing: which single piece of evidence used in this lab would have looked healthy in every failure you produced? The answer is the reason this list has eleven items rather than one.

Expected Outcome

edge carries two source-NAT rules and two destination-NAT rules: source rule 100 masquerading the LAN outbound, source rule 110 reflecting inside clients back to the server, destination rule 100 forwarding TCP 2222 inbound from the WAN, and destination rule 110 doing the same for traffic that arrives from the LAN. The forward chain defaults to drop and carries four accept rules, one of which you fixed after reading its counter. All four nodes still hold /config/pre-lab.boot, so the topology resets to its baseline without a rebuild.

The journal holds every capture named in Validation, in order, plus the written answer to the closing question.

Troubleshooting

Outbound pings from lan fail even after the masquerade rule. Check that lan has a default route pointing at 192.168.10.1 (show ip route 0.0.0.0/0) and that edge has the route to 198.51.100.0/24 via 203.0.113.9. A NAT rule cannot translate traffic that never reaches the router.

sudo conntrack -L shows no entries at all. Nothing has traversed the router since the last flush, or you are looking at a router that is not on the path. Run it while a ping is in flight — ICMP entries are short-lived.

The port forward works from inet but not from lan, after Task 6. Three rules have to line up for the inside path and each fails differently. Destination rule 110 must be scoped to inbound-interface name eth2 and match the public address and port, because it runs before the rewrite. Source rule 110 must match the private address and port 22, because it runs after. Forward rule 30 must name eth2 for both interfaces. A capture on eth2 tells you which stage you are at: a reset from 203.0.113.10 means the destination rule never matched, a reset from the client means the source rule did not.

Rule 20’s counter moves but the probe still fails. The forward chain is permitting the packet and something further on is not. Capture on eth2 to confirm the SYN reaches the server, then check the server’s own firewall ipv4 input filter — srv is a router too, and its input chain governs traffic addressed to it.

Everything worked and then stopped after a commit that changed nothing relevant. Look at conntrack before looking at the rules. A commit that touches the NAT tree causes the rule set to be re-rendered, but existing entries are unaffected; a flow that was working under an old translation may now be the only thing behaving differently.

commit fails with a validation error on the NAT tree. Most likely you are on VyOS 1.3, where the interface match is outbound-interface eth1 without the name node. Check show version; this lab requires 1.5.

Cleanup

Everything in this lab lives on isolated bridges and management was never touched, so cleanup is about leaving a reproducible state rather than restoring service.

Step 1. On each of the four nodes, load the baseline you saved in Task 2 and confirm the difference before applying it:

configure
load /config/pre-lab.boot
compare
commit
save

compare before commit is the step that matters: it shows exactly what loading the file will change. An empty comparison means you are already at the baseline.

Step 2. Flush conntrack on edge. This is not optional — the point of Task 7 is that removing the rules does not remove their effects:

$ sudo conntrack -F

Step 3. Confirm the router is back to forwarding without translating. Ping 198.51.100.9 from lan again; it should fail exactly as it did in Task 2, and a capture on inet should again show a 192.168.10.50 source. Failing in the original way is the proof that cleanup worked.

Step 4. Remove the VMs, then the bridges.

# Run on the Proxmox host, after confirming these IDs with `qm list`.
for VMID in 260 261 262 263; do
  qm stop "$VMID"
  qm destroy "$VMID" --purge
done

Then remove the vmbr96 and vmbr97 stanzas from /etc/network/interfaces and run ifreload -a. Confirm with ip link show vmbr96, which should report that the device does not exist.

What You Learned

• A router with no NAT forwards private addresses quite happily. The Task 2 capture showed 192.168.10.50 arriving at a host that had no route back. Nothing dropped it; the failure was entirely on the return path, and every NAT failure after that was a variation on the same theme.
• Conntrack is the source of truth, and it holds both directions. You read the same table for a source translation and a destination translation and told them apart by which half had changed — without looking at a single rule.
• DNAT translates; it does not permit. The port forward worked until the firewall existed, then needed a rule of its own, written against the private address because prerouting had already rewritten it.
• A permit rule at zero while the default-action counter climbs is a complete diagnosis. You found an inverted interface pair from two numbers, not by re-reading the rule that looked correct.
• Hairpin NAT is a fix with a cost you have to accept on purpose. It works by hiding every inside client behind the router’s LAN address, and the server’s logs lose the ability to say which machine.
• A NAT change is a rule change plus a conntrack decision. In-flight flows keep the translation they were born with, so the change is not complete — and not honestly rollback-tested — until you have said what happens to them.