NAT Fails After WAN Change

Reported symptoms

br-edge-01 is a VyOS 1.5 LTS router with two circuits and two internal networks. eth0 is the primary WAN at 203.0.113.6/30 with next hop 203.0.113.5; eth1 is the backup at 198.51.100.6/30 with next hop 198.51.100.5. eth2 carries the office at 192.168.20.0/24 and eth3 carries a small DMZ at 192.168.30.0/24. The router runs the office’s DNS forwarder.

At 02:40 the primary circuit failed — a genuine hard failure, carrier lost — and the backup default route installed. The on-call engineer checked show ip route, saw the backup selected, recorded the failover as successful and went back to bed. That entry is accurate.

• 08:05, the office. Nothing on the Internet is reachable. Connections hang and time out rather than failing immediately, which users describe as “slow” for the first half hour.
• 08:05 onwards, the service desk. Name resolution works. Users can resolve any hostname they try, so DNS is ruled out early and everybody moves on to the firewall.
• 08:20, the DMZ owner. No problem at all. Their hosts have been reaching the Internet all night, through the same router, on the same default route.
• Throughout. The router reaches the Internet fine, and the monitoring system’s synthetic Internet check — which runs on the router — is green and has never dipped.

A router with working Internet access, a working second subnet, and working DNS is not where anybody starts looking for a broken office network. The first three hours go to the office switch stack, the DHCP scope and the firewall.

Evidence provided

Routing is not the problem, and it is worth proving that early rather than assuming it:

show ip route 0.0.0.0/0

S>* 0.0.0.0/0 [200/0] via 198.51.100.5, eth1, weight 1, 05:41:12

The NAT rules, which have not been touched in a year:

show configuration commands | match "nat source"

set nat source rule 10 outbound-interface name 'eth0'
set nat source rule 10 source address '192.168.20.0/24'
set nat source rule 10 translation address 'masquerade'
set nat source rule 20 outbound-interface name 'eth0'
set nat source rule 20 source address '192.168.30.0/24'
set nat source rule 20 translation address 'masquerade'
set nat source rule 30 outbound-interface name 'eth1'
set nat source rule 30 source address '192.168.30.0/24'
set nat source rule 30 translation address 'masquerade'

show nat source rules prints a packet counter beside each rule. Rule 10 stopped counting at 02:40. Rule 20 stopped at 02:40. Rule 30 has been counting all night. Those three counters are the entire incident, if anybody reads them.

$ sudo conntrack -L -s 192.168.20.0/24 -p tcp

tcp 6 110 SYN_SENT src=192.168.20.50 dst=192.0.2.40 sport=51992 dport=443 [UNREPLIED] src=192.0.2.40 dst=192.168.20.50 sport=443 dport=51992 mark=0 use=1
tcp 6  98 SYN_SENT src=192.168.20.71 dst=192.0.2.80 sport=39114 dport=443 [UNREPLIED] src=192.0.2.80 dst=192.168.20.71 sport=443 dport=39114 mark=0 use=1

$ sudo conntrack -L -s 192.168.30.0/24 -p tcp

tcp 6 431982 ESTABLISHED src=192.168.30.25 dst=192.0.2.80 sport=44112 dport=443 [ASSURED] src=192.0.2.80 dst=198.51.100.6 sport=443 dport=44112 mark=0 use=1

The two outputs differ in one field, and that field is the whole diagnosis. Then the capture that removes any remaining doubt:

$ sudo tcpdump -ni eth1 'src net 192.168.20.0/24' -c 2

09:14:22.118437 IP 192.168.20.50.51992 > 192.0.2.40.443: Flags [S], seq 2419887311, win 64240, length 0
09:14:22.664901 IP 192.168.20.71.39114 > 192.0.2.80.443: Flags [S], seq 881204773, win 64240, length 0

show interfaces ethernet eth0 shows the link down with one carrier transition, timestamped 02:40. The DMZ rule on eth1 turns out to have been written a year ago by a different engineer, deliberately, because DMZ traffic to a partner is policy-routed out the backup circuit. Nobody wrote the equivalent for the office, because the office had no reason to use eth1.

Work the evidence before reading on

Routing is correct, the circuit is healthy, and one subnet works. Everything needed is in the rule listing and the two conntrack outputs.

1. Compare the reply direction of the office flow with the reply direction of the DMZ flow. Which field differs, and what does that field mean in a SNAT’d flow?
2. Read the three NAT rules as a set and ask, for each internal subnet, which egress interfaces it has a rule for. Then compare that list with the interfaces routing can currently choose.
3. The router itself reaches the Internet. Explain why, in terms of what its packets carry when they leave, and decide what that fact does and does not prove about the estate.
4. DNS works from office clients while nothing else does. Which host is actually sending those queries to the upstream resolver?
5. Office connections hang instead of being refused. What happens to a TCP SYN with an RFC 1918 source address once it reaches an ISP?

Before continuing: name the rule that does not exist, and predict what the DMZ would have experienced if its second rule had never been written.

Root cause

1. A source NAT rule is scoped by the egress routing already chose

Source NAT runs after the routing decision, on the way out of the interface routing selected. outbound-interface name eth0 is therefore not a description of where traffic goes; it is a condition that is true only while routing is choosing eth0.

For three years that condition was always true, because eth0 was the only WAN. Rule 10 read as “translate the office”, and behaved as “translate the office”, and the interface constraint on it was invisible. At 02:40 routing started choosing eth1 and the constraint became load-bearing for the first time. Rule 10 stopped matching. No rule covers 192.168.20.0/24 on eth1, so the office was left with no translation at all — not a wrong translation, which would have been noisier, but none.

Untranslated packets left the backup circuit carrying RFC 1918 sources and were discarded upstream without a response of any kind. That is why the office sees hangs rather than refusals: there is nothing on the other end of the failure to send an RST or an ICMP error, and the client simply retransmits until it gives up.

2. Two working things made the fault look impossible

The router’s own traffic was fine because the router does not need NAT. A packet it originates already leaves with the source address of the interface it leaves by, which is public and correct on either circuit. Every check that runs on the router — the synthetic monitor, a manual ping, the DNS forwarder’s queries to its upstreams — therefore passed throughout.

That last one is why DNS worked for users. Office clients ask the router, the router asks its upstream resolver on its own behalf, and the answer comes back. Name resolution was never crossing the broken part of the path, so “it resolves, so it is not DNS, so it must be the firewall” was a reasonable inference from a misleading fact.

The DMZ worked for an unrelated reason: someone had already needed DMZ traffic on eth1 for a partner path and had written rule 30 to match. Its correctness here is a coincidence, but it is the coincidence that makes the fault diagnosable — two subnets on one router, one default route, one working and one not, points at the rule set and nowhere else.

Resolution

1. Prove the mechanism before changing anything. sudo conntrack -L -s 192.168.20.0/24 -p tcp must show untranslated flows, and a capture on eth1 must show private sources leaving. If the reply destination is already a WAN address, translation is happening and the fault is somewhere else.
2. Read the whole rule set before choosing a rule number. Source NAT is evaluated in ascending order and the first match performs the translation, so a number placed in front of a narrower rule captures its traffic; the DMZ rule on eth1 is the one at risk here.
3. Add the missing rule: set nat source rule 110 outbound-interface name eth1, set nat source rule 110 source address 192.168.20.0/24, set nat source rule 110 translation address masquerade.
4. Use masquerade rather than a literal address. A static translation to an address the interface does not hold sends traffic out with a source nothing will route back, which is a harder fault than this one because packets do leave and the rule does count.
5. Review with compare, then apply with commit-confirm 10. Verify the counters before confirm, so a rule that captures the wrong traffic expires rather than persisting.
6. Do not flush the connection tracking table. The stuck office flows are unestablished and carry no translation, so they retry into the new rule within seconds; a flush would drop every working DMZ flow on the router to fix a subnet that is already fixed.
7. Close the gap rather than the ticket: rule 20 covers the DMZ on eth0 and rule 30 covers it on eth1, but the office now has eth0 and eth1 while any future subnet will have neither until someone remembers. Write the rule set as a matrix in the same change.
8. Tell the service owners what the fix changes. Office traffic now leaves with the backup circuit address, so partner allowlists, SPF records and anything else pinned to the primary address are wrong until they list both.

Verification

1. New office flows are translated. sudo conntrack -L -s 192.168.20.0/24 -p tcp shows a reply destination of 198.51.100.6 instead of the private client address. This is the same field that identified the fault, read after the change.
2. Nothing untranslated leaves the WAN. sudo tcpdump -ni eth1 'src net 192.168.20.0/24' now shows no packets at all; during the fault it showed every one of them.
3. The new rule counts and the DMZ rule keeps counting. Both advancing in show nat source rules proves rule 110 was inserted without capturing rule 30 traffic, which is the failure mode of this particular fix.
4. Test from a host in the office subnet, not from the router. Router-originated traffic passed through this entire fault and will pass through a repeat of it.
5. An external endpoint sees the backup circuit address as the source. This is the assertion that turns a NAT rule into a working path, and it is the one that tells the allowlist owners what to add.
6. The named casualties were re-tested: partner endpoints filtering on source address, and any outbound mail path whose SPF record lists only the primary circuit.
7. The other direction of the failover is tested when the primary circuit returns. Traffic moves back to eth0, rule 10 resumes counting, and the office still reaches the Internet - failback is a second path with a second set of rules and it has never been exercised either.

Prevention

• Write NAT as a matrix. Internal source prefixes on one axis, every WAN routing can select on the other, a rule in every cell. Three subnets and two circuits is six rules, and the gap between that and three is invisible until a circuit fails at 02:40.
• Audit the egress set from the routing configuration. The question is not which interface is the WAN; it is which interfaces routing is permitted to choose. Every one of them needs the same policy, and that includes firewall rules, MSS clamps and policy routes, not only NAT.
• Make the failover drill assert what users experience. A host in each internal subnet reaching the Internet while the primary is disabled. A drill that stops at show ip route validates the half of the failover that was never in doubt.
• Do not monitor the estate from the router. Router-originated traffic needs no translation and crosses none of the policy that applies to users. Probe from a host in each internal subnet, or the green check will outlive the outage.
• Read the per-rule counters first. A NAT rule whose counter froze at a timestamp names the fault and the minute it started, and it is the cheapest evidence available in any NAT incident.
• Keep the address dependencies on file. Partner allowlists, SPF records and anything else pinned to a public address must list both circuits. That work cannot be done during an incident and is the part of a failover that no routing change covers.