DNAT Publishes Service Incorrectly

Reported symptoms

The edge router is a VyOS 1.5 LTS box with eth0 on the WAN carrying two addresses from the same /29 — 203.0.113.10 and 203.0.113.11 — and eth1 on the LAN at 192.168.1.1/24. Two hosts are published from behind it: the web server at 192.168.1.100 and, since Tuesday, a partner appliance at 192.168.1.120.

The reports arrive from four directions and none of them says NAT.

• Wednesday 09:05, customer support. Visitors to the public site get a certificate warning. The name on the certificate belongs to the partner appliance.
• Wednesday 09:20, the web team. The access log on 192.168.1.100 has had no external client addresses since Tuesday 14:12. Requests from inside the LAN are arriving normally, so the server is up and serving.
• Wednesday 09:30, the partner’s operations team. Their appliance has been taking requests for hostnames it has never served since Tuesday afternoon, answering them with a 404, and they have opened a ticket asking whether the estate is being scanned.
• Throughout. The external HTTPS monitor for the public site is green, and has never gone red.

Two facts push the investigation away from the router. The SSH publish on 203.0.113.10:22022 still works, which reads as “inbound traffic reaches the web server, so the path is fine and HTTPS is the problem”. And show nat destination rules still lists a rule translating port 443 to 192.168.1.100, unchanged for two years. Tuesday’s change ticket says “publish the partner appliance on the second WAN address”, which nobody connects to a site on the first address.

Evidence provided

The configuration, as it stands:

show configuration commands | match "nat destination"

set nat destination rule 5 inbound-interface name 'eth0'
set nat destination rule 5 protocol 'tcp'
set nat destination rule 5 destination port '443'
set nat destination rule 5 translation address '192.168.1.120'
set nat destination rule 5 translation port '443'
set nat destination rule 10 inbound-interface name 'eth0'
set nat destination rule 10 destination address '203.0.113.10'
set nat destination rule 10 destination port '443'
set nat destination rule 10 protocol 'tcp'
set nat destination rule 10 translation address '192.168.1.100'
set nat destination rule 10 translation port '443'
set nat destination rule 20 inbound-interface name 'eth0'
set nat destination rule 20 destination address '203.0.113.10'
set nat destination rule 20 destination port '22022'
set nat destination rule 20 protocol 'tcp'
set nat destination rule 20 translation address '192.168.1.100'
set nat destination rule 20 translation port '22'

show nat destination rules prints a packet counter beside each rule. Rule 5 has been counting since Tuesday 14:12. Rule 10 has counted nothing since — which is the whole incident in two numbers, if anybody reads them.

$ sudo conntrack -L -n -p tcp --dport 443

tcp 6 299 ESTABLISHED src=198.51.100.5 dst=203.0.113.10 sport=51244 dport=443
src=192.168.1.120 dst=198.51.100.5 sport=443 dport=51244 [ASSURED]
tcp 6 291 ESTABLISHED src=198.51.100.31 dst=203.0.113.11 sport=40118 dport=443
src=192.168.1.120 dst=198.51.100.31 sport=443 dport=40118 [ASSURED]

$ openssl s_client -connect 203.0.113.10:443 -servername www.example.com </dev/null 2>/dev/null | openssl x509 -noout -subject

subject=CN = appliance.partner.example.net

monitor traffic interface eth1 filter 'port 443' shows the same thing from the LAN side: every inbound HTTPS flow leaves the router addressed to 192.168.1.120, whichever public address the client used.

Work the evidence before reading on

Rule 10 is unchanged, correct, and has not translated a packet since Tuesday. Everything you need is in the configuration listing above.

1. Take a TCP packet arriving on eth0 with destination 203.0.113.10:443. Walk the destination NAT rules in the order the router evaluates them. Which rule does it match first, and does that answer agree with the per-rule counters?
2. Write down every field rule 5 constrains, then every field rule 10 constrains. What is in the second list that is missing from the first, and what does its absence mean for the set of packets rule 5 matches?
3. Rule 20 publishes SSH on the same address and still works. What makes it immune?
4. The monitor never went red. What did it actually measure, and what would it have had to measure to see this?

Before continuing: state the one field missing from rule 5, and predict what the estate would have looked like if the same rule had been numbered 30 instead of 5.

Root cause

1. An omitted match condition is a wildcard

Rule 5 constrains four things: the inbound interface, the protocol, the destination port, and — on the translation side — where the packet is sent. It says nothing about the destination address.

In a match-based rule language, a condition you do not write is not a condition that defaults to something sensible. It is a condition that is not tested. Rule 5 therefore matches every TCP packet arriving on eth0 with destination port 443, on any address the interface holds, and rewrites its destination to 192.168.1.120.

The rule reads perfectly well on its own. It is a correct rule for a router with one public address, and it was written by someone thinking about the address they were adding rather than about the address that was already there.

2. First match wins, and the rule number is the whole ordering

VyOS evaluates destination NAT rules in ascending rule number, and the first rule whose conditions match performs the translation. There is no second pass and no “more specific rule wins” — specificity plays no part, only the number.

Rule 5 was numbered 5 deliberately, on the reasoning that the new partner service should be “checked first”. That reasoning inverted the outcome. Every HTTPS packet arriving on eth0 now matches rule 5 before rule 10 is ever considered, so rule 10 — the rule the whole team keeps reading and finding correct — has been dead configuration since the commit at 14:12.

Rule 20 survives because it matches on port 22022 and rule 5 matches on port 443. The two rules do not overlap, so ordering never comes up. That is the entire reason SSH still works, and it is also why the symptom looked like “HTTPS is broken on the web server” rather than “the router is publishing the wrong host”.

3. Nothing in the estate was watching for identity

The monitor opened a TCP connection to 203.0.113.10:443 every minute and reported success. Every one of those connections succeeded, because a healthy host was answering — just not the intended one. The check was true and useless.

The two human reports were the same event from opposite ends. The certificate warning is the client seeing the partner appliance’s identity on the web server’s address. The partner’s “scan” is the appliance receiving the website’s traffic. Neither team could see the other half, so the estate carried two open tickets describing one misdirected translation.

Resolution

1. Confirm the mechanism before changing anything. sudo conntrack -L -n -p tcp -d 203.0.113.10 --dport 443 must show the reply source as the partner appliance; if it shows the web server, the fault is somewhere else and this rule set is not it.
2. Add the missing constraint: set nat destination rule 5 destination address 203.0.113.11. This is the minimal change that makes the rule mean what its author intended.
3. Review the change as a rule set, not a diff. Run compare and read rules 5, 10 and 20 together, asking of each pair whether the earlier rule can match traffic intended for the later one.
4. Commit with a confirmation window — commit-confirm 5 — so a wrong constraint expires by itself rather than requiring a second correct decision from an operator who has just made a wrong one. Then confirm and save.
5. List before you delete. sudo conntrack -L -p tcp -d 203.0.113.10 --dport 443 shows exactly the flows the next command will remove; the two commands take the same filter, which is the point.
6. Remove only those flows: sudo conntrack -D -p tcp -d 203.0.113.10 --dport 443. Do not flush the table.
7. Test both published addresses from outside the estate before declaring the incident closed, because the change moved a boundary between two services rather than repairing one.
8. Close the partner ticket with the explanation rather than as a scan. Their appliance was receiving the traffic this router sent it.

Verification

1. Each public address presents its own certificate. openssl s_client to 203.0.113.10:443 returns the web server subject and to 203.0.113.11:443 returns the partner appliance subject. During the fault both returned the partner subject, so testing one address proves nothing.
2. The translation is right at the router. sudo conntrack -L -n -p tcp -d 203.0.113.10 --dport 443 shows a reply source of 192.168.1.100 for new flows.
3. The translation is right on the wire. monitor traffic interface eth1 filter 'port 443' shows post-NAT destinations of 192.168.1.100 for clients using the first address and 192.168.1.120 for clients using the second.
4. The web server is receiving external clients again. Its access log shows public source addresses, not only LAN addresses.
5. The partner appliance has stopped receiving requests for hostnames it does not serve. This is the other half of the same assertion and confirms nothing was over-corrected.
6. The unaffected publish is still unaffected. An SSH connection to 203.0.113.10:22022 reaches the web server, proving rule 20 was not disturbed by the edit or by the conntrack delete.
7. The new monitor can fail. Point the certificate-subject assertion at the partner address and require it to go red, then point it back. An assertion that has only ever passed has not been tested.

Prevention

• Constrain the destination address on every destination NAT rule, including on a router that has one public address today. The rule that omits it is not simpler; its match set expands the day an address is added to the interface, in a change that never mentions the rule.
• Treat rule numbers as policy. Reserve a numbered block per published address, leave gaps inside each block, and keep the plan written down next to the configuration. Numbers chosen to express importance produce exactly this failure.
• Review NAT as a set. First-match ordering means a rule can only be judged against the rules it now precedes. A diff of one rule is not a review of the change that rule makes.
• Monitor identity, not reachability. A TCP connect on 443 proves a socket answered. Assert something only the intended host can produce — a certificate subject, a version endpoint, a header — for every published address.
• Test every published address on an interface after any change to destination NAT, not the one that changed. The failure mode of this rule class is always somebody else’s service.
• Keep the conntrack -D filter in the change plan. Write it before the window, state its blast radius, and never let conntrack -F become the habit that follows every commit.