Firewall Blocks Routed Traffic · Break/Fix · VyOS

Reported symptoms

A partner integration went live on Tuesday. The design is unremarkable: an SFTP endpoint published on 203.0.113.30 at the DC-East edge, destination NAT to 10.30.0.12 in the DMZ, and a permit in the WAN-DMZ chain that governs traffic from the WAN zone to the DMZ zone.

The partner cannot connect. Their client sits there and times out; it is never refused. From where they are standing, the host is switched off.

By Thursday the incident had three owners and no cause:

The change ticket says the router is fine. show nat destination shows the rule matching, with a counter that rises in step with the partner’s retry interval. NAT is working. That is not a guess, it is a counter.
The server team has closed their half. The service answers from a jump host inside the DMZ, and it answers from the office LAN through the public address. The daemon is up, listening and reachable — from everywhere except the one place it is meant to be reachable from.
The firewall has never heard of the address. show log firewall | grep 203.0.113.30 returns nothing at all. The obvious reading is that the partner’s traffic is not arriving, so the network team started with the upstream provider.

Two facts nobody has connected yet. A publication configured fourteen months ago on this same router, through this same chain, works perfectly. And a second engineer published an SMTP relay in the same change window; it is also silently unreachable, and it has its own ticket.

Evidence provided

Read-only / SafeWAN side: the partner SYN is arriving, and retrying

$ sudo tcpdump -ni eth0 'host 203.0.113.30 and tcp port 22'

09:41:02.113455 IP 198.51.100.77.44120 > 203.0.113.30.22: Flags [S], seq 1884213, win 64240, length 0
09:41:03.137902 IP 198.51.100.77.44120 > 203.0.113.30.22: Flags [S], seq 1884213, win 64240, length 0
09:41:05.177881 IP 198.51.100.77.44120 > 203.0.113.30.22: Flags [S], seq 1884213, win 64240, length 0

Illustrative output

Read-only / SafeDMZ side: nothing leaves the router towards the server

$ sudo tcpdump -ni eth2 'host 10.30.0.12 and tcp port 22'

0 packets captured

Illustrative output

Read-only / Safethe reply tuple already names the internal address

$ sudo conntrack -L -d 203.0.113.30 -p tcp

tcp 6 108 SYN_SENT src=198.51.100.77 dst=203.0.113.30 sport=44120 dport=22 [UNREPLIED] src=10.30.0.12 dst=198.51.100.77 sport=22 dport=44120 mark=0 use=1
conntrack v1.4.7 (conntrack-tools): 1 flow entries have been shown.

Illustrative output

Read-only / Safechain WAN-DMZ: rule 40 exists, reads correctly, and has never fired

$ show firewall

Rule     Action   Packets    Bytes   Description
----     ------   -------    -----   -----------
10       accept   91422      7.3M    Established/related
20       drop     4          320     Invalid
30       accept   208114     14M     HTTPS to web server
40       accept   0          0       SFTP to partner endpoint
9999     drop     11908      714K    Default deny

Illustrative output

Read-only / Safethe rule that passed review

$ show configuration commands | match 'WAN-DMZ rule 40'

set firewall ipv4 name WAN-DMZ rule 40 action 'accept'
set firewall ipv4 name WAN-DMZ rule 40 description 'SFTP to partner endpoint'
set firewall ipv4 name WAN-DMZ rule 40 destination address '203.0.113.30'
set firewall ipv4 name WAN-DMZ rule 40 destination port '22'
set firewall ipv4 name WAN-DMZ rule 40 protocol 'tcp'
set firewall ipv4 name WAN-DMZ rule 40 state 'new'

Illustrative output

Read-only / Safethe publication that works, for comparison

$ show configuration commands | match 'WAN-DMZ rule 30 destination'

set firewall ipv4 name WAN-DMZ rule 30 destination address '10.30.0.9'
set firewall ipv4 name WAN-DMZ rule 30 destination port '443'

Illustrative output

Read-only / Safethe log entries that were said not to exist— the drops are here, logged with DST=10.30.0.12 — the search everybody ran was for 203.0.113.30

$ show log firewall | grep 10.30.0.12 | tail -2

Work the evidence before reading on

Nothing here is missing. Every command answers, every counter is consistent with every other counter, and the contradiction is in what they mean rather than in what they say.

The SYN arrives on the WAN interface and nothing leaves on the DMZ interface. Between those two captures the packet was discarded. Which parts of the router had a chance to discard it, and in what order?
The conntrack entry’s reply tuple names 10.30.0.12. What does the presence of that address, in that half of the entry, tell you about what has already happened to the packet before any filter rule sees it?
Rule 40 reads correctly and has fired zero times. Rule 30, which works, is written differently in exactly one respect. Put the two side by side.
grep 203.0.113.30 on the firewall log is empty and grep 10.30.0.12 is not. What is the log recording — the packet as it arrived, or the packet as the firewall saw it?

Before continuing: at the moment the WAN-DMZ chain evaluates this packet, what is in its destination address field?

Root cause

1. NAT happens first, and it changes the thing the rule matches on

The packet path is fixed. Destination NAT is evaluated in the prerouting hook; the routing decision comes next; the forward filter chain comes after that. So the sequence for this flow is:

The SYN arrives with destination 203.0.113.30:22.
Prerouting DNAT rewrites the destination to 10.30.0.12:22. This is the counter everybody has been citing as proof the router is healthy.
The routing decision selects the DMZ interface.
The WAN-DMZ chain evaluates a packet whose destination is 10.30.0.12.

Rule 40 says destination address 203.0.113.30. At step 4 there is no packet anywhere in the router with that destination address, and there has not been since step 2. The rule is unmatchable. Rules 10, 20 and 30 do not describe an inbound SFTP connection either, so the packet reaches the end of the chain and the default action drops it.

The rule was reviewed and approved because it is the rule a reasonable person writes. It names the address in the ticket, in the diagram and in the partner’s connection string. It is simply written against the wrong stage of the packet path.

2. The NAT counter is the trap, not the reassurance

The DNAT counter rising is true, and its meaning is the opposite of the one it was given. NAT firing is not evidence that the router is passing the traffic — it is the event that made the filter rule stop matching.

This is worth sitting with, because the same shape recurs: a working component whose correct operation is a precondition for the failure. As long as “NAT works” is treated as evidence for “the firewall must be fine”, the investigation cannot reach the answer.

3. The log recorded the truth under a name nobody searched for

Firewall logging happens in the filter chain, so the entry describes the packet as the filter saw it — post-translation. Every log line for this flow carries DST=10.30.0.12. Every search anybody ran was for 203.0.113.30, because that is the address in the ticket.

An empty grep was then read as “the traffic never arrived”, which sent the network team to the upstream provider for most of a day. The evidence was present the entire time, indexed under the address that the incident had no reason to know about yet.

Which address does each subsystem see?

One table settles most of this class of question, and it is worth holding in your head when writing rules on a router that also translates.

For traffic arriving from outside to a published service, destination NAT runs first, so every filter chain that follows sees the internal destination. Write those rules against the internal address.

For traffic leaving to the internet from inside, source NAT runs last, in postrouting, so the filter chains that precede it still see the internal source. Write those rules against the internal address too.

The rule of thumb that falls out is easier to remember than the hook ordering: the firewall almost always sees the private address. The public address belongs to the NAT rule and to the conntrack entry, and appears in the filter chain only on paths where no translation applies at all.

The conntrack entry is where both views are visible at once, which is why it is the fastest single piece of evidence here. The original tuple is the packet as it arrived; the reply tuple is the packet as it will be answered. When those two disagree about an address, translation has happened, and you know immediately which of the two the filter chain is working with.

Resolution

Say out loud in the incident channel that the NAT counter is not evidence the router is passing traffic, and show the conntrack reply tuple. Until that is agreed, the investigation keeps returning to the provider and the server team.
Correct rule 40 to name the post-translation destination — the DMZ host address and the port the host is listening on — and leave the NAT rule untouched. It was never wrong.
Scope the corrected rule to the ingress it is meant to serve. An internal destination address is reachable from every zone that can route to it, whereas the public address it replaced was implicitly external-only, so pin the rule to the inbound interface or the source zone. Restoring the service must not silently widen it.
Apply the change under commit-confirm. This is a WAN-facing chain on a router you are reaching across the network you are editing, and a mistake here is a console visit.
Watch rule 40 counter before confirming the commit. A rule that reads correctly and counts zero is precisely the state you started in, so the counter moving is the first real evidence you have had.
Fix the SMTP publication in the same change. It is the same defect, it was made in the same window, and leaving it to its own ticket means running this incident again next week.
Refuse the broad any-to-DMZ permit, including as a temporary measure. If there is no window to make the change properly, the honest position is that the service is not published yet — with a named owner and a date — rather than published through a hole.

Verification

The permit rule fires. The rule 40 packet counter advances when the partner reconnects. Configuration review passed on this rule for two days while the service was dead, so the counter is the check that means something.
The state is complete. sudo conntrack -L for the four-tuple shows ESTABLISHED with the [ASSURED] flag and non-zero counters in both directions, replacing the SYN_SENT [UNREPLIED] entry.
The drops stop. The chain default-deny counter no longer climbs in step with the partner retry interval, and show log firewall no longer shows drops addressed to the internal service address.
A real transfer completes, not just a handshake. A publication that accepts the connection and then stalls has a second fault, and finding it now is cheaper than finding it during the partner batch run.
The negative test still fails as intended. From a source that has no business reaching the DMZ, the connection is still dropped — this is what distinguishes a corrected rule from a widened one.
The SMTP publication passes every check above, on its own evidence rather than by assumption.

Prevention

Treat a publication as one change with two halves that must agree about which address they name. The NAT rule names the public address; the filter rule names the private one. Write them together and review them together.
Make it a checklist line rather than folklore: a filter rule carrying a public destination address, in a chain evaluated after prerouting, is a defect on sight. That single check would have caught this in review.
Verify changes from state, not from configuration. Rule counters, conntrack entries and a real connection from outside are checks that can fail; reading the rule back and agreeing that it looks right is not.
Alert on default-deny drops whose destination is an internal service address. Very little else produces that pattern, and it is this fault announcing itself.
Keep drop on the WAN chain, but write down what it costs you: the partner sees a dead host rather than a refusal, and the first hour of every such incident goes to the server team. Give the service desk a documented way to tell the two apart.
When one change window produces two unreachable services, work them as one fault until the evidence separates them.

What if the publication had used port translation as well?

Say the public endpoint is 203.0.113.30:443 and the DMZ host listens on 8443, so the NAT rule carries a translation port too.

The failure mode is the same and the diagnosis gets one step harder, because now the filter rule can be wrong in two independent ways: it can name the pre-translation address, or the pre-translation port, or both. A rule that has been half-corrected — internal address, external port — still counts zero, and it looks much more convincing than the original because somebody clearly thought about it.

The check that cuts through it is the conntrack reply tuple, which shows the translated address and the translated port together. Whatever appears in that half of the entry is what the filter chain is matching against. Read both fields, correct both fields, and confirm on the counter rather than on the second read-through of the rule.

The general habit: when a translation has more than one component, the verification must exercise every component. A rule that is right about the address and wrong about the port fails exactly as silently as one that is wrong about both.

Why this matters

Nothing in this incident was broken. The NAT rule worked, the routing worked, the server worked, the firewall enforced the policy it was given, and the change passed a review by people who knew what they were looking at.

The failure was a mismatch between the address a human thinks about and the address the router is holding at the moment a rule is evaluated. Those are the same value for most of the packet’s life and different for the part that matters, and the difference is invisible in the configuration file — you can only see it in the packet path.

That is the durable lesson, and it is not really about NAT. A router is a pipeline, and every rule is written against a specific point in it. Rewriting, encapsulation, translation and marking all change what a later stage sees, so “this rule looks correct” is only meaningful once you have said correct at which stage. When a rule that reads perfectly has a counter of zero, the rule is not being evaluated against the packet you are imagining — and the counter, not the configuration, is what will tell you so.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. NAT happens first, and it changes the thing the rule matches on

2. The NAT counter is the trap, not the reassurance

3. The log recorded the truth under a name nobody searched for

Resolution

Verification

Prevention