Stateful Firewall Breaks Asymmetric Routing

Reported symptoms

The estate is the course reference topology: DC-East with two edge routers, R1-A on ISP-A and R1-B on ISP-B, both advertising the site’s own 203.0.113.0/24 and both carrying a full WAN-facing rule set. The office network is publicly addressed out of that /24 and is deliberately not NATed, which is normal for a BGP-multi-homed site and turns out to matter a great deal.

The ISP-B session on R1-B was turned up on Tuesday night. It came up clean, the prefix was accepted, the change was signed off, and nothing broke.

On Thursday the service desk had four tickets that nobody connected:

• A SaaS vendor’s portal times out from every office machine. Another vendor, linked from the same intranet page, loads instantly. The split follows no pattern anybody can describe.
• The nightly off-site backup to a partner endpoint has failed three nights running with a connection timeout. It has run for two years.
• Two engineers independently ran ping and traceroute to the failing destinations. Both succeeded, both times, with normal latency. Both tickets were closed as an application problem at the far end.
• The on-call engineer opened a security ticket, because R1-B had gone from a handful of default-deny log lines a minute to several thousand, all sourced from a large cloud provider. It reads exactly like a scan.

Monitoring agrees that everything is fine. The synthetic checks that reach those endpoints are ICMP, and they are green.

The one thing that does line up, and that nobody noticed for a day and a half, is that long-lived sessions to the affected destinations all died inside the same minute — and sessions to everything else were untouched.

Evidence provided

$ sudo conntrack -L -d 198.51.100.25 -p tcp

tcp 6 112 SYN_SENT src=203.0.113.61 dst=198.51.100.25 sport=51344 dport=443 [UNREPLIED] src=198.51.100.25 dst=203.0.113.61 sport=443 dport=51344 mark=0 use=1
conntrack v1.4.7 (conntrack-tools): 1 flow entries have been shown.

$ sudo conntrack -L -d 198.51.100.25 -p tcp

conntrack v1.4.7 (conntrack-tools): 0 flow entries have been shown.

$ sudo tcpdump -ni eth0 'host 198.51.100.25 and tcp port 443'

14:02:11.418823 IP 203.0.113.61.51344 > 198.51.100.25.443: Flags [S], seq 2948117, win 64240, length 0
14:02:12.441907 IP 203.0.113.61.51344 > 198.51.100.25.443: Flags [S], seq 2948117, win 64240, length 0
14:02:14.481930 IP 203.0.113.61.51344 > 198.51.100.25.443: Flags [S], seq 2948117, win 64240, length 0

$ sudo tcpdump -ni eth0 'host 198.51.100.25 and tcp port 443'

14:02:11.463102 IP 198.51.100.25.443 > 203.0.113.61.51344: Flags [S.], seq 771204, ack 2948118, win 65160, length 0
14:02:12.487551 IP 198.51.100.25.443 > 203.0.113.61.51344: Flags [S.], seq 771204, ack 2948118, win 65160, length 0

$ show log firewall | grep 203.0.113.61 | tail -2

$ show firewall

Rule     Action   Packets      Bytes   Description
----     ------   -------      -----   -----------
10       accept   4218        312K     Established/related
15       accept   88214       6.1M     ICMP (monitoring)
20       drop     31          2.4K     Invalid
30       accept   0           0        SSH from jump host
9999     drop     2841177     198M     Default deny

$ show ip bgp summary

Work the evidence before reading on

Four facts are in tension, and reconciling them is the whole exercise.

1. R1-A has a conntrack entry for the flow and it is [UNREPLIED]. R1-B has no entry for the same flow at all. Both routers are telling the truth. What does each one’s answer tell you about which packets it has seen?
2. The SYN leaves on one router’s uplink and the SYN/ACK arrives on the other’s, 45 milliseconds later. Neither packet is lost. Which device is deciding where the reply lands, and is it a device you own?
3. Rule 10 accepts established and related traffic and its counter is flat, while rule 9999 is doing millions of packets. What has to be true about the SYN/ACK for rule 10 not to match it?
4. ping works and TCP does not, to the same address, over the same path. Read rule 15 and rule 20 in the chain above and work out why. The ordering is the answer, and it is older than this incident.

Before continuing: which router created the state, and which router is being asked to use it?

Root cause

1. Connection state is per-router, and there are now two routers

nf_conntrack is a table in one kernel’s memory. When the office client sends its SYN, the packet leaves through R1-A, and R1-A writes an entry for that four-tuple. Nothing about that entry exists anywhere else. R1-B has never seen the flow.

When the SYN/ACK arrives at R1-B, the kernel there looks the four-tuple up, finds nothing, and hands the packet to the chain without an ESTABLISHED state. Rule 10 matches state established and state related, so it does not fire. Rules 30 and above are explicit permits for services the site publishes — none of them describe an inbound TCP segment from port 443 to an ephemeral port. The packet falls off the end of the chain and the default action drops it.

Every component behaved correctly. That is what makes this hard: there is no misconfiguration to find, and the four people who went looking for one spent a day and a half not finding it.

2. BGP made the return path somebody else’s decision

Both edge routers advertise 203.0.113.0/24. Before Tuesday, only R1-A advertised it, so every remote network on the internet had exactly one way back and it was symmetric by construction. From Tuesday there are two, and each remote network picks its own — by its own best-path selection, over which you have no direct control.

This is also why the failure is partial. Destinations whose operators prefer ISP-A stay symmetric and keep working. Destinations whose operators prefer ISP-B break. That distribution is stable, arbitrary from the office’s point of view, and looks exactly like “some vendors are having a bad week”.

The absence of NAT is what exposes the estate to this. A NATed office would carry R1-A’s own address on the wire, and the reply would be addressed to a host route inside ISP-A’s assignment — no choice for the remote network to get wrong. Public addressing plus two announcements is what puts the decision in someone else’s routing table.

3. ICMP survived, and that is why nobody believed the network was broken

Rule 15 accepts protocol icmp and carries no state clause. It was moved above the invalid drop years ago because a monitoring vendor complained about dropped latency probes, and it has worked perfectly ever since.

An echo reply arriving at R1-B for a request that left via R1-A belongs to no tracked flow. Rule 10 cannot match it. But rule 15 does not ask about state — it asks about protocol — so it matches, and the reply is admitted and forwarded to the client. Ping works end to end across a split path that TCP cannot cross.

Two engineers used that as proof the network was healthy, and it was a reasonable thing to believe. Rule 15 is the difference between a stateful check and a protocol check, and the page is testing whichever one it happens to use.

Resolution

1. Stop the security investigation. The drops on R1-B are your own return traffic, and every hour spent on the scan hypothesis is an hour the office is half offline. Say so in the channel explicitly, with the tcpdump from both uplinks as the evidence.
2. Decide whether you are restoring path symmetry or removing the dependency on it. These are different changes with different costs and the choice belongs to whoever owns the multi-homing design, not to the person holding the pager.
3. If restoring symmetry: make R1-B less attractive as a return path — AS-path prepending towards ISP-B, or the provider community that marks an announcement as backup — so both directions meet on R1-A again. Record what this costs: inbound load sharing is gone and ISP-B becomes a failover path rather than an active one.
4. Make the change with commit-confirm so a mistake on a WAN-facing router undoes itself rather than requiring a console. Confirm it only after the verification below has passed.
5. If holding rather than fixing: shut the ISP-B eBGP session or withdraw the prefix from it. That is one command, it returns the estate to the state it was in on Monday, and everyone understands the rollback. Give the hold a named owner and an end time, because a temporary shutdown of a redundant uplink is a quiet way to lose the redundancy permanently.
6. Do not relax the return chain to stateless to make the traffic flow. A rule admitting inbound TCP with no state clause admits every out-of-state segment anybody sends you, it will never be removed, and it converts an availability incident into a standing exposure.
7. Do not deploy conntrack state synchronisation expecting it to fix this. State replication exists so a standby can take over a live flow; the packet you need here arrives on the other router one round trip after the state was created, which is not a race replication is designed to win.
8. Once service is restored, open the design question properly: either steer both directions of a flow to one router, or build an edge that holds no per-flow state that only one router has.

Verification

1. The failing case completes. Open a TCP connection from an office client to a destination that was timing out and require a completed handshake — not a successful ping, which was green throughout and proved nothing.
2. The state looks right on R1-A. sudo conntrack -L shows that flow ESTABLISHED, with non-zero counters in both directions and the [ASSURED] flag, where before it sat in SYN_SENT marked [UNREPLIED].
3. The drops stop on R1-B. The catch-all deny counter stops climbing at the connection-attempt rate, and show log firewall stops showing TCP segments addressed to office client addresses.
4. The routing change actually took. Capture on R1-B uplink and confirm no return traffic for office prefixes arrives there at all. Read this from the wire, not from the configuration — the configuration was never the thing in doubt.
5. The negative test still passes. From an external position, an unsolicited TCP segment to an office client address must still be dropped. A fix that restores the traffic and also makes this test pass has restored service by removing the firewall.
6. The monitoring can now see the fault. Re-check that the default-deny drop rate on both edge routers is graphed and alerting, and confirm the alert would have fired on Tuesday night.

Prevention

• Ask the return-path question before every change that adds a second way into the estate — a second uplink, a second announcement, a new peering, an ECMP path. Does any stateful device now sit on one direction of a flow and not the other? Put it in the multi-homing turn-up runbook as a gate with a signature, not as a note in the appendix.
• Alert on the per-router default-deny drop rate. It is the cheapest signal this fault produces and it moved two nights before the first ticket. Route the alert to the network team: it will look like a scan every single time, and sending it to the security queue guarantees a day of investigating the wrong thing.
• Give every rule an explicit state clause, ICMP included. The stateless ICMP accept did not cause this incident, but it is why four people concluded the network was healthy.
• Do not let ICMP be the only synthetic check on a path that carries TCP. A probe that opens a real connection would have failed on Tuesday night and named the destination.
• Write the assumption down. “Forward and return traverse the same edge router” was true for years, is load-bearing, and stopped being true during a change that was reviewed, executed cleanly and signed off as successful. Unwritten invariants are only discovered when they break.