Return-path — forward and reverse, asymmetric routing, return-path sanity

The return-path discipline is the operator’s commitment to verifying both directions of a flow. The operator who verifies only the forward path — source to destination — misses the failure on the reverse path, and the reverse path is where a large share of “the network is broken” reports actually live. A host sends packets fine and never sees the replies. An application connects and then transfers nothing.

This lesson is the fifth in Part LII: the discipline of return-path verification, the canonical diagnostic, the asymmetric-routing failure mode and what it really does on a Linux router, and the production workflow that converts a one-way verification into a bidirectional one.

Forward path vs reverse path

A bidirectional flow has two paths, and nothing guarantees they are the same:

sequenceDiagram
  participant Client
  participant Router1 as R1 (this router)
  participant Internet as Internet
  participant Server as Server

  Note over Client,Server: Forward path
  Client->>Router1: SYN (src=client, dst=server)
  Router1->>Internet: Forward via ISP-A
  Internet->>Server: SYN
  Note over Server,Client: Reverse path
  Server->>Internet: SYN-ACK (src=server, dst=client)
  Internet->>Router1: Arrives via ISP-B (different!)
  Router1->>Client: SYN-ACK

The forward path leaves through ISP-A. The reverse path comes back through ISP-B. That is asymmetric routing, and on the public Internet it is normal rather than exceptional — you control which way your packets leave, and the far side controls which way theirs come back.

Three failure modes follow:

1. One-way black hole. The forward path works, the reverse path does not. The host can send and cannot receive. TCP connections never establish; unidirectional UDP appears to work, which sends people looking in the wrong place.
2. Asymmetric routing plus filtering. Both paths physically work, but something on the reverse path rejects the packet because it arrived somewhere unexpected. This is the subtle one, and the next section is about it.
3. Return-path MTU. The reverse path has a smaller MTU than the forward path. Small packets pass, large ones vanish. The handshake succeeds and the transfer stalls.

The discipline: verify both directions before forming a hypothesis.

The return-path sanity check

flowchart TD
  S["Symptom: connections fail or one-way"]
  S --> Q1{Forward path works?}
  Q1 -->|no| Q2["Forward path broken<br/>fix forward first"]
  Q1 -->|yes| Q3{Reply reaches the router?}
  Q3 -->|no| Q4["Return path broken upstream<br/>investigate off-box"]
  Q3 -->|yes| Q5{Reply leaves the router?}
  Q5 -->|no| Q6["Dropped on this box:<br/>source-validation or ruleset"]
  Q5 -->|yes| Q7["Both directions cross R1<br/>look downstream"]

The ladder, in order:

1. Verify the forward path. ping 198.51.100.10 from the router, then traceroute 198.51.100.10 for the hops.
2. Find out whether the reply reaches the box at all. Capture on each uplink, not just the one you expect the reply on. This is the step that separates an upstream problem from a local one.
3. If the reply reaches the box but not the client, the drop is local. Then it is source-validation, a firewall rule, or NAT.

The capture, with concrete addresses rather than placeholders so it can be pasted and adapted:

sudo tcpdump -ni eth1 host 203.0.113.10 and host 198.51.100.10
sudo tcpdump -ni eth2 host 203.0.113.10 and host 198.51.100.10

Run both, in two sessions, while the connection is attempted. Forward packets appearing on eth1 and replies appearing on eth2 is the signature of asymmetry — and it also proves the reply exists, which rules out the upstream black hole immediately.

What actually drops an asymmetric reverse packet

The folk explanation is that the stateful firewall has no state for the reverse packet because the state was created on the other interface. On a Linux router that explanation is wrong, and believing it sends the operator to the wrong knob.

Conntrack is a property of the host, not of an interface. An entry created when the SYN left eth1 is found by the reply arriving on eth2; the tuple matches, the packet is classified ESTABLISHED, and conntrack is satisfied. Three other things drop it instead:

flowchart LR
  R["Reverse packet<br/>arrives on eth2"] --> RPF{"source-validation<br/>strict?"}
  RPF -->|drop| D1["Dropped before<br/>any rule runs"]
  RPF -->|pass| CH{"Which chain does<br/>this interface enter?"}
  CH -->|"per-interface chain<br/>without an established rule"| D2["Dropped by<br/>default-action"]
  CH -->|"chain accepts established"| OK["Forwarded"]

Reverse-path validation. With source-validation strict — RFC 3704 strict mode — the kernel drops a packet whose source address is not reachable back out of the interface it arrived on. Under asymmetric routing that is precisely the situation. The drop happens before the ruleset is evaluated, which is why no rule counter in show firewall attributes it to anything: no rule saw the packet.

configure
set firewall global-options source-validation loose
commit
save

loose requires only that the source be reachable by some route, which keeps most of the anti-spoofing value on a multi-homed edge. It can also be set per interface with set interfaces ethernet eth2 ip source-validation loose when only one uplink needs the relaxation.

A ruleset written per inbound interface. Most edge configurations jump to a different chain per uplink. If eth1 jumps to a chain that accepts established traffic and eth2 jumps to a chain that does not, the reply is dropped by default-action in a chain that was never written with return traffic in mind. Nothing about conntrack is at fault; the rule that would have matched was in the other chain.

configure
set firewall global-options state-policy established action accept
set firewall global-options state-policy related action accept
set firewall global-options state-policy invalid action drop
commit
save

The global state policy is evaluated for every ruleset, so it accepts return traffic regardless of which chain the packet entered. That is usually the right answer for a multi-homed edge, and it is a smaller change than relaxing an individual chain.

Two separate firewalls. When the forward and reverse paths cross different boxes, each has its own conntrack table, and the box that only ever sees the SYN-ACK genuinely has no entry for it. It classifies the packet INVALID and drops it. This is the case where the folk explanation is correct — and it is a topology problem, not a configuration one. No setting on either firewall fixes it, because neither box has the whole conversation. The fix is to make both directions cross the same box, or to accept a stateless posture for that traffic with the consequences that carries.

Return-path MTU

The reverse path may have a smaller MTU than the forward path — commonly because it traverses a tunnel that the forward path does not. Small packets pass in both directions, so the handshake succeeds; the first full-size data segment in the reverse direction disappears, and the transfer stalls with the connection still nominally open.

The diagnostic is a do-not-fragment ping from the far side, sized against the payload rather than the frame:

ping -M do -s 1472 203.0.113.10
ping -M do -s 1372 203.0.113.10

1472 bytes of payload plus 8 bytes of ICMP header plus 20 bytes of IPv4 header is 1500. If 1472 fails and 1372 succeeds, the path MTU is between the two and the usual answer is a tunnel at 1400.

On VyOS 1.5 the clamp is an interface property, not a firewall rule:

configure
set interfaces vti vti0 ip adjust-mss 1360
set interfaces vti vti0 ipv6 adjust-mss 1340
commit
save

1360 is 1400 minus 40 bytes of IPv4 and TCP header; the IPv6 figure subtracts 60. clamp-mss-to-pmtu in place of the number lets the kernel derive it from the interface MTU, which is the safer choice when the tunnel MTU is not fixed.

The bidirectional verification workflow

sequenceDiagram
  participant Operator
  participant R1 as R1 (this router)
  participant Peer as Peer

  Note over Operator: Step 1: forward path
  Operator->>R1: ping 198.51.100.10
  R1->>Peer: ICMP echo
  Peer-->>R1: ICMP reply
  Note over Operator: Step 2: does the reply reach R1?
  Operator->>R1: tcpdump on eth1 and on eth2
  R1-->>Operator: replies observed on eth2
  Note over Operator: Step 3: would R1 accept it?
  Operator->>R1: ip route get 203.0.113.10 from 198.51.100.10 iif eth2
  R1-->>Operator: resolves via eth1 — strict RPF would drop
  Note over Operator: Step 4: is there state, and is a rule counting?
  Operator->>R1: show conntrack table ipv4
  Operator->>R1: show firewall
  R1-->>Operator: entry present, no rule counting the drop

Steps 1 and 2 establish where the packet gets to. Step 3 predicts the source-validation verdict. Step 4 separates a ruleset drop from a silent one. Each step produces evidence the operator can put in the post-mortem.

Production failure modes

• Strict source validation on a multi-homed edge. Both paths work; strict RPF drops the reply because the best route back points out of the other uplink. No rule counts it. Fix: loose globally, or per interface where only one uplink needs it.
• Per-interface ruleset missing return traffic. The secondary uplink’s chain has no established rule. Fix: global state policy, or add the rule to that chain.
• Split firewall. Forward and reverse cross different boxes; the second sees only replies and calls them INVALID. Fix: topology.
• Return-path MTU smaller than forward. Handshake works, transfer stalls. Fix: ip adjust-mss on the interface that carries the smaller MTU.
• Return-path black hole. The reply never reaches the router. Fix: it is off-box; take the capture and the traceroute to the upstream operator rather than changing local configuration.
• ICMP filtered on the reverse path. TCP works, ping and traceroute do not, and the operator concludes the path is down when it is not. Fix: diagnose with TCP-based tools before believing an ICMP-only result.
• NAT asymmetry. Source NAT applies on the primary uplink’s address; the reply comes back to that address but arrives on the secondary interface. Conntrack still reverses the translation correctly — but if a rule matches on inbound interface before the state policy runs, the translated reply is dropped anyway. Fix: order the state policy ahead of the interface match.

Rollback

Return-path work is diagnosis first and change second. The discipline:

• Capture the diagnostic output for both directions, and keep it — it is the evidence that justifies the change.
• Name the drop — source-validation, a specific rule, a topology split, or MTU. “Asymmetric routing” is not a cause, it is a condition.
• Apply the fix with commit-confirm 5, because every fix here alters how traffic is filtered.
• Validate by re-running the same capture, not by asking whether it feels better.

Production discipline

Cross-course references

• Part LII-01 (LII-VyOS-Troubleshoot / define-and-scope) covers the incident-definition that precedes return-path verification.
• Part LII-02 (LII-VyOS-Troubleshoot / evidence-first) covers the evidence collection that precedes return-path verification.
• Part LII-03 (LII-VyOS-Troubleshoot / hypothesis-driven) covers the hypothesis formation that precedes return-path verification.
• Part LII-04 (LII-VyOS-Troubleshoot / subsystem-by-subsystem) covers the subsystem isolation that uses return-path verification.
• Part LI-04 (LI-VyOS-MTU / MSS clamping) covers ip adjust-mss in full.
• Part XXXVII (XXXVII-VyOS-Firewall) covers state policy and per-interface rulesets.

Quiz