Lab: Evidence-Based Routing Troubleshooting

Objective

This lab has one symptom and four causes. Every time, the ticket says the same thing — host A cannot reach host B — and every time, most of the commands you would run on the routers return identical, healthy output.

That is not a contrived arrangement. It is the ordinary condition of a routing incident, and it is why Part LII spends six lessons on method rather than on commands. The interfaces are up in all four cases. The OSPF adjacencies are Full in all four cases. The routing table is correct in all four cases. Three of the four faults are invisible to show ip route, and one of them is invisible to every rule counter on the box that is dropping the packet.

By the end you will have a decision card that separates all four with a packet capture and a counter reading, in roughly a minute — and, more importantly, a worked example of why the ladder has to be climbed in order. Skipping a rung does not make you faster; it makes you confident about the wrong subsystem.

Architecture

Three VyOS routers and two VyOS hosts. The estate is deliberately built so the forward and return paths differ, because that is the condition under which three of the four faults exist at all.

   host-a                                                 host-b
198.51.100.10                                        198.51.100.130
      |                                                     |
   LAN-A 198.51.100.0/25                LAN-B 198.51.100.128/25
      |                                                     |
  +---------+    203.0.113.0/30   +------+  203.0.113.4/30  +---------+
  |   r1    |eth2 ------------ eth1|  r2  |eth2 ------------ eth2|  r3  |
  | eth1 .1 |     cost 10          |      |      cost 10         | eth1 .129
  |         |                      +------+                      |         |
  |  eth3   |                                                    |  eth3   |
  |   .9    +------------- 203.0.113.8/30 ---------------------+  .10      |
  +---------+          r1 cost 100 / r3 cost 10                +---------+

Every address is from a range RFC 5737 reserves for documentation.

The OSPF costs are the design. On r1, the direct link to r3 costs 100 and the link to r2 costs 10, so r1 reaches LAN-B through r2. On r3, the link to r2 costs 100 and the direct link to r1 costs 10, so r3 reaches LAN-A directly. Traffic from A to B crosses three routers; the reply crosses two, on a different segment.

Nobody built it that way on purpose in the story. Somebody set a cost during a maintenance window two years ago and it was never revisited, which is how most production asymmetry happens.

Requirements

• A hypervisor with roughly 5 GiB of free RAM and 40 GiB of free disk — five VMs at 1 GiB and 8 GiB each.
• The VyOS 1.5 LTS ISO. The firewall ipv4 forward filter shape, action jump, global-options source-validation and ip adjust-mss are all 1.4-and- later constructs; none of them exists in the 1.2 form you may find in older notes.
• Five isolated layer-2 bridges with no physical port: MGMT, LAN-A, LAN-B, and the three transit segments — six bridges in total if you count them properly, and counting them properly is a good habit here.
• Two terminal sessions per router, because the central technique in this lab is capturing on two interfaces simultaneously.
• Console access to all five VMs. Task 6 installs a default-drop forward filter on r1.
• Roughly two and a quarter hours.

Scenario

A ticket arrives: “the network is broken, users in Site A cannot get to the application in Site B”. It has been open for forty minutes and has already collected two comments — one saying the firewall was changed last week, and one saying it is probably DNS.

You have three routers, two of which you did not build. Nobody has run a packet capture. Somebody has offered to reboot r1.

Tasks

Configuration blocks are written for the [edit] prompt. Where a block opens with configure and you are already at [edit], skip that line. Blocks tagged bash are the operational-mode shell. Blocks made up of show commands are operational mode; from inside configure, prefix each one with run.

Each fault task follows the same four steps: define and scope, capture evidence, form and test one hypothesis, then fix and validate. Do them in that order even when the answer is obvious to you, because the deliverable is the procedure.

Task 1 — Build the estate, and prove the paths differ while it works

r1:

configure
set system host-name r1
set interfaces ethernet eth0 address 192.0.2.11/24
set interfaces ethernet eth1 address 198.51.100.1/25
set interfaces ethernet eth1 description 'LAN-A'
set interfaces ethernet eth2 address 203.0.113.1/30
set interfaces ethernet eth2 description 'to r2'
set interfaces ethernet eth3 address 203.0.113.9/30
set interfaces ethernet eth3 description 'direct to r3'
set protocols ospf parameters router-id 203.0.113.1
set protocols ospf area 0 network 198.51.100.0/25
set protocols ospf area 0 network 203.0.113.0/30
set protocols ospf area 0 network 203.0.113.8/30
set protocols ospf interface eth1 passive
set protocols ospf interface eth2 cost 10
set protocols ospf interface eth3 cost 100
commit
save

r2:

configure
set system host-name r2
set interfaces ethernet eth0 address 192.0.2.12/24
set interfaces ethernet eth1 address 203.0.113.2/30
set interfaces ethernet eth2 address 203.0.113.5/30
set protocols ospf parameters router-id 203.0.113.2
set protocols ospf area 0 network 203.0.113.0/30
set protocols ospf area 0 network 203.0.113.4/30
set protocols ospf interface eth1 cost 10
set protocols ospf interface eth2 cost 10
commit
save

r3:

configure
set system host-name r3
set interfaces ethernet eth0 address 192.0.2.13/24
set interfaces ethernet eth1 address 198.51.100.129/25
set interfaces ethernet eth1 description 'LAN-B'
set interfaces ethernet eth2 address 203.0.113.6/30
set interfaces ethernet eth2 description 'to r2'
set interfaces ethernet eth3 address 203.0.113.10/30
set interfaces ethernet eth3 description 'direct to r1'
set protocols ospf parameters router-id 203.0.113.6
set protocols ospf area 0 network 198.51.100.128/25
set protocols ospf area 0 network 203.0.113.4/30
set protocols ospf area 0 network 203.0.113.8/30
set protocols ospf interface eth1 passive
set protocols ospf interface eth2 cost 100
set protocols ospf interface eth3 cost 10
commit
save

host-a and host-b are plain hosts with a default route:

configure
set system host-name host-a
set interfaces ethernet eth0 address 192.0.2.14/24
set interfaces ethernet eth1 address 198.51.100.10/25
set protocols static route 0.0.0.0/0 next-hop 198.51.100.1
set service ssh port 22
commit
save

configure
set system host-name host-b
set interfaces ethernet eth0 address 192.0.2.15/24
set interfaces ethernet eth1 address 198.51.100.130/25
set protocols static route 0.0.0.0/0 next-hop 198.51.100.129
set service ssh port 22
commit
save

Now the baseline that everything else is read against, and it is not a ping. It is a simultaneous capture on both of r1’s uplinks, taken while the service is completely healthy.

Open two sessions on r1:

# Session 1 on r1.
sudo tcpdump -ni eth2 host 198.51.100.10 and host 198.51.100.130

# Session 2 on r1.
sudo tcpdump -ni eth3 host 198.51.100.10 and host 198.51.100.130

Then, from host-a:

$ ping 198.51.100.130 count 3

PING 198.51.100.130 (198.51.100.130) 56(84) bytes of data.
64 bytes from 198.51.100.130: icmp_seq=1 ttl=61 time=1.44 ms
64 bytes from 198.51.100.130: icmp_seq=2 ttl=61 time=1.11 ms
64 bytes from 198.51.100.130: icmp_seq=3 ttl=61 time=1.19 ms

Read the two captures together. Echo requests leave on eth2, toward r2. Echo replies arrive on eth3, from r3. Different interfaces, different routers, same conversation.

Confirm it from the routing table rather than the wire, so you have both:

show ip route 198.51.100.128/25
show ip ospf interface eth2
show ip ospf interface eth3

Write it into the journal as a statement, because it is the fact the rest of the lab keeps returning to: this path is asymmetric and the service works. Asymmetry is a condition, not a cause. On the public Internet it is normal rather than exceptional — you choose which way your packets leave and the far side chooses which way theirs come back — and an incident report that names “asymmetric routing” as the root cause has stopped one step short of the actual one.

Task 2 — Fault 1: the reply that never leaves the far side

Inject it on host-b:

configure
delete protocols static route 0.0.0.0/0
commit
save
exit

Now work it as an incident, in order.

Define and scope. From host-a, ping 198.51.100.130 fails. Convert the complaint into a symptom and answer the three scope questions from Part LII-01 before doing anything else:

• Who is affected? Test from a second address on LAN-A if you have one, and from r1 itself. Everything on LAN-A, or one host?
• What is affected? Connectivity to one destination, or to everything beyond r1? Try ping 203.0.113.6 from host-a — a router interface on the far side of r2.
• When did it start? show system commit on each router, and the commit timestamps against the ticket time.

Those three answers already eliminate whole subsystems. If everything on LAN-A fails to reach exactly one destination and nothing else, r1’s LAN interface, host-a’s addressing and the LAN switch are all out of scope, and you have not run a single diagnostic yet.

Evidence, before any hypothesis. Capture the Part LII-02 checklist into a dated file. The shape matters as much as the content: each section names the command that produced it and ends with the one-line conclusion you drew, because in an hour “state Full” will still be legible and “and therefore OSPF is not the problem” will not be obvious.

# On r1, operational mode.
J="$HOME/lab24"
mkdir -p "$J"
E="$J/fault1-evidence.txt"

{
  echo "# fault 1 evidence -- r1"
  echo "# captured: $(date -Iseconds)"
  echo
  echo "== show configuration commands | match ospf"
  vtysh -c 'show running-config ospf'
  echo
  echo "== show ip ospf neighbor"
  vtysh -c 'show ip ospf neighbor'
  echo
  echo "== show ip route 198.51.100.128/25"
  vtysh -c 'show ip route 198.51.100.128/25'
  echo
  echo "== ip -s link show eth2"
  ip -s link show eth2
} | tee "$E"

Every one of those comes back healthy. That is the finding, not a dead end: the subsystems Part LII-04 walks — interface, kernel, FRR — are all clean on r1, which is what makes the next step the right one instead of a guess.

The ladder. Re-run the two-interface capture from Task 1 while host-a pings. Requests leave on eth2. Nothing comes back on eth3, or on eth2, or anywhere.

Now go one hop further out. On r3:

# On r3, operational mode.
sudo tcpdump -ni eth1 host 198.51.100.10 and host 198.51.100.130

The echo request arrives on LAN-B. No reply follows it.

That is the whole diagnosis: the reply is not being generated, or is not being routed, on the far side. r1, r2 and r3 are all doing exactly what they were asked to. The remaining work is on host-b — show ip route there shows no default route, and a host with no route to the source cannot answer an echo request no matter how healthy the network is.

Fix and validate.

configure
set protocols static route 0.0.0.0/0 next-hop 198.51.100.129
commit
save
exit

Re-run the same capture, not a different one. Part LII-06 calls the alternative change-and-pray: the change is made, the symptom is assumed to be gone, and nobody re-runs the diagnostic that identified it.

Task 3 — Fault 2: the reply that arrives and dies with nothing counting it

Restore the estate to health, confirm it, then inject on r1:

configure
set firewall global-options source-validation strict
commit
save
exit

Define and scope. Identical complaint, identical scope answers. Note that in your journal — the scoping step did not distinguish these two faults, and it was still worth doing, because it eliminated the same subsystems both times.

The ladder, rung by rung. Forward path first:

ping 198.51.100.130 count 3
traceroute 198.51.100.130

From r1 itself the destination answers. From host-a it does not. That gap is the finding: transit traffic is being treated differently from traffic the router originates, which points at forwarding, filtering or source validation — and rules out addressing and adjacency in one line.

Now the capture, on both uplinks at once, while host-a pings.

$ sudo tcpdump -ni eth3 host 198.51.100.10 and host 198.51.100.130

listening on eth3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
11:42:03.118442 IP 198.51.100.130 > 198.51.100.10: ICMP echo reply, id 41, seq 1, length 64
11:42:04.119901 IP 198.51.100.130 > 198.51.100.10: ICMP echo reply, id 41, seq 2, length 64

The reply reaches r1 and never reaches host-a. The drop is local. That is rung three, and it is where the ladder stops being generic and starts being about this router.

Now the counters — and the absence of movement is the evidence.

show firewall
show conntrack table ipv4

There is a conntrack entry for the flow, in ESTABLISHED, and no rule anywhere counts a drop. Read that pair carefully, because it is the single most misinterpreted state in this subject: state exists, the packet was observed, and nothing in the ruleset saw it.

A packet that is dropped before the ruleset is evaluated is dropped by something that runs earlier. Predict the verdict:

$ ip route get 198.51.100.10 from 198.51.100.130 iif eth3

RTNETLINK answers: Invalid cross-device link

With source-validation strict — RFC 3704 strict mode — the kernel drops a packet whose source address is not reachable back out of the interface it arrived on. r1’s best route to LAN-B goes out eth2, through r2. The reply arrives on eth3. Under symmetric routing that never happens; under the asymmetry you proved in Task 1, it happens to every single reply.

Fix, and choose the smallest one.

configure
set firewall global-options source-validation loose
compare
commit-confirm 5
exit

loose requires only that the source be reachable by some route, which keeps most of the anti-spoofing value on a multi-homed edge. It can also be applied to one interface — set interfaces ethernet eth3 ip source-validation loose — when only the asymmetric uplink needs the relaxation, and that is the smaller change of the two.

Validate by re-running the capture and the ping, then confirm and save.

Task 4 — Fault 3: the reply that arrives and a rule counts the drop

Restore health, confirm it, then build the ruleset a real edge router has — per-uplink chains — with the mistake a real edge router makes.

configure
set firewall ipv4 name FROM-R2 rule 10 action accept
set firewall ipv4 name FROM-R2 rule 10 state established
set firewall ipv4 name FROM-R2 rule 20 action accept
set firewall ipv4 name FROM-R2 rule 20 state related
set firewall ipv4 name FROM-R2 default-action drop

set firewall ipv4 name FROM-R3 rule 10 action accept
set firewall ipv4 name FROM-R3 rule 10 protocol tcp
set firewall ipv4 name FROM-R3 default-action drop

set firewall ipv4 forward filter default-action accept
set firewall ipv4 forward filter rule 10 action jump
set firewall ipv4 forward filter rule 10 inbound-interface name eth2
set firewall ipv4 forward filter rule 10 jump-target FROM-R2
set firewall ipv4 forward filter rule 20 action jump
set firewall ipv4 forward filter rule 20 inbound-interface name eth3
set firewall ipv4 forward filter rule 20 jump-target FROM-R3
commit
save
exit

Read what has been built before you run anything. Two chains, one per uplink. FROM-R2 accepts established and related traffic; FROM-R3 accepts TCP and drops everything else. Both look like reasonable rulesets, and a configuration review would pass them, because nothing in either one is obviously wrong.

The ladder. Same complaint, same scope, same capture. The reply appears on eth3 exactly as in Task 3. So rungs one to three give an identical result, and the fourth rung is where the two faults separate:

$ show firewall ipv4 name FROM-R3

Ruleset Information

ipv4 Firewall "FROM-R3"

Rule     Action   Protocol   Packets   Bytes   Conditions
-------  -------  ---------  --------  ------  ----------
10       accept   tcp        0         0
default  drop     all        6         504

Six packets on the default action. That is the difference between this fault and the last one, and it is the whole content of the fourth rung: a drop that a rule counts is a ruleset problem; a drop that no rule counts is not.

Read the counter as a delta rather than as an absolute. Note the number, run the ping again, note it again. A counter that moved during your test belongs to your test; a counter that was already at six from something else last Tuesday does not.

The mechanism has nothing to do with conntrack. eth2 jumps to a chain that accepts established traffic; eth3 jumps to a chain that does not. The reply is dropped by a default-action in a chain nobody wrote with return traffic in mind. Under symmetric routing the reply would have arrived on eth2 and matched FROM-R2 rule 10, and this ruleset would have been correct for years.

Fix. The lesson’s answer is the global state policy, and it is smaller than editing the chain:

configure
set firewall global-options state-policy established action accept
set firewall global-options state-policy related action accept
set firewall global-options state-policy invalid action drop
compare
commit-confirm 5
exit

The global state policy is evaluated for every ruleset, so return traffic is accepted regardless of which chain the packet entered. Validate, confirm, save — and then re-read show firewall ipv4 name FROM-R3 and confirm the default-action counter has stopped advancing.

Task 5 — Fault 4: the handshake that works and the transfer that stalls

Restore health, confirm it, then shrink the return path and blind the mechanism that would normally repair it.

On r1 and on r3, on the direct link:

configure
set interfaces ethernet eth3 mtu 1400
commit
save
exit

On r3 only:

configure
set firewall ipv4 output filter default-action accept
set firewall ipv4 output filter rule 10 action drop
set firewall ipv4 output filter rule 10 description 'legacy ICMP suppression, ticket long closed'
set firewall ipv4 output filter rule 10 protocol icmp
set firewall ipv4 output filter rule 10 destination address 198.51.100.128/25
commit
save
exit

Define and scope — and this time the answer is different. The complaint is still “cannot reach”. The symptom is not.

$ ping 198.51.100.130 count 3

PING 198.51.100.130 (198.51.100.130) 56(84) bytes of data.
64 bytes from 198.51.100.130: icmp_seq=1 ttl=61 time=1.31 ms
64 bytes from 198.51.100.130: icmp_seq=2 ttl=61 time=1.08 ms
64 bytes from 198.51.100.130: icmp_seq=3 ttl=61 time=1.22 ms

Ping passes, so half the people on the call conclude the network is fine. Now move some data:

# On host-b, make something worth transferring.
dd if=/dev/urandom of=/tmp/blob bs=1M count=20

# On host-a. Substitute your own credentials when prompted:
scp vyos@198.51.100.130:/tmp/blob /tmp/blob

The connection establishes, the transfer reports a percentage, and then it stops. That is a different symptom from the previous three and it deserves a different sentence in the ticket: TCP connections to 198.51.100.130 establish and then stall; ICMP echo succeeds. Part LII-01’s whole point is that this sentence is worth five minutes of anybody’s time, because it eliminates “no route” and “firewall drops the flow” before either is investigated.

The ladder. Both directions still work at small sizes — you just proved that with ping. The reply reaches the router; the capture shows it. No rule counter moves for the small packets. Rungs one to four all say healthy.

The distinguishing test is a do-not-fragment ping from the far side, sized against the payload rather than the frame:

ping 198.51.100.10 count 3 size 1350 do-not-fragment
ping 198.51.100.10 count 3 size 1450 do-not-fragment

Run those on host-b. 1350 bytes of payload plus 8 bytes of ICMP header plus 20 bytes of IPv4 header is 1378, which fits. 1450 makes 1478, which does not fit the 1400-byte hop and comes back as nothing at all — because the router that should be telling host-b so is dropping its own ICMP.

That pair of results locates the path MTU between 1378 and 1478 and, just as importantly, proves that Path MTU Discovery is not working. On a healthy path this fault repairs itself: r3 would send an ICMP fragmentation-needed message back to host-b, host-b would shrink its segments, and nobody would ever file a ticket. The stall exists only because that message is being discarded, which is the production condition — somebody blocked ICMP years ago for a reason that is no longer in anybody’s memory.

Confirm the second half rather than inferring it:

show firewall ipv4 output filter

Fix — and there are two, at different layers. Unblocking the ICMP restores Path MTU Discovery and is the correct answer. Clamping the MSS makes the path work even when PMTUD is broken somewhere you do not control, which is why production carries it anyway:

configure
delete firewall ipv4 output filter rule 10
set interfaces ethernet eth3 ip adjust-mss 1360
compare
commit-confirm 5
exit

1360 is 1400 minus 20 bytes of IPv4 header and 20 bytes of TCP header; clamp-mss-to-pmtu in place of the number lets the kernel derive it from the interface MTU, which is safer when the tunnel MTU is not fixed. Apply the same clamp on r1’s eth3.

Validate with a new connection. Clamping rewrites the MSS on TCP SYN packets, so it affects only connections established after the change; the stalled scp will not recover until it is restarted, and testing with the broken session is how people conclude the fix did not work.

Task 6 — Write the decision card, and run it against the clock

You have now produced four incidents that were reported identically and that gave identical output from almost every command available on the router.

Write the card. It should fit on one side of paper and it should be readable at three in the morning by somebody who did not build the network:

Two commands separate the first four rows: a simultaneous capture on every uplink, and a rule-counter reading before and after. The fifth needs one more — a do-not-fragment ping from the far side, sized against the payload.

Now run it. Re-inject one of the four faults, start a timer, and climb the ladder from the top without skipping. Record the time.

Close the lab by answering the question the card exists to make answerable: for which of the four faults did show interfaces, show ip route and show ip ospf neighbor on all three routers return identical, healthy output?

The answer is all four of them. That is the reason the ladder starts with a packet capture rather than with a show command, and it is the single most useful thing to carry out of this lab.

Validation

• The Task 1 baseline capture shows requests leaving r1 on eth2 and replies arriving on eth3, while ping from host-a succeeds.
• A symptom statement and a three-question scope exist in the journal for each of the four faults, written before any hypothesis.
• Four evidence files exist, each in the Part LII-02 shape: every section names the command that produced it and ends with a conclusion.
• For fault 1, a capture on r3’s LAN interface shows the echo request arriving and no reply following it.
• For fault 2, a capture shows the reply on r1 eth3, show firewall shows no counter moving, and ip route get with iif eth3 returns an error.
• For fault 3, the same capture is accompanied by a FROM-R3 default-action counter that advanced during the test, recorded before and after.
• For fault 4, ping succeeds while a TCP transfer stalls, and a do-not-fragment ping from host-b fails at 1450 bytes and succeeds at 1350.
• Each fix was applied with commit-confirm, validated by re-running the diagnostic that found the fault, and only then confirmed and saved.
• The decision card exists, fits on one page, and has a measured run time with an honest note about what that measurement is worth.

Expected Outcome

A three-router estate with genuinely asymmetric paths, four reproducible faults that present as one complaint, and a written procedure that separates them with a capture and a counter.

The specific thing you should be able to state afterwards is which evidence is load-bearing. show ip route was correct in all four cases and told you nothing. show ip ospf neighbor was Full in all four cases and told you nothing. The capture told you whether the reply existed and where it got to; the counter told you whether anything on the box admitted to dropping it. Everything else in the investigation was context.

Troubleshooting

The paths are symmetric — replies arrive on eth2. The OSPF costs did not take. Check show ip ospf interface eth3 on r1 and show ip ospf interface eth2 on r3; both should read cost 100. A cost set on the wrong router produces a working, symmetric network and three of the four faults will not reproduce.

ping from host-a fails before you have injected anything. Work outward in dependency order. show ip ospf neighbor on all three routers first — all adjacencies Full — then show ip route 198.51.100.128/25 on r1 and show ip route 198.51.100.0/25 on r3. A missing network statement for one of the transit /30s is the usual answer, and it produces a route that exists on one router and not the other.

tcpdump shows nothing at all on either interface. Check the filter. host 198.51.100.10 and host 198.51.100.130 matches only that pair; if you are pinging a router interface instead of host-b, nothing matches. Drop the filter entirely to confirm the interface is live, then put it back.

In Task 3, ip route get succeeds instead of returning an error. source-validation strict did not commit, or the iif names the wrong interface. show configuration commands | match source-validation settles the first; the second is whichever interface the capture actually saw the reply on.

In Task 4, no counter moves anywhere and the ping still fails. The named rulesets exist but nothing jumps to them, or the jump names the wrong inbound interface. show configuration commands | match jump-target lists the jumps — a named ruleset with no jump is valid, visible in show configuration, and completely inert.

In Task 5 the transfer completes normally. Path MTU Discovery is working, which means r3’s output filter is not dropping the fragmentation-needed message. Check show firewall ipv4 output filter for a rule counter that is advancing, and confirm the destination address on the rule covers LAN-B.

In Task 5 the do-not-fragment ping fails at both sizes. The size is being measured against the wrong hop, or eth3’s MTU was set on only one of the two routers. A link with mismatched MTUs on its two ends misbehaves differently from one that is uniformly small; set both.

After the MSS clamp, the transfer still stalls. You are testing with the connection that was already broken. Clamping rewrites the MSS on the SYN, so it applies to new connections only. Kill the scp and start a fresh one.

Cleanup

Step 1. Remove every injected fault. Order matters in exactly one place: take the jump rules out before the named rulesets they target, or the commit fails on a dangling reference.

On r1:

configure
delete firewall ipv4 forward filter
delete firewall ipv4 name FROM-R2
delete firewall ipv4 name FROM-R3
delete firewall global-options source-validation
delete interfaces ethernet eth3 mtu
delete interfaces ethernet eth3 ip adjust-mss
compare
commit
save
exit

On r3:

configure
delete firewall ipv4 output filter
delete interfaces ethernet eth3 mtu
delete interfaces ethernet eth3 ip adjust-mss
compare
commit
save
exit

Step 2. Decide deliberately what to do with the global state policy. Unlike the other changes it is not a fault — it is the fix from Task 4, and on a multi-homed edge it is the right default. If you keep it, note in the journal that it is there and why, because a state policy that nobody remembers configuring is the thing that makes the next firewall investigation confusing.

Step 3. Confirm the estate is back to the Task 1 baseline, using the Task 1 method rather than a ping:

# Two sessions on r1, while host-a pings.
sudo tcpdump -ni eth2 host 198.51.100.10 and host 198.51.100.130
sudo tcpdump -ni eth3 host 198.51.100.10 and host 198.51.100.130

Requests on eth2, replies on eth3, ping succeeding. That is the baseline, asymmetry included.

Step 4. Remove the temporary file on host-b:

# On host-b, operational mode.
rm -f /tmp/blob

Step 5. Keep the journal, the four evidence files and the decision card. If you are keeping the topology for the routing break/fix scenarios, save on every router and record in the journal that the paths are deliberately asymmetric — the next person to open it will otherwise spend their first twenty minutes proving the same thing you proved in Task 1.

Step 6. To remove the topology, delete the five VMs and the six bridges. Confirm with ip -brief link show on the host that no lab bridge survives.

What You Learned

• A complaint is not a symptom, and converting one into the other eliminates subsystems for free. “Cannot reach” covered four different faults; “TCP establishes and stalls while ICMP succeeds” covered exactly one.
• Asymmetry is a condition, not a cause. You proved the paths differed while the service was healthy. An incident report that stops at “asymmetric routing” has stopped one step short of the thing that actually dropped the packet.
• Capture on every uplink, not the one you expect. It is the rung that separates an off-box problem from a local one, it costs nothing, and it is the step that gets skipped because testing the return path needs an interface nobody suspected.
• A drop that no rule counts is not a firewall problem. Reverse-path validation runs before the ruleset, so the evidence is a counter that stays at zero while the packet is visible on the wire. Relaxing the firewall would have left a weaker router and the same outage.
• Conntrack is a property of the host, not of an interface. The entry created on one uplink is found by a reply arriving on another. The thing that breaks under asymmetry is a ruleset written per inbound interface, and the fix is the global state policy rather than an accept rule.
• Read counters as deltas. A number is not evidence; a number that moved during your test is.
• Change one thing, then re-run the diagnostic that found the fault. Not a different one, and not a general “is it better now”. That is the difference between a validated fix and change-and-pray.
• Every show command on every router was healthy in all four faults. That is not a quirk of the lab. It is why the method starts where it does.