Network troubleshooting methodology - the systematic approach · Linux

Network problems are uniquely stressful because every other system depends on the network, and the failure surface is large. A systematic methodology - “go down the layers, gather evidence at each, form a hypothesis” - is what separates a five-minute diagnosis from a half-day exercise.

The loop

1. Define the symptom
   What exactly is failing? Who reports it? Since when?
2. Isolate the layer
   Application? Transport? Network? Link? Physical?
3. Gather evidence
   ip, ss, ping, tcpdump, dig, ethtool
4. Form a hypothesis
   "I think the gateway's ARP is broken because..."
5. Test the hypothesis
   One change. Verify. Repeat.
6. Restore service
7. Find root cause and prevent recurrence

The loop is the same regardless of the layer. The tools change.

Start with the symptom

“I cannot reach the database” is not a symptom; it is a description. The actual symptom is one of:

psql returns “Connection timed out”.
psql returns “Connection refused”.
psql returns “No route to host”.
DNS for the database hostname fails.
TLS handshake fails.
The connection works but queries fail.

Each maps to a different layer:

Error	What it means
Connection timed out	Something is dropping packets silently - usually a firewall, sometimes a black-holed route
Connection refused	The packet arrived and nothing accepted it: no listener on that address, or a `REJECT` rule
No route to host (`EHOSTUNREACH`)	Neighbour resolution failed for an on-link destination, or a firewall rejected with `icmp-host-unreachable`. Not usually routing - see below
Network is unreachable (`ENETUNREACH`)	Genuinely no matching route. This is the layer 3 error
Name or service not known	Resolution, layer 7

No route to host is the one that misleads. Its wording says routing, but the kernel returns it when ARP or ND fails for a destination on a directly connected subnet - ip route get answers perfectly while ip neigh show reports FAILED. A genuinely missing route reports ENETUNREACH instead. Read linux-arp-and-neighbor-discovery for the full separation.

The ladder

One named order, referenced by the end-to-end connectivity lab and by the network incident runbook so that all three teach the same procedure. It is sorted by cost of evidence - cheapest and most decisive first. Stop at the first rung that gives a definite answer.

THE LADDER

1. LISTENER      ss -tlnp '( sport = :PORT )'
                 Bound to 0.0.0.0/[::], or only to 127.0.0.1?
                 A loopback bind is the #1 cause of
                 "works locally, not remotely".

2. LOCAL ROUTE   ip route get DST
                 Answered by the kernel FIB on this host, so
                 nothing can filter it. Check the interface
                 and the src address, not just "a route exists".

3. LINK          ip -br link ; ip -br addr ; ip neigh show DST
                 ethtool -S IFACE | grep -E 'err|drop'

4. HOST FIREWALL sudo nft list ruleset | grep -n PORT
                 sudo tcpdump -i any -nn "tcp port PORT and \
                   tcp[tcpflags] & tcp-syn != 0"
                 SYN in, nothing out  -> local DROP
                 SYN in, RST out      -> REJECT, or no listener
                 no SYN at all        -> never arrived; rung 5

5. PATH MTU      tracepath -n HOST
                 ping -M do -s 1472 HOST
                 The rung everyone skips. Handshake fine,
                 large transfers hang -> you are here.

6. PATH          traceroute -T -p PORT HOST

7. REMOTE END    repeat rungs 1 and 4 on the far host

8. DNS           dig +short NAME ; getent hosts NAME
                 Last, not first. "It must be DNS" is a
                 conclusion you reach after an IP-addressed
                 connection is proven to work.

The commands, on a real host:

PORT=5432
HOST=db.internal.example.com
DST=10.0.0.20

# 1. Is anything listening, and on which address?
ss -tlnp "( sport = :$PORT )"
# 2. What will the kernel do with the packet?
ip route get "$DST"
# 3. Link and neighbour
ip -br link show; ip neigh show "$DST"
# 4. Does the port answer, and is the host firewall eating it?
nc -vz "$HOST" "$PORT"
sudo nft list ruleset | grep -n "$PORT"
# 5. Path MTU - the step people skip
tracepath -n "$HOST"
ping -M do -s 1472 -c3 "$HOST"
# 6. Path, on the port you actually need
sudo traceroute -T -p "$PORT" "$HOST"
# 8. Only now, DNS
dig +short "$HOST"; getent hosts "$HOST"

Interpretation:

nc succeeds but the application fails: layer 7 - auth, TLS, or the application itself.
nc succeeds, small requests work, larger responses hang or stall part-way: suspect MTU / PMTUD before you suspect the application. See linux-mtu-and-jumbo-frames and linux-icmp-and-path-mtu.
ping fails but nc on the service port succeeds: ICMP is filtered. That is a firewall policy fact, not a fault.
ip link shows DOWN: layer 1.
Conclude “routing” only when ip route get returns no route, or returns the wrong source address or outgoing interface.

A ping failure proves nothing on its own. ICMP echo is filtered by policy in most cloud VPCs, in most enterprise DMZs and by plenty of host firewalls, so a silent ping is at least as likely to be policy as breakage. Test routing with ip route get <ip>: it is answered by the kernel FIB on this host, so no firewall anywhere can filter it. Test reachability with nc -vz <ip> <port> or traceroute -T -p <port>, which probe the service you actually care about. Only conclude layer 3 when ip route get returns no route, or returns the wrong source address or the wrong outgoing interface.

“It works on the box but not from anywhere else”

This is the single most common shape of a “service is up but unreachable” ticket, and it has only two common causes. Both are diagnosed on the server, in three commands, before you involve the network team.

1. Which address is it listening on?

ss -tlnp '( sport = :8080 )'

Read the local-address column, not just the port:

LISTEN 0 4096   127.0.0.1:8080   users:(("api",pid=911,fd=6))  # this host only
LISTEN 0 4096     0.0.0.0:8080   users:(("api",pid=911,fd=6))  # every IPv4 address
LISTEN 0 4096        [::]:8080   users:(("api",pid=911,fd=6))  # every address
LISTEN 0 4096   10.0.0.10:8080   users:(("api",pid=911,fd=6))  # that one address

grep :8080 matches all four lines equally, so an operator who greps for the port concludes “it is listening” and escalates. A 127.0.0.1 bind is the answer: curl localhost:8080 succeeds on the host and every remote client gets connection refused, because the kernel never accepts the connection on an external address.

The fix is in the daemon’s own configuration, not the network:

Service	Directive	Value to listen everywhere
PostgreSQL	`listen_addresses` in `postgresql.conf`	`'*'` (then fix `pg_hba.conf`)
MySQL / MariaDB	`bind-address` in `my.cnf`	`0.0.0.0` or `::`
Redis	`bind` in `redis.conf`	the interface address, plus a password
sshd	`ListenAddress` in `sshd_config`	`0.0.0.0` and `::`
A systemd unit	the daemon’s own `--bind`/`--host` flag	the address, not `127.0.0.1`

2. Is the packet even arriving?

If it is bound to 0.0.0.0/[::] and remote clients still fail, find out whether their SYN reaches the host. tcpdump taps below netfilter, so it sees packets the firewall is about to drop:

sudo tcpdump -i any -nn "tcp port 8080 and tcp[tcpflags] & tcp-syn != 0"

What you see	What it means
SYN in, no SYN-ACK, no RST	The local firewall is dropping it
SYN in, RST out	Nothing listening on that address, or a REJECT rule
SYN in, SYN-ACK out	The host is fine; look at the return path or the client
No SYN at all	Upstream: routing, security group, or a filter on the path

Then confirm against the ruleset, and read the counters rather than only the rules - a rule with a rising counter is the rule that is eating your traffic:

sudo nft list ruleset | grep -n 8080
sudo nft list chain inet filter input
# iptables hosts:
sudo iptables -L INPUT -v -n --line-numbers

This is the difference between evidence and a guess. “Firewall: check the rules” as a blind fix invites the worst pitfall in the list below - flushing the ruleset “to test”. The tcpdump result tells you whether the firewall is the cause before you touch it.

Form a hypothesis before changing anything

The temptation is to “try things” - restart the service, restart networking, change the IP. Each change destroys evidence and may introduce new symptoms. Stop and ask: what do I expect to change if I do X?

# Bad: change things, see what happens
sudo systemctl restart networking
sudo ip route add default via 10.0.0.1
sudo iptables -F

# Good: form a hypothesis, test it
# Hypothesis: the gateway is unreachable.
ip route get 1.1.1.1
# Test result: route is correct.
ip neigh show
# Test result: gateway MAC is REACHABLE.
ping 10.0.0.1
# Test result: gateway responds.
# Hypothesis refuted; problem is elsewhere.

Common pitfalls

Restart as the first action: it loses evidence and rarely identifies the cause.
Disabling the firewall “to test”: it opens the host to attack and rarely identifies the cause.
Changing configuration under pressure: a config change without a hypothesis is gambling.
Looking at logs last: logs are evidence; collect them early.
Trusting “it worked yesterday”: it may not have; the symptom may have been latent.

When you cannot solve it

Escalate. A methodology does not mean refusing to ask for help. Bring:

The symptom (verbatim error message).
The evidence (commands run, outputs).
The hypothesis (what you think it is).
The changes tried (and their outcomes).

A 30-minute colleague conversation with this information is worth 4 hours of solo struggle.