Lab: End-to-end connectivity walk-through on a real host

This lab is the diagnostic methodology applied to a live host. By the end you will have walked every layer from physical to application, captured the output, and produced a baseline that you can use as the comparison point for future incidents.

Objective

By the end of this lab, you can:

Test connectivity from link to application, layer by layer.
Document the output at each layer.
Identify the layer where a hypothetical failure would show up.
Build a complete inventory as a baseline.

Architecture

The host is a typical production server with one or more Ethernet interfaces, a default route, and several listening services. You will validate connectivity to a known-good remote host (an internal server or a public service like example.com).

Tasks

Task 1: Layer 1 (physical) and Layer 2 (link)

ip -br link show
ip -d link show eth0
ethtool eth0 | grep -E 'Speed|Duplex|Link detected|Auto-negotiation'
ethtool -S eth0 | grep -E 'errors|dropped|missed'
ip neigh show

For eth0 (or whichever interface is up), record:

Interface state (UP/DOWN).
MAC address.
Speed and duplex.
Any non-zero error or drop counters.
Current neighbour cache entries.

If Link detected: no, layer 1 is broken - check the cable, switch port, and NIC.

If the neighbour cache is empty for the gateway, layer 2 is broken - the host cannot ARP for its next hop.

Task 2: Layer 3 (network)

ip -br addr show
ip -4 route show
ip -6 route show
ip route get 8.8.8.8
ip route get 8.8.8.8 from 127.0.0.1
ping -c 3 $(ip route | awk '/default/ {print $3; exit}')
ping -c 3 8.8.8.8

For each interface, record:

IP address(es) and prefix length.
Default route and gateway.
Whether the gateway responds to ping.

If the host has an IP but no default route, layer 3 is broken on this host.

If the gateway does not respond to ping, layer 2 or layer 3 of the gateway is broken - check with ip neigh and tcpdump.

Task 3: Layer 4 (transport)

ss -tlnp
ss -unlp
ss -s

For every listening socket, record:

Protocol (TCP or UDP).
Local address (note 0.0.0.0 or [::] for all interfaces).
Local port.
Process and PID.

If the service you expect is not in the list, it is not listening. Check the application logs and the unit status.

ss -tnp state established | wc -l
ss -tan state time-wait | wc -l
ss -tan state close-wait | wc -l

Record the counts of ESTABLISHED, TIME-WAIT, CLOSE-WAIT connections. These are the baseline numbers for this host.

Task 4: Layer 7 (application) - DNS

cat /etc/resolv.conf
resolvectl status 2>/dev/null
dig example.com
dig @8.8.8.8 example.com
dig +short example.com

Record:

Configured resolvers in /etc/resolv.conf (or resolvectl).
Whether systemd-resolved is in use.
The first three answers for dig example.com.
Whether the system resolver and an external resolver agree.

If the answers disagree, you have a stale cache, split horizon, or DNS misconfiguration.

Task 5: Layer 7 (application) - HTTP

curl -s -o /dev/null -w "%{http_code}\n" https://example.com
curl -v -o /dev/null -w '
DNS lookup:        %{time_namelookup}s
TCP connect:       %{time_connect}s
TLS handshake:     %{time_appconnect}s
Server response:   %{time_starttransfer}s
Total:             %{time_total}s
' https://example.com

Record:

HTTP status code returned.
Time spent in each phase (DNS, TCP, TLS, server).

If the request fails, the verbose output shows exactly where (name resolution, TCP connect, TLS handshake, server response).

Task 6: Build the inventory

Combine all the recorded information into a single document:

HOST BASELINE INVENTORY
======================
Date: ...
Hostname: ...
Distribution: ...
Kernel: ...

Layer 1 / 2 (Link):
- Interface eth0: UP, MAC aa:bb:cc:dd:ee:ff, 10000Mb/s full
- Link errors: 0
- Neighbour cache: gateway REACHABLE, ...

Layer 3 (Network):
- IPv4: 10.0.0.10/24
- IPv6: 2001:db8:1::10/64
- Default route: via 10.0.0.1 dev eth0
- Gateway reachable: yes

Layer 4 (Transport):
- Listening TCP: 22 (sshd), 80 (nginx), 443 (nginx)
- Listening UDP: 53 (systemd-resolved)
- ESTABLISHED count: 42
- TIME-WAIT count: 18
- CLOSE-WAIT count: 0

Layer 7 (Application):
- Resolver: 127.0.0.53 (systemd-resolved)
- External resolver check: agrees with local
- HTTP example.com: 200, total time 145ms

This inventory is the baseline. During incidents, compare current state to the baseline; deviations are diagnostic.

Task 7: Hypothetical failures

For each layer, describe what a failure looks like:

The Rung column is the rung of the ladder in linux-network-troubleshooting-methodology that catches each failure. Learn the table in that column’s order, not the layer’s.

Rung	Layer	Failure symptom	Diagnostic
1	4	Service not listening	ss -tlnp does not show the port
1	4	Bound to loopback	`ss -tlnp '( sport = :8080 )'` shows `127.0.0.1:8080`; works from the host, refused from everywhere else
2	3	Routing wrong	ip route get returns wrong interface or wrong src
3	1	Link DOWN	ethtool shows “Link detected: no”
3	2	ARP fails	ip neigh FAILED for the destination
4	3/4	Host firewall dropping	tcpdump sees the client SYN arrive, nothing leaves; rule counters increment
5	3	Path MTU mismatch	Connection opens, small requests work, larger transfers hang; `ping -M do -s 1472` fails while `ping` succeeds
6	3	Break in the path	`traceroute -T -p 443` stops with `* * *` before the target
8	7	DNS misconfigured	dig returns wrong IP or NXDOMAIN
-	7	HTTP service down	curl returns connection refused/timeout

Rung 5 is the one this lab would otherwise never show you. Record the working MTU for the interface now, while the host is healthy:

ip -br link show eth0
tracepath -n 8.8.8.8 | tail -3
ping -M do -s 1472 -c3 8.8.8.8    # 1472 + 28 = 1500

If -s 1472 fails but a plain ping succeeds, the path MTU is below 1500 and belongs in the baseline inventory.

Local success is not remote reachability

The two layer-4 rows added above are the pair that looks identical from the host and behaves completely differently from the network.

A loopback bind gives you a LISTEN line and a working curl localhost:8080, and remote clients get connection refused - the kernel never accepts the connection on an external address. ss -tlnp | grep :8080 matches 127.0.0.1:8080 and 0.0.0.0:8080 identically, so grepping for the port is what hides it. Read the local-address column.

A firewall DROP gives you a correct 0.0.0.0 bind and remote clients time out. tcpdump taps below netfilter, so the SYN is visible arriving with nothing going back. Confirm with the rule counters; do not flush the ruleset to test.

Refused versus timed out is the fastest discriminator you have between the two.

Validation

Layer-by-layer output is recorded.
A baseline inventory exists.
The hypothetical failure table is complete and accurate.
For every listening service in the layer-4 inventory, the local address is recorded, not just the port.

Cleanup

This lab is read-only. No state changes.

What you learned

A complete diagnostic walks every layer.
Each layer has its own diagnostic tool.
A baseline inventory is the comparison point for future incidents.

Objectives

Prerequisites