Runbook: Network incident - service unreachable triage

This runbook triages a “service unreachable” report. Use it when a user, monitor, or service reports it cannot reach a specific service on a known host. The goal is to identify the failing layer and restore service within 30 minutes for most incidents.

When to use this runbook

Use this runbook when:

• A user reports “X is down” or “I cannot reach Y”.
• A monitor reports a connectivity alert.
• An integration test or health check fails.
• A new host or service is not reachable from expected clients.

Inputs

Gather before starting:

• Service name: what the user is trying to reach (ssh, database, web app, internal API).
• Target host: hostname or IP.
• Target port: the specific port the service listens on.
• Source: where the user is connecting from (their host, an internal network, etc.).
• When it started: timestamp or approximate time.

Procedure

Step 1: Confirm the symptom

Reproduce the failure from a known-good source (your workstation or a debug host):

nc -vz target.example.com 80
curl -I https://target.example.com

Note the exact error: “Connection refused”, “Connection timed out”, “No route to host”, or another message. The error maps to a layer.

Step 2: Test DNS

dig target.example.com
dig @8.8.8.8 target.example.com

If DNS fails, jump to the DNS failure runbook. If DNS works but the IP is unexpected, check the resolver cache and the authoritative zone.

Step 3: Test routing, then reachability

Ask the local kernel what it will do with the packet before you send anything onto the wire. ip route get is answered by the FIB on this host, so no firewall anywhere can filter it and its answer is never ambiguous:

ip route get <target-ip>

Read the source address and the outgoing interface, not just the presence of a route. A route via the wrong interface or with an unexpected src is a real layer 3 finding.

Then probe the port you actually need:

nc -vz <target-ip> <port>

Only then, optionally, ICMP:

ping -c 3 target.example.com
ping -c 3 <target-ip>

If ping fails by hostname but works by IP, the problem is DNS (already addressed in step 2).

Step 4: Test the layer 3 path

traceroute -T -p <port> target.example.com
mtr -T -P <port> target.example.com

If traceroute stops with * * * early, routing or filtering is breaking. If the path looks complete, the issue is at the target.

Step 4a: Test the path MTU

Run this whenever the symptom is not a clean failure: the connection opens, the TLS handshake completes, small requests succeed, and then a larger response hangs or stalls part-way. That shape is path MTU, and no other step in this runbook will find it.

TARGET=target.example.com
tracepath -n "$TARGET"
ping -M do -s 1472 -c3 "$TARGET"   # 1472 + 28 = a 1500-byte packet
ping -M do -s 1372 -c3 "$TARGET"   # 1400 MTU, typical of a tunnel

If the 1472 probe reports Message too long or simply times out while a plain ping succeeds, the path MTU is below 1500. A tunnel (VPN, VXLAN, GRE) somewhere on the path is the usual cause, together with a firewall dropping the ICMP fragmentation needed messages that would otherwise let PMTUD discover it. Escalate rather than clamping MTU on the host: the fix belongs on the device that owns the tunnel.

Step 5: Test layer 2 — only for an on-link target

Skip this step unless the kernel says the target is on your own segment. arping is a link-local probe: it puts an ARP request on the wire and waits for the owner of that address to answer. For anything reachable via a gateway it will always fail, and that failure is the expected result, not a finding. Running it against a remote target and concluding “layer 2 is broken” sends the incident down a path that has no end.

Ask the kernel which case you are in, and read the answer you already captured in step 3:

ip route get <target-ip>

• No via in the output (... dev eth0 src 10.0.0.10): the target is on-link. ARP is meaningful — probe the target.
• A via in the output (... via 10.0.0.1 dev eth0 src ...): the target is off-subnet. Probe the gateway instead. It is the only layer 2 neighbour on the path that you own.

Then check the neighbour cache before generating traffic — the kernel has usually already answered this question:

ip neigh show <ip>
arping -I <iface> -c 3 <ip>

REACHABLE means layer 2 is fine and this step is done. FAILED means the kernel already tried and got nothing. STALE is normal and is not a finding.

An arping that fails for an address the kernel says is on-link narrows to four causes, and three of them are not “the network is broken”:

• The host is genuinely down, or its NIC is gone.
• net.ipv4.conf.*.arp_ignore or arp_filter on the target is suppressing the reply. Common on multi-homed hosts, and it affects only ARP — the service itself may be perfectly reachable.
• Switch port security or dynamic ARP inspection is dropping it.
• You passed the wrong interface to -I. Compare it against the dev that ip route get printed, and do not assume eth0.

Step 6: Test the target service

On the target host, or via SSH:

ss -tlnp '( sport = :<port> )'
systemctl status <service>
journalctl -u <service> -n 50

Read the local-address column of the LISTEN line, not just the fact that a line came back.

If nothing is listening at all, the temptation is to restart the service. Collect the evidence first — a restart destroys most of it, and “restart as the first action” is the top pitfall in the methodology lesson this runbook depends on.

# Why did it stop? Capture this BEFORE any restart.
systemctl status <service> --no-pager -l
journalctl -u <service> --since "1 hour ago" --no-pager | tail -100

# Has it been flapping? NRestarts is the whole story in one line.
systemctl show <service> -p NRestarts,ExecMainStatus,Result

# Did the packet even arrive?
sudo tcpdump -i any -nn "tcp port <port> and tcp[tcpflags] & tcp-syn != 0"

ExecMainStatus and Result tell you whether it exited cleanly, was OOM-killed, or failed to start. NRestarts tells you whether this is the first time. A service that has restarted 40 times has a different problem from one that stopped once.

Write down a hypothesis for why it stopped. Then, and only then:

sudo systemctl restart <service>

A restart that brings the service back is a mitigation, not a diagnosis. It restores the SLA and it buys you time to read the journal you just captured. If you close the incident here, the same page arrives at 03:00 with the evidence gone.

Step 6a: Prove whether the target’s firewall is dropping it

If the listener is bound correctly and remote clients still time out, find out whether their SYN reaches the host at all. tcpdump taps below netfilter, so it sees packets the firewall is about to drop:

PORT=5432
sudo timeout 30 tcpdump -i any -nn "tcp port $PORT and tcp[tcpflags] & tcp-syn != 0"

Then read the ruleset and its counters - a rule with a rising counter is the rule eating your traffic:

PORT=5432
sudo nft list ruleset | grep -n "$PORT"
sudo nft list chain inet filter input
# iptables hosts:
sudo iptables -L INPUT -v -n --line-numbers

Step 7: Capture evidence

Throughout the procedure, capture:

# All command outputs to a single file
exec > /tmp/incident-$(date +%Y%m%d-%H%M%S).log 2>&1
echo "Incident: $SERVICE unreachable"
echo "Time: $(date)"
echo "Source: $(hostname)"
echo "Target: $TARGET:$PORT"
echo "---"
# ... run all commands ...

The log file is the input to the post-incident review.

Step 8: Restore service

Common fixes, by failure layer:

• DNS: flush caches (resolvectl flush-caches), check authoritative zone.
• Routing: ip route get and ip route add for missing routes.
• Firewall: check iptables/nftables rules; check the cloud security group.
• Service: restart the service. If it fails to start, check logs.
• Hardware: replace cable, NIC, or escalate to ops.

Step 9: Verify and document

After the fix:

nc -vz target.example.com <port>
curl -I https://target.example.com

Confirm the original reproducer now succeeds.

Write a one-paragraph incident summary: what was the symptom, what was the cause, what fixed it, what should change to prevent recurrence.

Common patterns

Escalation

Escalate to senior engineering when:

• The failure involves multiple hosts.
• The fix requires a change to a production firewall or routing configuration.
• The root cause is not identified within 30 minutes.
• The same failure has occurred twice in a week (recurring incident).

Bring: the symptom, the evidence (logs from the incident capture), the changes tried, the hypothesis about root cause.