Failure: DNS, routing, packet loss, firewall and certificate expiry

“The network is down” is never a diagnosis. Four different failures produce it, and each is confirmed or eliminated by a single command. Work down the layers in order; do not start with tcpdump.

The localisation ladder

1. Does the name resolve? dig +short host - and check which resolver answered, not just the answer.
2. Is there a route to the address? ip route get ADDR names the interface, source address and gateway that will be used.
3. Does the transport connect? Wait for a SYN-ACK, an RST, or nothing at all - the three outcomes mean three different things.
4. Does the TLS session complete? openssl s_client speaks to the endpoint that is actually serving, not to the file you think it is serving.

Each rung eliminates everything below it. Skipping a rung is how a DNS incident gets diagnosed as a firewall incident.

Resolution

dig +short api.example.com
dig api.example.com @10.0.0.10          # ask a specific resolver
resolvectl status                        # which resolver does this host use?
getent hosts api.example.com             # what the application actually gets

dig and the application do not necessarily agree. dig talks to the resolver directly; the application goes through NSS, so /etc/hosts, nsswitch.conf and the systemd stub resolver can all change the answer. When they disagree, getent is the truth for the application.

A stale answer with a long TTL is its own failure: the record was corrected an hour ago and this host will keep the old value until the TTL expires. Check the TTL in the full dig output before concluding the DNS change did not work.

Routing

ip route get is the single most under-used command in network troubleshooting, because it answers the question the routing table only implies:

$ ip route get 10.20.0.5

10.20.0.5 via 10.0.0.1 dev eth0 src 10.0.0.42 uid 1000
  cache

That output names the interface, the gateway and - critically

• the source address the kernel will put on the packet. A wrong source address is the usual cause of “it works from the host but not from the container”, and of asymmetric routing on a multi-homed host.

Asymmetry has a distinctive signature: the SYN arrives (you can see it in tcpdump on the server) but the client never sees a reply, because the reply left by a different interface and was dropped by reverse-path filtering or by a stateful firewall that never saw the outbound half.

sysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.eth1.rp_filter
nstat -az | grep -i -E 'rpfilter|martian'

Filtering: three outcomes, three causes

The failure signature of a connection attempt tells you which of three things happened:

Confirm which end is silent before blaming a firewall. On the server, is anything listening at all, and on which address?

# ss -tlnp | grep 8443

LISTEN 0  511  127.0.0.1:8443  0.0.0.0:*  users:(("app",pid=2311,fd=7))

A service bound to 127.0.0.1 refuses every remote connection while looking perfectly healthy in systemctl status and in a local curl. This is one of the most common “the firewall is blocking it” tickets, and there is no firewall involved.

If something is listening on the right address, count packets at the firewall rather than reading rules:

nft list ruleset | grep -n counter        # counters on the rules you care about
nft -a list chain inet filter input       # handles, in evaluation order

Rules are evaluated in order. A rule that looks correct but sits after an earlier match never runs, and only a counter proves which rule matched.

Packet loss and MTU

Intermittent slowness with working connectivity is usually loss or MTU, and both are measurable rather than debatable:

mtr -rwc 100 api.example.com          # loss per hop, 100 probes
ip -s link show eth0                   # errors, drops, overruns per interface
ping -M do -s 1472 api.example.com     # do-not-fragment probe at 1500 MTU

Loss that appears at one hop and disappears at the next is usually that router deprioritising ICMP, not real loss. Loss that persists to the destination is real. A DF-bit ping that fails at 1472 bytes but succeeds at, say, 1372 is a classic MTU black hole - typical over a tunnel or an overlay network, and the reason “small requests work, large uploads hang”.

Certificate expiry

Check the certificate the endpoint actually serves. The file on disk may have been renewed hours ago and never loaded:

$ openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>/dev/null | openssl x509 -noout -subject -dates

subject=CN = api.example.com
notBefore=May  9 00:00:00 2026 GMT
notAfter=Aug  7 23:59:59 2026 GMT

-servername matters: without it you get the default virtual host’s certificate and may diagnose the wrong service entirely.

Two failures that look like expiry and are not: an incomplete chain, where the leaf is valid but the intermediate is missing so only some clients fail; and a client whose clock is wrong, which rejects a perfectly valid certificate. Both are why the error message from the client is part of the evidence.

Knowledge check