Lab: DNS troubleshooting on a live host

This lab walks through DNS troubleshooting on a live host. By the end you will have a baseline DNS inventory and a documented procedure for the most common DNS failures.

Objective

By the end of this lab, you can:

• Read the DNS configuration of a host.
• Test the resolver chain end-to-end.
• Recognise the difference between a resolver, upstream, and authoritative DNS failure.
• Restore DNS resolution under time pressure.

Architecture

The host has:

• A configured resolver (systemd-resolved, NetworkManager- written /etc/resolv.conf, or static).
• A reachable recursive resolver (often the company DNS).
• Internet access to upstream resolvers (Google, Cloudflare).
• One or more services that depend on DNS.

Tasks

Task 1: Read the configuration

ls -l /etc/resolv.conf
cat /etc/resolv.conf

resolvectl status 2>/dev/null
systemctl is-active systemd-resolved

grep hosts /etc/nsswitch.conf

Document:

• Is /etc/resolv.conf a real file or a symlink?
• Who owns the file (which subsystem writes it)?
• What resolvers are configured?
• Is systemd-resolved in use?
• What NSS modules are configured for hosts?

Task 2: Test the local resolver

dig example.com
dig +short example.com
getent hosts example.com

Record:

• The first answer from dig example.com.
• Whether getent returns the same answer.
• The TTL of the answer.

If dig and getent disagree, the resolver library is not reading the same configuration as dig. Check the /etc/resolv.conf symlink and systemd-resolved state.

Task 3: Test specific record types

dig example.com A
dig example.com AAAA
dig example.com MX
dig example.com NS
dig example.com SOA

Verify:

• A and AAAA records exist.
• MX records point to reachable mail servers.
• NS records match the SOA MNAME.
• The SOA serial is reasonable (not ancient).

Task 4: Test the upstream chain

dig @8.8.8.8 example.com
dig @1.1.1.1 example.com

Verify that external resolvers agree with the local resolver. If they differ, there is a split-horizon (expected for internal zones) or stale cache (verify with resolvectl flush-caches and retry).

Task 4b: Isolate the transport

Read-only. DNS is not a UDP-only protocol. When an answer does not fit the advertised EDNS buffer the server sets the TC (truncated) flag and the client retries the same question over TCP/53. A path that carries small UDP answers but drops large or fragmented UDP, or blocks TCP/53, produces the most confusing DNS symptom there is: small answers resolve and large ones time out.

Hold the resolver and the name constant. Change only the transport:

# Force UDP, small buffer - should always work
dig @8.8.8.8 example.com TXT +notcp +bufsize=512

# Force UDP, large buffer - tests large/fragmented UDP
dig @8.8.8.8 example.com TXT +notcp +bufsize=4096

# Force TCP/53 directly
dig @8.8.8.8 example.com TXT +tcp

+notcp forbids the automatic fallback to TCP, so a truncated answer stays truncated and you see the failure instead of it being papered over.

Record which of the three succeed, then read the row:

Two things to look for in the output: flags: qr tc rd ra (the tc flag means the answer was truncated) and ;; Truncated, retrying in TCP mode (dig telling you the fallback happened).

Task 5: Trace the resolution chain

dig +trace example.com

Identify each step:

• Root servers (.).
• TLD servers (com.).
• Authoritative servers for example.com.
• The final answer.

Task 6: Test the search list

dig does not apply the search list from /etc/resolv.conf unless you ask it to — man 1 dig says so under +search: “The search list is not used by default.” Your applications, which go through the glibc resolver, do apply it. That asymmetry is the whole point of this task, and it is the reason “dig works but the app cannot resolve it” is such a common report.

First, read the configuration you are about to test:

# Substitute your own values before running:
SHORTNAME=web01

grep -E '^(search|domain|options)' /etc/resolv.conf
resolvectl domain            # if systemd-resolved is in use

Now ask dig to use the search list, and show its working:

# Substitute your own values before running:
SHORTNAME=web01

# Default: no search list. The literal name is queried as-is.
dig "$SHORTNAME"

# With the search list, and showing each suffix as it is tried
dig +search +showsearch "$SHORTNAME"

Then compare against the resolver your applications actually use:

# Substitute your own values before running:
SHORTNAME=web01

getent hosts "$SHORTNAME"

Verify:

• dig "$SHORTNAME" alone returns NXDOMAIN for the bare label — that is correct behaviour, not a broken search list.
• dig +search and getent hosts agree, and both land in the zone you expect.
• The suffixes +showsearch tries are the ones in /etc/resolv.conf, in that order.

Task 7: Test cache behaviour

# First lookup (cache miss)
time dig example.com

# Second lookup (cache hit)
time dig example.com

# Flush cache and retry
resolvectl flush-caches
time dig example.com

The first lookup (or the one after flush) should be slower than the cache hit. A working cache reduces upstream load.

Task 8: Test failure modes

Simulate an upstream resolver failure (disposable lab host only, per the warning at the top of this lab):

# 1. Record the ruleset before you touch it. This is the
#    rollback of last resort, and the diff you compare against.
#    The redirection runs inside sudo, because a plain
#    `sudo cmd > /root/file` redirects as your own user and
#    fails with "Permission denied".
sudo sh -c 'iptables-save > /root/iptables-pre-lab.rules'
sudo sh -c 'iptables -S OUTPUT > /root/iptables-output-pre-lab.txt'

# 2. Drop DNS to ONE upstream resolver only.
#    Do not block port 53 wholesale: that also kills the
#    fallback resolver you are about to test, and it kills
#    name resolution for every other process on the host.
sudo iptables -A OUTPUT -p udp -d 8.8.8.8 --dport 53 -j DROP
sudo iptables -A OUTPUT -p tcp -d 8.8.8.8 --dport 53 -j DROP

# 3. Confirm the blocked resolver now times out
dig @8.8.8.8 example.com

# 4. Confirm the fallback resolver still answers
dig @1.1.1.1 example.com

# 5. Restore: delete exactly the two rules you added
sudo iptables -D OUTPUT -p udp -d 8.8.8.8 --dport 53 -j DROP
sudo iptables -D OUTPUT -p tcp -d 8.8.8.8 --dport 53 -j DROP

# 6. Prove you are back where you started
sudo sh -c 'iptables -S OUTPUT | diff - /root/iptables-output-pre-lab.txt' && echo "OUTPUT chain restored"

Record how long dig takes to give up on the blocked resolver. That timeout is what your applications inherit when an upstream resolver dies, and it is usually the reason a service looks “slow” rather than “broken”.

Simulate stale cache:

# Resolve a name, then change the upstream answer
# (Requires a host you control, or a known-stale example)

Task 9: Document the recovery procedure

For each of the following failure modes, document the recovery:

Validation

• A complete DNS inventory exists for the host.
• All resolver chain steps are tested.
• Recovery procedures for common failures are documented.

Cleanup

Only Task 8 changes state. Remove the two rules it added, and nothing else. If you already ran step 5 of Task 8, these deletes report Bad rule (does a matching rule exist in that chain?). That is the expected answer, and it is the proof that the rules are gone:

# Remove ONLY the rules this lab added.
# Never flush a chain you did not create.
sudo iptables -D OUTPUT -p udp -d 8.8.8.8 --dport 53 -j DROP
sudo iptables -D OUTPUT -p tcp -d 8.8.8.8 --dport 53 -j DROP

# Verify the chain is back to its pre-lab state
sudo iptables -S OUTPUT
sudo sh -c 'iptables -S OUTPUT | diff - /root/iptables-output-pre-lab.txt' && echo "OUTPUT chain restored"

If diff still reports a difference, restore the saved ruleset wholesale on a host you own:

sudo sh -c 'iptables-restore < /root/iptables-pre-lab.rules'

If Task 7 flushed the resolver cache, warm it again with a lookup so the next request does not pay the miss:

dig +short example.com

What you learned

• A complete DNS inventory includes the resolver, the upstream chain, and the authoritative servers.
• Stale caches, wrong resolvers, and DNSSEC failures are common DNS problems.
• “Small answers resolve, large ones time out” is a transport fault, not a data fault. Vary only the transport - +notcp +bufsize=512, +notcp +bufsize=4096, +tcp - to place it.
• Recovery procedures should be pre-written.
• Fault injection is only safe when it is reversible. Save the ruleset first, add the narrowest rule that reproduces the fault, then delete that exact rule and diff against the baseline to prove you restored it.