Runbook: DNS failure - restore name resolution

This runbook triages a DNS failure: hosts cannot resolve names, services that depend on DNS fail. The goal is to identify the failing layer (local resolver, upstream resolver, or authoritative server) and restore resolution within 30 minutes.

When to use this runbook

Use this runbook when:

• A host or service reports “DNS not working”.
• A user reports they cannot reach a service by name.
• An integration test fails because of a hostname resolution failure.
• DNS monitoring alerts.

Inputs

Gather before starting:

• Symptom: exact error (NXDOMAIN, SERVFAIL, timeout, connection refused).
• Affected hosts: how many, which ones.
• Affected services: which services depend on DNS.
• Time started: when did it start?
• Recent changes: DNS changes, network changes, host changes.

Procedure

Step 1: Confirm the symptom

Reproduce the failure from a known-good source:

dig example.com
nslookup example.com
getent hosts example.com
curl -I https://example.com

Note the exact error. NXDOMAIN, SERVFAIL, timeout, and connection refused are different failures with different causes.

Step 2: Capture the configuration

ls -l /etc/resolv.conf
cat /etc/resolv.conf
resolvectl status 2>/dev/null
grep hosts /etc/nsswitch.conf

Document the resolver configuration. If the file is a symlink to systemd-resolved, the resolver is systemd-resolved.

Step 3: Test the local resolver

dig @127.0.0.1 example.com          # if running a local resolver
dig @127.0.0.53 example.com         # systemd-resolved

If this fails, the local resolver is the problem. Try flushing the cache before restarting anything — it is non-disruptive and resolves the most common case, a cached negative answer:

resolvectl flush-caches           # systemd-resolved
sudo sss_cache -E                 # sssd: expire all cached entries

If a flush does not fix it, restart only the caching layers this host actually runs. Do not restart a daemon blindly: on a host without it the command fails and adds noise to the incident, and nscd in particular is deprecated in modern glibc and is not installed by default on current RHEL, Fedora, Debian or Ubuntu — its role is filled by systemd-resolved or sssd.

for u in systemd-resolved sssd nscd unbound dnsmasq; do
systemctl list-unit-files "$u.service" --no-legend | grep -q . || continue
systemctl is-active --quiet "$u" || continue
echo "restarting $u"; sudo systemctl restart "$u"
done

Step 4: Test the configured upstream

dig @<configured-resolver-ip> example.com

If this fails, the upstream resolver is the problem. Try the fallback:

dig @8.8.8.8 example.com
dig @1.1.1.1 example.com

If both upstream and fallback fail, the problem is upstream of your resolvers (your network or the global DNS).

If small answers resolve but large ones do not, hold the resolver and the name constant and vary only the transport:

dig @<configured-resolver-ip> example.com TXT +notcp +bufsize=512
dig @<configured-resolver-ip> example.com TXT +notcp +bufsize=4096
dig @<configured-resolver-ip> example.com TXT +tcp

+notcp forbids the automatic fallback to TCP, so a truncated answer stays truncated instead of being papered over. +tcp sends the query over TCP/53 directly.

Step 5: Test the authoritative server

dig +trace example.com

If +trace fails at the authoritative server, the zone is broken. Wait for recovery or escalate to the zone owner.

Step 6: Check the cache

resolvectl statistics

High cache hit rate is good. Low hit rate may indicate a cache eviction problem.

If you suspect stale cache:

resolvectl flush-caches

Retry the failing query.

Step 7: Bypass DNS

If you need immediate service restoration while fixing DNS, bypass DNS by editing /etc/hosts:

echo "10.0.0.5 db.example.com" | sudo tee -a /etc/hosts

The files NSS module reads /etc/hosts before DNS. This is a temporary fix; remove the entry once DNS is restored.

Step 8: Common fixes

Wrong resolver IP:

sudo nmcli connection modify "Wired connection 1" \
ipv4.dns "10.0.0.1 10.0.0.2"
sudo nmcli connection down "Wired connection 1"
sudo nmcli connection up "Wired connection 1"

systemd-resolved not running:

sudo systemctl enable --now systemd-resolved
sudo systemctl restart systemd-resolved

Stale cache:

resolvectl flush-caches

DNSSEC chain broken:

Identify the broken link with dig +trace +dnssec example.com. Either fix the chain or temporarily disable validation.

/etc/resolv.conf overwritten by DHCP:

Configure DHCP client to preserve your nameservers (see the /etc/resolv.conf lesson).

Step 9: Verify

After the fix:

dig example.com
getent hosts example.com
curl -I https://example.com

All three should agree and succeed.

Step 10: Document

Write a one-paragraph summary:

• Symptom
• Failing layer (local resolver, upstream, authoritative)
• Fix applied
• Root cause (if identified)
• Recommendation for prevention

Common patterns

Escalation

Escalate when:

• The fix involves changing authoritative DNS for a production zone.
• The root cause is not identified within 30 minutes.
• The same failure has occurred twice in a week.

Bring: symptom, evidence (logs from the incident capture), fix tried, hypothesis.