DNS troubleshooting — unbound-control, drill, tcpdump, SERVFAIL patterns

DNS failures in production are silent until they are not. A misconfigured forwarder, a stale trust anchor, an expired upstream certificate, an access-list change that nobody documented — each one produces the same surface symptom (queries stop returning answers) and a different underlying cause. The operator’s job is to identify the cause without breaking the resolver further. This lesson covers the diagnostic command sequence, the SERVFAIL pattern recognition, and the evidence-first discipline that prevents diagnosis from making the failure worse.

The diagnostic sequence

The diagnostic sequence is a fixed order. Each step produces evidence that narrows the cause; the operator does not skip steps because they have a hypothesis.

Step 1: Query the firewall directly from a known-good source. This separates the resolver is broken from the client cannot reach the resolver. Use drill with the firewall’s IP explicitly:

drill @192.0.2.1 example.com

If the query returns an answer, the resolver works and the failure is on the path between the failing client and the firewall. If the query returns REFUSED, SERVFAIL, or no answer, the resolver or its dependencies are the problem.

Step 2: Read the drill flags. The flags at the top of drill’s output (qr rd ra ad) tell the operator what happened to the query:

The combination of flags and rcode is the first diagnostic. NOERROR with ad and a populated ANSWER section is healthy. NOERROR with empty ANSWER is a valid “no record of this type” response. SERVFAIL is a failure; the cause is in the chain. REFUSED is an ACL refusal; the source is not allowed.

Step 3: Check the resolver’s state with unbound-control. The daemon reports its own health through unbound-control stats and unbound-control status. The stats output is read-only and safe.

Step 4: Check the service status. configctl unbound status or the GUI equivalent (Services → Unbound DNS → Status) shows whether the daemon is running, its uptime, and any startup errors.

Step 5: Capture packets. tcpdump -ni igb0 port 53 (or the WAN interface for upstream captures) shows what the resolver is actually doing on the wire.

Recognising the SERVFAIL patterns

SERVFAIL is the rcode the resolver returns when something has gone wrong but the resolver cannot tell you what. The drill output above the rcode is the diagnostic gold.

SERVFAIL with ad flag set: This is the DNSSEC pattern. The chain of trust did not close. The fix is at the zone — a missing DS record at the parent, a key that has rolled and the resolver has not picked up the new DS, a zone that is signed but whose parent is not.

$ drill -S broken-signer.example.com @192.0.2.1

;; ->>HEADER<<- opcode: QUERY, rcode: SERVFAIL, id: 42387
;; flags: qr rd ra ; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
;; EDNS: version 0; flags: do; udp: 4096
;; QUESTION SECTION:
;; broken-signer.example.com. IN A
;; ANSWER SECTION:
;; AUTHORITY SECTION:
;; ADDITIONAL SECTION:
;; ;; Got answer:
;; ;; ->>HEADER<<- opcode: QUERY, rcode: SERVFAIL, id: 42387
;; ;; flags: qr rd ra ; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; ;; QUESTION SECTION:
;; ;; broken-signer.example.com. IN A
;; WARNING: response has an incomplete chain: missing DS record for example.com

SERVFAIL with empty ANSWER and no chain warning: The resolver tried to walk the chain and got back an answer that did not parse, or the upstream timed out. This is usually a forwarding or upstream-connectivity problem. Check unbound-control stats — total.recursion.time.avg near zero with total.num.cachemiss near total.num.recursivereplies means the work was done; if those numbers diverge, the work was not done and the upstream is the suspect.

SERVFAIL with ;;COMMUNICATION or no response: The resolver cannot reach the upstream at all. The fix is upstream connectivity — a routing problem, a firewall rule on the WAN, an upstream that has gone down. The capture confirms it: tcpdump -ni igb1 port 53 shows the query go out and no answer come back.

NOERROR with empty ANSWER: The name exists but the requested record type does not. This is not a failure; this is a correct response. If the application expected an AAAA and got nothing, the application should fall back to A.

The common failure modes

Five patterns recur in production DNS troubleshooting.

Forwarder upstream is unreachable. Unbound in forwarder mode with one upstream; the upstream goes down. Every query returns SERVFAIL. The diagnostic: unbound-control stats shows the query count but no recursion replies, the daemon log shows “no servers reachable” or similar. The fix is a second upstream or a gateway group.

DNSSEC chain breaks at a child zone. The child zone is signed but the parent is not. Drill -S reports the missing DS record. The fix is upstream of the firewall — sign the parent or remove the child DS record. The operator’s mitigation in the meantime is to disable DNSSEC for the specific zone by adding it to the Insecure Domains list under Advanced settings.

Access list denies the source. Drill returns REFUSED. The query never enters recursion; the ACL refused it. The diagnostic: query from a known-allowed source and the answer comes back; query from the failing source and REFUSED. The fix is the ACL page.

Port 53 blocked on WAN. Unbound in resolver mode cannot reach the root servers because the WAN firewall blocks outbound UDP/53 and TCP/53. Drill against an external resolver works from the LAN (because the LAN queries the firewall’s cache) but the cache misses cannot refresh. The diagnostic: tcpdump -ni igb1 port 53 shows queries to root servers with no responses; drill @192.0.2.1 example.com shows SERVFAIL or no answer for cold-cache names. The fix is the WAN firewall rules — allow outbound 53 to the public DNS infrastructure.

Unbound is not running. The GUI shows the service is stopped; configctl unbound status reports the daemon is not running. The fix is to read the daemon log (/var/log/resolver.log or the system log filtered by unbound) and identify the startup failure. Common causes: a malformed generated configuration, a missing dependency, a port conflict.

$ configctl unbound status

unbound is not running.
check /var/log/resolver.log for details.
last lines of resolver log:
Aug 14 12:34:56 fw unbound: [4238:0] error: could not bind to port 53

Reading drill output flags

The flags at the top of drill’s output are the single most informative diagnostic in DNS troubleshooting. The combination tells the operator which subsystem the failure is in:

The flags are produced by the resolver or the upstream that answered; they reflect the resolver’s state at the moment the answer was assembled. Reading them carefully turns a 30-minute investigation into a 2-minute one.

Evidence-first diagnosis

The discipline of evidence-first diagnosis is to collect before changing. The temptation when DNS is broken is to start changing things — restart Unbound, clear the cache, change the upstream — in the hope that one of them fixes the problem. Each change destroys evidence about what the actual failure was.

The diagnostic sequence in this lesson is a sequence of read-only commands. Each one tells you something; none of them modify state. By the time the operator reaches the point where a change is the right action, the evidence is on the screen and the change is targeted, not speculative.

Summary

• The diagnostic sequence is fixed: query the firewall directly, read the flags, check unbound-control stats, check service status, capture packets. Do not skip steps.
• SERVFAIL with ad set is the DNSSEC pattern. Drill -S walks the chain and reports the failure.
• The five common failure modes: forwarder upstream unreachable, DNSSEC chain break, ACL denial, WAN port 53 blocked, Unbound not running.
• Drill flags tell the operator which subsystem is the problem: REFUSED is the ACL, SERVFAIL is the chain, NOERROR with empty answer is the type, NXDOMAIN is the name.
• The discipline is evidence-first: collect before changing, read the daemon log, capture packets before restarting anything.