DNS Incident Patterns

The on-call engineer is paged at 14:08. The user-visible metric for the customer portal is 35% error rate. The HTTP probe is red. The TCP probe is red. The DNS probe is red. The on-call engineer opens the runbook. The first step is “check the DNS panel.” The panel shows probe_success 0 for the portal. The metric is the cause. The runbook is the recovery. The postmortem is the prevention. This lesson is about the runbook that turns a DNS incident into a recovery in fifteen minutes.

What it is

A DNS incident is a production outage whose root cause is in the resolution path. The user-visible symptom is variable: a sharp uptick in error rate for a single endpoint, a slow application across the cluster, a “site is down” report from a user in a region the team does not have a probe in. The root cause is in one of the four steps of the resolution path: the system resolver, the upstream resolver, the authoritative server, or the registrar.

The four common shapes of DNS incident:

1. Authoritative server failure. The authoritative server is unreachable or returning SERVFAIL. The shape is a hard outage for the records that the operator serves. The application’s other dependencies are fine.
2. Resolver failure. The recursive resolver is unreachable or returning SERVFAIL. The shape is a hard outage for every record that the resolver serves. The authoritative server is fine.
3. Registrar hold. The registrar has applied a hold on the domain. The shape is a hard outage for the entire domain. The authoritative server is unreachable from the public internet.
4. Slow authoritative server. The authoritative server is slow. The shape is a slow look-up for the records that the operator serves. The user-visible symptom is a slow application.

The diagnostic order is the order in which the operator checks the four shapes. The first to confirm is the cheapest to confirm; the last is the most expensive. The right diagnostic order is the order that catches the shape the team is most likely to face.

Why a sysadmin cares

A DNS incident is a production outage. The cost is the SLO breach; the recovery is the diagnostic order; the postmortem is the prevention. The team’s job is to recover the platform in the shortest time possible without shipping the wrong fix.

The diagnostic order matters. The team that diagnoses in the wrong order is the team that spends an hour on the application before discovering the registrar is the cause. The team that diagnoses in the right order is the team that recovers the platform in fifteen minutes.

The escalation matters. The team that escalates early is the team that recovers the platform within the SLO. The team that escalates late is the team that breaches the SLO twice. The escalation is the call to the network team, the DNS provider, or the registrar. The team’s job is to escalate when the runbook says escalate.

The postmortem matters. The team that runs a postmortem is the team that prevents the next incident. The postmortem is the timeline, the root cause, the contributing factors, and the prevention. The team’s job is to run the postmortem within a week of the incident.

How it works

A DNS incident is a tree of diagnostics. The root is the user-visible symptom. The leaves are the four shapes. The team’s job is to walk the tree from the root to the leaf that matches the diagnostic.

   user-visible symptom (error rate or latency)
        |
        +-- check the blackbox panel
        |     |
        |     +-- success metric is 0
        |     |     |
        |     |     +-- read the resolver log
        |     |     |     |
        |     |     |     +-- SERVFAIL at the resolver
        |     |     |     |     |
        |     |     |     |     +-- authoritative server
        |     |     |     |     |     unreachable
        |     |     |     |     |     |
        |     |     |     |     |     +- SCENARIO 1
        |     |     |     |     |
        |     |     |     |     +-- SERVFAIL at the upstream
        |     |     |     |           |
        |     |     |     |           +- SCENARIO 2
        |     |     |     |
        |     |     |     +-- NXDOMAIN at the resolver
        |     |     |           |
        |     |     |           +- SCENARIO 3
        |     |     |
        |     |     +-- timeout at the resolver
        |     |           |
        |     |           +- SCENARIO 4
        |     |
        |     +-- success metric is 1
        |           |
        |           +-- resolution-time metric is amber
        |                 |
        |                 +- SCENARIO 5 (slow shape)
        |
        +-- check the network panel
        |     |
        |     +-- network is fine
        |           |
        |           +- SCENARIO 1-5
        |
        +-- check the application panel
              |
              +-- application is fine
                    |
                    +- SCENARIO 1-5

The diagnostic order is the order of the tree. The first node is the user-visible symptom. The second node is the blackbox panel. The third node is the resolver log. The fourth node is the scenario. The team’s job is to walk the tree from the root to the leaf.

How to configure it

A useful production layout includes the runbook, the escalation paths, and the postmortem template.

# /etc/prometheus/rules/dns.yml
groups:
  - name: dns_incident
    rules:
      # Page when the success metric is 0 for two minutes.
      - alert: DnsProbeFailed
        expr: |
          probe_success{job=~"blackbox_dns.*"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          runbook: "https://runbooks.example.com/dns/incident"
          summary: "DNS probe failed on {{ $labels.instance }}"
          description: |
            probe_success is 0 for 2 minutes. Open the
            blackbox panel for the instance. Read the
            resolver log. Run the per-step probes. The
            diagnostic order is the runbook.

      # Page when the resolution-time metric exceeds the
      # timeout. The probe is red; the failure shape is
      # the timeout.
      - alert: DnsResolverTimeout
        expr: |
          probe_dns_lookup_time_seconds{job=~"blackbox_dns.*"}
            > 1
        for: 2m
        labels:
          severity: critical
        annotations:
          runbook: "https://runbooks.example.com/dns/slow"
          summary: "DNS resolver timeout on {{ $labels.instance }}"
          description: |
            probe_dns_lookup_time_seconds is {{ $value }}
            for 2 minutes. The walk is timing out. Open
            the resolver log. Run the per-step probes.

The runbook:

# /runbooks/dns/incident.md

## Symptom
DNS probe is red. probe_success is 0 for 2 minutes.

## Diagnostic order
1. Read the blackbox panel for the affected instance.
   - probe_success = 0
   - probe_dns_lookup_time_seconds = ?
2. Read the resolver log on the resolver.
   - SERVFAIL at the resolver -> authoritative server
   - NXDOMAIN at the resolver -> delegation
   - timeout at the resolver -> upstream
3. Run the per-step probes.
   - system resolver -> upstream resolver -> authoritative server
   - The diff between the probes names the slow step.
4. Cross-check with `dig +trace` from the exporter host.
   - The trace shows the delegation chain.
5. Check the authoritative server's health.
   - Query the server directly.
   - Check the server's log.
6. Check the registrar's status page.
   - A domain hold is invisible to the resolver.
7. Escalate to the network team if the resolver is unreachable.
8. Escalate to the DNS provider if the authoritative server is unreachable.
9. Escalate to the registrar if the domain is held.

## Recovery
- Re-establish the delegation.
- Restore the zone from the operator's backup.
- Restore the resolver's forwarder config.
- Restore the registrar's status.

## Postmortem
- Timeline: when did the symptom appear?
- Root cause: which step in the chain?
- Contributing factors: which step in the runbook?
- Prevention: which probe or alert was missing?

The runbook is the operator’s friend. The team’s job is to follow the runbook.

The escalation paths

The escalation paths are the calls to the network team, the DNS provider, or the registrar. The team’s job is to escalate when the runbook says escalate.

The escalation is the right move when:

• The resolver is unreachable and the per-step probe shows the resolver is the slow step. The network team is the right escalation.
• The authoritative server is unreachable and the per-step probe shows the authoritative server is the slow step. The DNS provider is the right escalation.
• The domain is held and the registrar’s status page confirms the hold. The registrar is the right escalation.
• The walk is slow and the per-step probe shows the upstream is the slow step. The upstream operator is the right escalation.

The escalation is the wrong move when:

• The per-step probe shows the application is the slow step. The application team is the right escalation; the DNS team is the wrong escalation.
• The walk is fast and the application is slow. The application team is the right escalation.

The escalation is the right move when the runbook says escalate. The team’s job is to follow the runbook.

How to validate it

# 1. Confirm the alert is configured.
promtool query instant http://prometheus:9090 \
  'ALERTS{alertname="DnsProbeFailed"}'
# {} (empty, no active alerts)

# 2. Confirm the runbook is reachable.
curl -sf https://runbooks.example.com/dns/incident | head -1
# # /runbooks/dns/incident.md

# 3. Run the per-step probes by hand.
curl -sf "http://blackbox.internal:9115/probe?module=dns_portal_system_resolver&target=portal.example.com" \
  | grep -E '^probe_'
# probe_dns_lookup_time_seconds 0.412
# probe_success 1

curl -sf "http://blackbox.internal:9115/probe?module=dns_portal_upstream_resolver&target=portal.example.com" \
  | grep -E '^probe_'
# probe_dns_lookup_time_seconds 0.310
# probe_success 1

curl -sf "http://blackbox.internal:9115/probe?module=dns_portal_authoritative&target=portal.example.com" \
  | grep -E '^probe_'
# probe_dns_lookup_time_seconds 0.040
# probe_success 1

# 4. Validate the delegation with dig +trace.
dig +trace portal.example.com A | tail -10
# portal.example.com.  300  IN  A  104.16.45.7

# 5. Confirm the escalation paths are valid.
#    - Network team: #network on Slack
#    - DNS provider: support@cloudflare.com
#    - Registrar: support@namecheap.com

The runbook is the operator’s friend. The team’s job is to follow the runbook.

How it can fail

1. The diagnostic order is wrong. The team diagnoses the application before the DNS. Symptom: the team spends an hour on the application before discovering the DNS is the cause. Fix: follow the runbook.
2. The escalation is wrong. The team escalates to the DNS provider when the network is the cause. Symptom: the DNS provider’s response is “it’s not us”; the escalation is a deadlock. Fix: validate the escalation path with the per-step probe.
3. The runbook is missing. The team has no runbook. Symptom: the team invents the diagnostic order during the incident. Fix: write the runbook before the incident.
4. The postmortem is missing. The team has no postmortem. Symptom: the team is paged again for the same incident. Fix: run the postmortem within a week.
5. The probe is missing. The team has no probe for the affected endpoint. Symptom: the team learns about the incident from the user. Fix: add the probe.
6. The alert is missing. The team has no alert on the resolution-time metric. Symptom: the team learns about the slow shape from the user. Fix: add the alert.
7. The escalation is late. The team escalates after the SLO is breached. Symptom: the team breaches the SLO twice. Fix: escalate when the runbook says escalate.

How to troubleshoot it

The diagnostic order is the order of the runbook. The cheapest diagnostic is first; the most expensive is last.

1. Read the blackbox panel. The metric is the symptom. The team’s job is to confirm the DNS is the cause.
2. Read the resolver log. The log is the source of truth for the resolver’s view. The team’s job is to read the SERVFAIL.
3. Run the per-step probes. The diff between the probes names the slow step. The team’s job is to read the diff.
4. Cross-check with dig +trace from the exporter host. The trace is the delegation chain. The team’s job is to walk the chain.
5. Check the authoritative server’s health. The authoritative server is the next layer. The team’s job is to confirm the authoritative server is healthy.
6. Check the registrar’s status page. The registrar is the last layer. The team’s job is to confirm the registrar is healthy.
7. Check the upstream’s metrics. The upstream is the next layer. The team’s job is to confirm the upstream is healthy.
8. Check the resolver’s socket pool. The pool is the last diagnostic. The team’s job is to confirm the pool is not exhausted.
9. Escalate when the runbook says escalate. The team’s job is to follow the runbook.

Security implications

A DNS incident is a production outage. The security implication is the same as the production implication: the team’s job is to recover the platform in the shortest time possible without shipping the wrong fix.

The wrong fix is a fix that introduces a new vulnerability. The team’s job is to follow the runbook; the runbook is the recovery. The escalation is the call to the network team, the DNS provider, or the registrar. The team’s job is to escalate when the runbook says escalate.

The postmortem is the prevention. The team’s job is to run the postmortem within a week of the incident. The postmortem is the timeline, the root cause, the contributing factors, and the prevention. The team’s job is to identify the prevention.

Performance implications

A DNS incident is a production outage. The performance implication is the same as the production implication: the team’s job is to recover the platform in the shortest time possible without shipping the wrong fix.

The slow shape is the warning that precedes the failure shape. The team’s job is to catch the slow shape. The slow shape is the canary; the failure shape is the incident. The team’s job is to read the canary.

Production guidance

• Write the runbook before the incident. The runbook is the operator’s friend.
• Configure the alert on the resolution-time metric. The alert is the canary.
• Configure the alert on the success metric. The alert is the failure shape.
• Run the per-step probes. The diff between the probes names the slow step.
• Escalate when the runbook says escalate. The escalation is the call to the network team, the DNS provider, or the registrar.
• Run the postmortem within a week. The postmortem is the prevention.
• Add probes for every user-facing endpoint. The probe is the canary.
• Read the resolution-time metric. The metric that drifts is the metric that pages.

Verification

You should now be able to answer:

• What are the four common shapes of DNS incident?
• What is the diagnostic order for a DNS incident?
• When should the team escalate to the network team, the DNS provider, or the registrar?
• How does the per-step probe diff identify the slow step?
• What is the leading indicator for a DNS incident?

Quiz