Cluster DNS failure modes - what breaks when DNS is down

In a cluster, DNS is not optional - it is the substrate on which the cluster runs. A DNS outage cascades: monitoring fails, service discovery fails, health checks fail, the cluster’s quorum logic may fail. This lesson is about how DNS outages cascade in clusters and how to design for resilience.

How DNS breaks a cluster

A typical cluster depends on DNS for:

• Service discovery (find peers by name).
• Configuration (the cluster’s own configuration is loaded by name).
• Health checks (monitoring pings services by name).
• TLS (certificates are issued to names, not IPs).
• Authentication (LDAP, AD, SSSD use DNS SRV records).
• Cluster logic (Corosync, Pacemaker may use DNS).

A DNS outage takes all of these offline simultaneously. The result is not “DNS is down” but “the cluster has fallen apart for reasons nobody can identify”.

Cascade scenarios

Cascade 1: resolver outage

1. The recursive resolver goes offline.
2. Every cluster member fails to resolve every name.
3. Service discovery returns nothing.
4. Health checks return "host not found".
5. Monitoring reports every host as down.
6. Operators restart the cluster.
7. Restarts fail because they need DNS for cluster config.

Cascade 2: TTL stickiness

1. A service IP moves from host A to host B.
2. The DNS record is updated.
3. Recursive resolvers still have the old IP cached (up to
   the TTL).
4. For up to TTL seconds, traffic still goes to host A.
5. Host A is unhealthy but still receiving traffic.
6. Cluster reports split-brain; manual intervention
   required.

Cascade 3: DNSSEC validation failure

1. A misconfigured DNSSEC chain causes SERVFAIL.
2. Validating resolvers refuse to return answers.
3. Cluster members cannot reach each other by name.
4. Cascading failures follow.

Design for DNS resilience

Multiple resolvers

Every cluster member must have at least two resolvers configured:

nameserver 10.0.0.1
nameserver 10.0.0.2

If both are inside the cluster, they form a SPOF. Consider:

• Resolvers on different subnets.
• Resolvers on different hosts.
• One internal + one external fallback.

Local resolver on each host

Run a recursive resolver (Unbound, Knot Resolver) on each cluster member as a fallback. The local resolver:

• Does not depend on any other service.
• Caches answers for resilience.
• Reduces dependency on centralised infrastructure.

Split-horizon DNS

Different answers for internal vs external queries:

internal: db.example.com -> 10.0.0.5 (cluster-internal IP)
external: db.example.com -> 192.0.2.5 (public IP)

This lets the cluster use cluster-internal IPs while external users use public IPs. Configure the cluster’s resolver to return the internal IP for cluster hosts.

Cluster-aware DNS

Some clusters run their own DNS server (Kubernetes CoreDNS, Consul, etcd-backed dns). This:

• Eliminates external DNS dependency.
• Enables service discovery by name.
• Provides split-horizon by default.

The trade-off: more moving parts to operate.

TTL discipline

For cluster-internal records that may move (failover, rolling update), use short TTLs (60-300 seconds). For stable records, use medium TTLs (3600+ seconds).

Document the TTL strategy in the runbook for cluster failover.

Specific cluster dependencies

Corosync / Pacemaker

Corosync can use DNS for node names. If DNS fails, the cluster cannot establish membership. Configure:

grep -i dns /etc/corosync/corosync.conf

Either:

• Hardcode node IPs in corosync.conf (no DNS dependency).
• Use DNS with short TTLs and ensure resolver redundancy.

NFS

NFSv4 uses DNS for the nfs.example.com name. A DNS outage makes NFS mounts hang. Mitigation: use direct IPs in /etc/fstab for cluster-internal mounts, or use NFSv3 with fixed ports.

NTP

NTP servers are usually referenced by hostname. A DNS outage makes time sync fail, which can break TLS validation and authentication. Mitigation: configure NTP by IP, or have local fallback NTP servers.

Authentication (SSSD, AD, LDAP)

Central identity uses DNS SRV records to find domain controllers. A DNS outage breaks authentication, which breaks every service that requires authentication. Mitigation: cache credentials locally (SSSD does this), or run a local LDAP proxy.

Testing DNS resilience

Synthetic tests:

# Test the configured resolver
dig @10.0.0.1 example.com +short
dig @10.0.0.2 example.com +short

# Test direct (bypass local resolver)
dig @8.8.8.8 example.com +short

# Test a specific cluster service
dig db.example.com +short

# Test SRV records (Kerberos, AD)
dig _kerberos._tcp.example.com SRV

Chaos testing (in non-production):

• Stop one resolver. Confirm queries still work.
• Stop both resolvers. Confirm the cluster does not panic.
• Block outbound DNS. Confirm cached answers still resolve.

Runbook for a DNS outage

If DNS is down:

1. Confirm the outage: dig @<resolver> example.com.
2. Bypass local resolver: use IP addresses directly.
3. Identify the failing layer: resolver, network, or upstream.
4. Restore from backups if configuration is corrupt.
5. Lower TTLs before any planned change.
6. Document the incident.

Knowledge check