DNS cluster failure impact - what happens when DNS is down

DNS is a dependency for almost every cluster service. A DNS failure cascades through every component that uses hostnames. This lesson covers the impact and the mitigation.

The cascade

DNS resolves hostnames to IPs. When DNS is down:

• Service A cannot reach service B by name.
• Service A cannot reach external services (e.g. apt mirror).
• Service A’s health check fails (cannot resolve its own name).
• Monitoring cannot resolve service names.
• Alerts cannot reach on-call by name.

Every component that uses hostnames is affected.

What depends on DNS

In a cluster:

• Corosync / Pacemaker: nodes identify each other by name. DNS failure breaks cluster membership.
• Database replication: primary connects to replica by name. DNS failure stops replication.
• Application services: connect to database, cache, other services by name.
• Monitoring: Prometheus, Grafana, etc. resolve service names.
• Logging: log shippers resolve destination names.
• Package management: apt or yum resolves mirror names.

A DNS outage affects all of these.

Symptoms

• Service connections fail with “name resolution failed”.
• Health checks fail (cannot resolve own name).
• Monitoring shows services as down (they are not - DNS is).
• Logs are missing (log shippers cannot connect).
• Updates fail (cannot resolve mirror).

The cluster is “broken” but the services are actually running. The failure is in the dependency.

Mitigation

Multiple DNS resolvers

Always configure at least two DNS resolvers. A single resolver is a single point of failure.

Local caching resolver

A local caching resolver (systemd-resolved, unbound) provides DNS even if external resolvers are down. The cache holds recent responses.

Service discovery

Use service discovery (Consul, etcd) for internal services. Service discovery is more resilient than DNS for internal service resolution.

IP-based fallback

For critical paths, configure IP-based fallback. The service tries DNS first, then falls back to IP.

Test for DNS failure

Quarterly, simulate a DNS failure:

# Block DNS at the firewall
sudo iptables -A OUTPUT -p udp --dport 53 -j DROP
sudo iptables -A OUTPUT -p tcp --dport 53 -j DROP

# Verify services still work
curl http://service.local/

If services fail with DNS down, the cluster is not resilient.

Knowledge check