DNS, routing and certificates during a failover

Active/passive and active/active settle where the capacity sits, and a running recovery site with no traffic on it is still an outage. Something has to move the clients, and the thing that moves them is not the failover decision, the runbook or the announcement in the incident channel. It is a small number of mechanical changes to name resolution, to routing, and to whatever sits in front of the service — each of which propagates at a speed set by systems belonging to other people. This lesson is about those mechanisms, the state they cannot reach, and the certificate problem that turns a technically successful cutover into a site nobody can connect to.

Three mechanisms, three different failure modes

The question a cutover answers is narrower than it looks: a client that was reaching one address has to come to reach another. There are three ways to arrange that, and they operate at different layers.

The first is to change the answer a name resolves to. The address changes; the name does not. This is the most portable mechanism, because it needs nothing from any network operator, and the slowest, because the answer is copied into caches all over the internet the moment it is first requested and each copy expires on its own clock.

The second is to change where the address itself is reachable. The name and the address both stay as they are, and the routing system is persuaded to deliver packets somewhere else — an anycast prefix advertised from a second location, or a virtual address moved between devices on a shared segment. Nothing in any resolver cache is stale, because nothing that resolvers hold has changed. What bounds this mechanism is the routing system and the people who run it.

The third is to change what the thing at the address does with the request. A reverse proxy, load balancer or traffic manager keeps a stable front door and selects a different backend behind it. This is the fastest and the most controllable of the three: no cache anywhere in the world needs persuading, and connections can be drained rather than dropped. Its condition is severe, though, and routinely unmet — the front door has to be outside the failure domain and reachable while the primary is not, or it fails with the site it was meant to survive.

Most estates use more than one of the three, layered: DNS points at a proxy, the proxy fronts a pool, the pool sits behind an anycast prefix. That is fine, and it has one consequence worth stating early. A layered cutover is only as fast as the slowest layer that has to change, and if any layer does not have to change, it drops out of the sum entirely. Moving the backend behind a stable proxy is a seconds-scale operation; moving a name whose answer has been cached worldwide is not, no matter how quickly the zone is edited.

A TTL is a parameter you set before you need it

A record’s time-to-live is the interval a resolver may keep reusing a cached answer before consulting the source again. That single sentence contains the whole operational problem, because it makes the population still using the old address a function of when each cache last refreshed — not of when you made the change.

Work the arithmetic. A name carries a 24-hour TTL. A recursive resolver somewhere refreshed its copy at 02:09. At 02:10 the site is lost and the zone is edited to point at the recovery address. That resolver will not ask again until roughly 02:09 the following day, and every client behind it arrives at a dead address until then. Nothing you do to the zone reaches it, because it is not asking.

This is exactly why the reflex of lowering the TTL at the moment of the cutover does not do what it appears to do. The new, smaller value is only read by a cache the next time it queries — and the reason it is not querying is that it is still holding the large value it fetched before you changed anything. A TTL reduction reaches a cache one full old-TTL after it is published, at worst. To have effect during an incident, it must therefore be published at least one old interval before the incident, which is a scheduling problem rather than an incident-response one.

The workable discipline is to decide, in advance, which names participate in a failover, and to keep those names permanently at the value you would want to have during the incident. That is not free. A short TTL multiplies query volume against your authoritative servers and tightens the estate’s dependency on them: clients re-ask often, and an answer they cannot refresh expires sooner. The trade is usually worth making for the handful of names a cutover actually touches, and rarely worth making across a whole zone.

There is a second, larger dependency hiding behind the same parameter. A TTL only matters if you can publish a new answer at all, and that requires the authoritative service for the zone to survive the disaster, along with the credential and the path used to edit it. An estate whose authoritative servers run only at the primary site has put its cutover mechanism inside its failure domain. Secondaries in a separate failure domain, and a rehearsed way to change the zone that does not traverse the failed site, are as much a part of the recovery estate as the application servers are.

What ignores the number you published

Even with the TTL set correctly and the zone editable, the published value is an instruction to caches you do not operate, and several of them are entitled to disregard it.

The clearest documented case is RFC 8767, Serving Stale Data to Improve DNS Resiliency, published in 2020. It permits a recursive resolver to answer from expired cache data in the exceptional circumstance that it is unable to refresh the information from the authoritative servers. The recommended parameters are a client response timer of 1.8 seconds, a stale answer TTL of 30 seconds on the answers handed back, and a maximum stale timer of between one and three days for how long expired data may be retained and used.

Read that against a site disaster and the interaction is uncomfortable. The failure in which your authoritative servers become unreachable is exactly the failure in which resolvers are permitted to keep handing out the old address for up to days, entirely regardless of the TTL you published. The resolver is not misbehaving; it is doing what a standards-track document tells it to do to improve resiliency. The defence is not a smaller number. It is keeping authoritative service answering from outside the failure domain, so that resolvers can refresh and never reach the stale path at all.

Client-side state is the other half. A process that resolved a name once at start-up and kept the result holds the old address for as long as it runs. Connection pools hold established sockets, and an established connection is not re-resolved — it continues to the address it was opened to until something closes it. Stateful firewalls and NAT devices hold their mappings on their own timers.

Anycast, routing and the proxy in front

Routing-based cutover avoids the caching problem completely, because the address the client holds never changes. The same prefix is advertised from more than one location and the routing system delivers each packet to whichever advertisement it prefers. RFC 4786, published as BCP 126, is the operational reference: availability of the service triggers the route advertisement, and non-availability triggers a route withdrawal.

The RFC is deliberately unwilling to promise a convergence time. Rather than prescribe rules, it says that the stability of the routing system, together with the transaction time of the service, should be carefully compared when deciding whether a service is suitable for distribution using anycast. That framing is the useful one for a failover plan: the question is not how fast routing converges in general, but whether it converges faster than your transactions take to complete.

The RFC is equally clear about what goes wrong. Operators should be aware that, especially for long-running flows, there are potential failure modes using anycast that are more complex than a simple destination-unreachable failure using unicast; when routing changes mid-transaction, different request packets associated with a single transaction might be delivered to more than one node. Its own suggested remedy is architectural — split large transactions into an initialisation phase handled by anycast servers and a sustained phase provided by non-anycast servers. Anycast, in other words, cuts over cleanly for short stateless exchanges and cuts through long-lived ones.

What anycast demands before it can be used at all is a short list that takes months rather than minutes to satisfy: address space you are permitted to advertise from both locations, a routing relationship at each location willing to accept that advertisement, and the operational ability to change the advertisement while the incident is running. That last item is where plans usually fail. If withdrawing the primary advertisement means opening a ticket with a transit provider, the convergence time of the cutover is that provider’s queue, and it belongs in the recovery time as such.

Reconfiguring a proxy or load balancer is the mechanism with the fewest moving parts and the tightest requirements. Because the client-facing name and address are untouched, the change takes effect for the next request rather than the next cache expiry, and connections can be drained instead of severed. The requirement is that the proxy tier survives the disaster it is arbitrating, and that its health checks are able to tell the difference between a site that is listening and a site that is serving. A check that confirms a port is open will send production traffic to a recovery site whose processes are up and whose data has not finished restoring.

Certificates the recovery site must already be able to serve

A cutover that succeeds at every layer above still ends at a TLS handshake, and three separate conditions have to hold there. Each of them fails as a hard error rather than a degradation, and the first two are settled long before the incident.

Coverage. Every name a client will actually present has to appear on a certificate the recovery site holds. That set is larger than the public web name: it includes failover-only names, the internal names used by service-to-service calls, and the names health checks use. A list of subject alternative names covers exactly what is enumerated in it, and a wildcard covers one label depth, so a recovery site introducing an extra label is not covered by the wildcard the primary was using.

Key presence. The certificate is public; the private key is not, and a recovery site holding the certificate without the matching key serves nothing at all. Placing that key at the recovery site is a deliberate pre-incident act with a real cost: the key now exists in a second location, so compromising the recovery site compromises the production identity. This is the same custody question the encryption-key material raised in Part IX, and it takes the same answer — a recorded custodian, a recorded location, and a retrieval that has been exercised without the primary site participating.

An issuance path that does not run through the failed site. Certificates expire on a calendar that takes no interest in your incident, and a recovery site that is rarely used still needs renewal. If the issuing authority runs only at the primary, then during a site loss nothing can be issued at all — which turns a certificate expiry inside the recovery window into an outage with no remedy available.

Automated issuance has its own ordering trap. Let’s Encrypt documents that the HTTP-01 challenge can only be done on port 80, that the validation server fetches a token beneath /.well-known/acme-challenge/ on the name being validated, that it follows redirects up to ten deep and accepts them only to http: or https: and only to ports 80 or 443, and that it cannot be used to issue wildcard certificates. The consequence for a failover is sharp: HTTP-01 validates against whatever the name currently resolves to, so a certificate for the recovery site cannot be obtained until the name already points at the recovery site. The dependency runs backwards through the cutover you are trying to perform.

DNS-01 inverts that. It requires a TXT record at _acme-challenge beneath the name, can issue wildcards, and depends on a DNS provider API — which moves the requirement from “the name already points here” to “the zone is writable during the disaster”, the same zone-control requirement the DNS cutover already had. The documentation also carries the warning that putting full DNS API credentials on a web server significantly increases the impact if that server is compromised, which is a reason to site the issuance credential deliberately rather than beside the application.

Validating from a client position outside the estate

Every internal check can pass while external clients still reach the old address, and the reasons are structural rather than unlucky. Split-horizon DNS answers internal queries from a different view than the world sees. The operator flushed the internal resolvers and nobody flushed the internet’s. Internal checks frequently reach the service by address, or over a path that bypasses the proxy and the WAN entirely. And the workstation running the check carries a trust store containing the internal root, so a chain that a customer cannot build validates without complaint.

Validation therefore has to be performed from a client position outside the estate. Ask several independent public resolvers directly and compare their answers, rather than asking the one your laptop is configured to use.

NAME=app.example.com
EXPECTED=203.0.113.40
for RESOLVER in 1.1.1.1 8.8.8.8 9.9.9.9; do
  ANSWER=$(dig +short "@$RESOLVER" "$NAME" A | tail -n 1)
  if [ "$ANSWER" = "$EXPECTED" ]; then STATE=moved; else STATE="still $ANSWER"; fi
  printf '%-9s %s\n' "$RESOLVER" "$STATE"
done

Then confirm what an outside client is actually handed at the address it reaches, for the name it asks about, which is a different question from whether the right file exists on the recovery host.

NAME=app.example.com
ADDR=203.0.113.40
echo | openssl s_client -connect "$ADDR:443" -servername "$NAME" 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

Two further checks are the ones teams skip. The first is a real transaction rather than a health endpoint: a login, a write, a read of something that had to be restored. A health endpoint answers whether a process is running, and the whole subject of this course is that a running process is not a recovered service. The second is the negative check — confirming that the old address has stopped serving. A partial cutover in which some clients still reach the primary and write to it is worse than a clean outage, because it produces two divergent datasets and hands the reconciliation problem to the failback.

Finally, measure the tail. The number that describes a cutover is not how quickly the zone was edited but how long after the change the last request arrived at the old address, and that number is only visible in the old site’s logs. Record it, because it is the figure that belongs in the recovery time, and it is the only evidence that the TTL discipline is working.

Production discipline

1. Set the failover TTL permanently, and monitor that it stays set. Decide which names participate in a cutover and hold them at the incident value at all times. A reduction published during the incident reaches a given cache one full old interval later, which is after the window it was meant to help.
2. Site the cutover control plane outside the failure domain. Authoritative DNS, the routing relationship or the proxy tier, and the credential and path used to change any of them, must all be reachable when the primary is not. A mechanism that fails with the site it was meant to move traffic away from is not a mechanism.
3. Plan for the tail, and make the old address fail usefully. RFC 8767 permits a resolver that cannot refresh to serve expired data for a recommended maximum of one to three days, and long-running processes, connection pools and hard-coded addresses obey no TTL at all. Arrange for the old address to refuse or redirect rather than accept.
4. Exercise the recovery site’s certificate path without the primary. Check name coverage, key presence and chain completeness, and prove that issuance or renewal works when the primary site and its issuing authority are unavailable. HTTP-01 cannot help before the cutover, because it validates against wherever the name currently points.
5. Validate from outside, validate the negative, and record the tail. Query several public resolvers directly, inspect the certificate an outside client receives, run a real transaction rather than a health check, confirm the old address has stopped serving, and write the measured time-to-last-request into the recovery time.

Cross-course references

• OPNsense for Production Network & Security Administrators — Part XVIII (DNS and Unbound) is the resolver side of everything above: it explains what a caching resolver does with a TTL and what a forwarder holds, which is the behaviour that decides how long this lesson’s cutover actually takes. Part XXX (Dynamic Routing) covers advertising and withdrawing prefixes, the operational skill the anycast cutover in this lesson depends on.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part X (ACME and Certificate Automation) is the full treatment of the HTTP-01 and DNS-01 challenges summarised here, and it is where to work out which challenge type your recovery site can complete without the primary participating.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part VI (Chains and Trust Stores) explains why a leaf served without its intermediates validates for you and fails for a clean client, and Part IX (Certificate Lifecycle and Revocation) covers the expiry and revocation checking that a rarely-used recovery site is most likely to get wrong.

Quiz