Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

intermediatebdr-dr-dns~45 min

The service is healthy in DR and customers still reach nothing

Reported symptoms

  • ●The failover of `rbdr-shop` to the recovery site completed at 09:38 and every step in the runbook passed: the application answers, the database is current to the declared recovery point, the load balancer reports every backend healthy, and the synthetic check the on-call engineer runs returns HTTP 200 against the recovery site.
  • ●From 09:50 the support queue fills with customers reporting connection timeouts on `rbdr-shop.example.net`. The rate climbs steadily through the morning instead of tailing off the way a propagation delay does.
  • ●Request volume at the recovery site sits at roughly 4% of a normal weekday morning, and almost all of it originates from the estate office range and from the monitoring probes.
  • ●A responder on a mobile connection, away from the corporate network, cannot reach the site either. The same responder on the office VPN reaches it on the first attempt, which is what convinces the bridge that something in the network path is selecting against external clients.
  • ●The primary site is confirmed down. Its address does not answer on any port and the incident that took it out is still open, so nothing is competing to serve the name.
  • ●No failed step exists to find. The change record shows the DNS update applied cleanly at 09:34 with no errors, and re-running the runbook verification step returns the recovery address every time anyone tries it.
  • ●A responder proposes that the recovery site upstream firewall or the provider routing is dropping customer traffic, on the grounds that internal clients succeed and external clients fail, and asks for a firewall change window.

Evidence

  • · Asked without recursion, every published authoritative name server for `example.net` returns `rbdr-shop.example.net` as the recovery address 203.0.113.45, with the `aa` flag set and identical SOA serials across all three. The change is live at the source, on every authority, and the secondaries have transferred it.
  • · The same question asked from a host outside the estate, using that host own provider resolver, returns the old primary address 198.51.100.20. Repeated 120 seconds later it returns the same address with the time to live 120 lower: 43891, then 43771. A value that counts down identifies an answer being served from a cache; an authoritative answer does not decrement.
  • · The time to live published on that record in the public zone is 86400. A resolver that fetched the answer at 22:33 yesterday is entitled to keep handing out the primary address until 22:33 today, and no operator action reaches inside it.
  • · The estate resolver never participated in that cache. It answers `example.net` from an internal authoritative copy whose records carry a time to live of 60, so every query made from inside the estate turned over within a minute of the change and has been correct ever since.
  • · Reaching the recovery address directly, with the name pinned to 203.0.113.45 from that same external host, returns HTTP 200, the expected body and a certificate that validates for `rbdr-shop.example.net`. The path from the public internet to the recovery site is open, the service answers on it, and the certificate is right.
  • · Step 14 of the failover runbook reads "Update DNS, then confirm the name resolves to the recovery address before proceeding". It does not say from where. The responder ran it on the jump host, which is the machine they already had a session on.
  • · The support tickets show no clustering by country, network operator or client type. They cluster by nothing, which is the shape of an effect distributed across resolver populations rather than the shape of a regional path fault.
Diagnosis and resolutionclick to reveal

Root cause

Two defects. One made the service unreachable; the other is the reason nobody saw it before the customers did. The immediate cause is a cached answer with hours of life left in it. The A record for `rbdr-shop.example.net` has been published with a time to live of 86400 for as long as the zone has existed. Every recursive resolver on the internet that looked the name up in the previous day holds the primary address 198.51.100.20 and is entitled to keep serving it, without asking anyone, until its own copy expires. Updating the authoritative record at 09:34 changed what a resolver will learn the next time it asks. It changed nothing for the resolvers that had already asked. Lowering the value now cannot shorten an answer that has already been handed out, because the countdown running in each cache was fixed at the moment that resolver fetched it. The failover therefore moved the service and did not move the customers, and the two events are separated by up to a full day. The second defect is the control that should have caught this within a minute of the cutover. `example.net` is served in two views. The public view carries the 86400 value the customers are subject to. The internal view, served to the estate resolver from a separate authoritative copy, carries 60, a choice made years ago so that internal renumbering would settle quickly. Step 14 of the runbook asks the responder to confirm the name resolves correctly and never says from which client position. Run on the jump host it queries the internal view, which had been correct since 09:35. The check was not weak; it was measuring a different system from the one that was broken, and it will pass during every future failover for the same reason. The consequence is not that a DNS change failed. The change succeeded. The consequence is that the recovery time the business experienced is set by a number chosen years earlier by whoever wrote the zone file, and no amount of speed in the failover itself can beat it.

Remediation

Accept first that a cached answer cannot be recalled. There is no protocol message that clears a third-party resolver, and the operators of the caches serving the customers are not reachable. Any plan that begins with purging caches is a plan to do nothing while believing otherwise. That leaves exactly one lever that acts inside the window: make the old address answer. If the primary address can be moved to the recovery site by reassignment, by a routing announcement or by an upstream translation, do that, and the customers holding the cached answer arrive at the recovery site without knowing anything changed. This is usually faster to arrange than it feels during an incident and it is the only remedy that helps the customers who are already affected. Republish the record with a short time to live in the same change, so that every resolver whose copy expires from now on takes a value that can be steered later in the incident. This does not help anyone currently stuck; it bounds the next change rather than this one. If the address cannot be moved, stop promising a restoration time the name cannot deliver. Communicate on a channel not bound to that name, tell customers the time the cached answers expire rather than the time the service came back, and offer any alternate hostname that was never cached. Repair the runbook in the same change, before the next failover inherits it. Step 14 must name the client position, and the confirmation must be performed from outside the estate against a resolver the estate does not operate.

Verification

The name resolves to the recovery address from at least three vantage points outside the estate, on networks the estate does not operate, and the answer at each is accompanied by a time to live consistent with the newly published value rather than a countdown from 86400. A request to `https://rbdr-shop.example.net/` from one of those external vantage points, with no address pinning and no VPN, returns the expected page and a certificate that validates. Resolution alone does not demonstrate this; the request has to be made the way a customer makes it. Request volume at the recovery site returns to the shape of a normal weekday rather than to a figure dominated by the office range and the probes. That is the measurement that says customers arrived, and it is independent of anything the responders queried. The repaired runbook step fails when it should. Prove this by running the new external check against a name deliberately left pointing at the old address and confirming it reports a mismatch, rather than by reading the step. The published time to live for every name in the recovery path is recorded with its value, read from the public view and not from the internal one.

Prevention

Treat the time to live as a pre-commitment made before the incident, not a setting adjusted during it. A name published at 86400 has a cutover floor of roughly a day no matter how quickly the failover itself runs, and the number was chosen when nobody was thinking about a failover. Reduce it on every name in the recovery path at least one old interval before a planned cutover, and keep the standing value for those names low enough that the figure is a deliberate trade against query volume rather than an inherited default. Require every validation step to name the client position it is performed from. This incident is a general failure mode wearing a DNS costume: a check run from a place that does not resemble the affected party will keep passing while the service is unusable. Synthetic checks for customer-facing names belong outside the estate, on a network the estate does not run. Record which names are served in more than one view, and which values differ between them. A split view is a legitimate design and an excellent trap: it makes the operator experience of the change permanently unlike the customer experience of it. Keep the authoritative name servers outside the failure domain of the service they name. This estate got that right, which is why the record could be published at all, and the failure mode it avoids is worse than the one that actually happened. Rehearse the failover including the name change, from outside, and record how long the answer took to turn over at external resolvers. That number is the part of the recovery duration nobody in the estate can shorten once the incident has started.

Reported symptoms

The failover of rbdr-shop finished at 09:38. Application healthy, database current to the declared recovery point, every backend green, the synthetic check returning 200. The bridge stood down.

At 09:50 the support queue starts filling with connection timeouts on rbdr-shop.example.net, and the rate climbs all morning rather than settling. Traffic at the recovery site is about 4% of a normal weekday, nearly all of it from the office range and the monitoring probes.

A responder on a mobile connection cannot reach the site; the same responder on the office VPN reaches it immediately. That observation convinces the bridge the problem is in the network path, and somebody asks for an emergency firewall change at the recovery site.

Nothing failed. The change record shows the DNS update applied at 09:34 without error, and the runbook’s own verification step returns the recovery address every time anyone runs it.

Evidence provided

Ask each published authority directly, with recursion off, so the answer comes from the source rather than from anything in between:

Read-only / Safewhat the authorities actually publish
$ for NS in ns1.example.net ns2.example.net ns3.example.net; do dig +norecurse +noall +comments +answer @"$NS" rbdr-shop.example.net A; done

All three return 203.0.113.45 with aa set and identical SOA serials. The change is live, everywhere, and the secondaries have it.

Now ask from a client position resembling a customer: a host outside the estate, on its own provider’s resolver.

Read-only / Safethe same name, asked from outside
$ ssh rbdr-probe-ext 'dig +noall +comments +answer rbdr-shop.example.net A; sleep 120; dig +noall +answer rbdr-shop.example.net A'

Both answers give 198.51.100.20 — the dead primary. The first reports a time to live of 43891, the second 43771. There is no aa flag on either.

The public zone explains the number; the internal zone explains why nobody noticed:

; public view — served to the internet
$TTL 86400
rbdr-shop       IN  A   203.0.113.45

; internal view — served to the estate resolver from a separate copy
$TTL 60
rbdr-shop       IN  A   203.0.113.45

And the firewall theory, tested rather than argued — pin the name to the recovery address and make the request from outside anyway:

Read-only / Safereach the recovery address without using the name
$ ssh rbdr-probe-ext "curl -sS -o /dev/null --resolve rbdr-shop.example.net:443:203.0.113.45 -w 'HTTP %{http_code} verify %{ssl_verify_result}\n' https://rbdr-shop.example.net/healthz"

It returns 200 with a certificate that validates.

Step 14 of the runbook reads, in full: Update DNS, then confirm the name resolves to the recovery address before proceeding.

Work the evidence before reading on

  1. The external answer decrements by exactly 120 over 120 seconds and carries no aa flag. What does a countdown tell you that a single reading cannot?
  2. The firewall theory explains why internal clients work and external ones do not. Which single piece of evidence kills it outright?
  3. The authorities publish the right address and the estate resolver returns the right address. Why did neither observation help?
  4. Somebody suggests dropping the record to a 60-second value right now. What does that change, and for whom?

Root cause

Take the firewall theory first, because it fits the symptoms perfectly and is wrong. The pinned request from the external host reached 203.0.113.45 over the public internet, completed a TLS handshake against a certificate that validates for the name, and got a 200 back. The path is open, the service answers on it, and the certificate is right. Nothing between the customer and the recovery site is dropping anything. What the customers cannot do is find the address at all.

The published answer had a day of life left in it

The check was run from the one place it could not fail

example.net is served in two views. The public view carries 86400. The internal view, on a separate authoritative copy the estate resolver is pointed at, carries 60 — chosen years ago so internal renumbering would settle fast.

Step 14 says confirm the name resolves and never says from where. Run on the jump host, it queries the internal view, which had been correct since 09:35. The check was not weak or skipped: it measured a different system from the one that was broken, and it will pass during every future failover for the same reason.

Resolution

Start from what is not available: a cached answer cannot be recalled. No message clears a resolver you do not operate, and these caches belong to other people. Any plan beginning with “flush the caches” is a plan to do nothing while feeling busy.

One lever acts inside the window — make the old address answer. If 198.51.100.20 can be moved to the recovery site by reassignment, by a routing announcement or by an upstream translation, the customers holding the cached answer arrive without knowing anything changed. Arrange that first: it is the only remedy that helps anyone already affected.

Republish with a short value in the same change so the next cutover is not hostage to this one:

Configuration changepublish a short value and confirm at the source
$ sudo rndc reload example.net && dig +norecurse +noall +answer @ns1.example.net rbdr-shop.example.net A

If the address cannot be moved, stop quoting a restoration time the name cannot deliver. Tell customers when the cached answers expire rather than when the service came back, on a channel not bound to that name.

Repair step 14 before the next failover inherits it: name the client position, and run the confirmation from outside the estate.

Verification

Read-only / Saferesolve and fetch the way a customer does
$ ssh rbdr-probe-ext 'dig +noall +answer rbdr-shop.example.net A && curl -sS -o /dev/null -w "HTTP %{http_code}\n" https://rbdr-shop.example.net/'

The name returns the recovery address from at least three external vantage points, each with a time to live consistent with the newly published value rather than a countdown from 86400, and the fetch returns the expected page on a certificate that validates.

Request volume at the recovery site returns to the shape of a normal weekday rather than a figure dominated by the office range and the probes — the measurement that says customers arrived, independent of anything a responder queried.

The repaired step fails when it should: run it against a name deliberately left on the old address and confirm it reports the mismatch.

Prevention

Set the time to live before the incident, not during it. A name published at 86400 carries a cutover floor of roughly a day however fast the failover runs. Reduce it on every recovery-path name at least one old interval ahead of a planned cutover, and keep the standing value a deliberate trade against query volume.

Make every validation step name the client position. This is a general failure wearing a DNS costume: a check run from somewhere that does not resemble the affected party keeps passing while the service is unusable.

Record which names are served in more than one view, and where the values differ. A split view is a legitimate design and an excellent trap: it makes the operator’s experience of a change permanently unlike the customer’s.

Rehearse the cutover from outside and time it. How long the answer takes to turn over at external resolvers is the part of the recovery nobody here can shorten once the incident has started.