DNS, DHCP and IPAM as recovery dependencies · Backup & DR

Recovering the switch, router and firewall configuration restores the paths packets travel. It does not restore the names those packets are addressed to, and almost nothing in an estate is addressed by anything else: configuration files name their databases, certificates are issued to names, users type names, controllers dial names. Yet in most written recovery orders the resolvers appear nowhere, or once as a bullet saying “check DNS”. The result is a recovery that succeeds technically and stays invisible.

Zone content and resolver configuration are two different artefacts

“DNS is backed up” is one sentence covering two systems that fail separately, are owned separately, and restore by different procedures.

The authoritative side is the zone content: the records themselves, the SOA and its timers, the NS set, and — where the zone is signed — the key material and the signed zone. It also includes something held in no file on your server: the delegation. The NS records that tell the world which servers are authoritative for your zone live in the parent zone, held by a registrar or a provider, and reaching them means an account and a credential. If that credential lives only in the estate you are rebuilding, the delegation is unreachable exactly when it matters.

The resolver side is everything about who asks what: which resolvers each client uses, which forwarders those resolvers consult, the stub zones and split views that make an internal name answer differently inside the perimeter, the trust anchors, the search domains, and the mechanism distributing all of it — a static /etc/resolv.conf, a DHCP option, a host-local stub resolver, a container runtime writing its own file, a cluster DNS service. None of that is in the zone. A perfectly restored authoritative server nobody is configured to ask is not a recovered name service, and neither is a fleet of correct resolvers pointed at a zone with no content.

Zone content also decays out of version control in a way config-managed files usually do not. A committed zone file is a backup of what was committed. Records created afterwards by dynamic update — a DHCP server registering a client, a certificate client writing a validation record, a controller publishing an endpoint — are applied to the zone the server is running, not written back to the checked-in file, and a rebuild from the repository discards them silently. The reliable capture of what a zone holds right now is therefore a request the running server answers: the zone transfer RFC 1035 defines for secondaries, taken on a schedule to a file stored alongside the repository rather than instead of it.

Two further pieces belong in the same bundle: the means of changing who is authoritative — the registrar or provider account, its API credential, and the person permitted to use them — and the resolver inventory, because a rebuild must prove clients are asking the server you just recovered.

A service reachable only by an address is not recovered

Recovery teams run one test and the business runs another, and the gap between them is where a great many “successful” recoveries sit.

The team’s test is a request to the restored host by address, usually with certificate verification switched off because the certificate does not match an address. It proves the process is listening and the data is there. It is not the user’s test. The user’s client resolves a name, validates a certificate issued to that name, and sends a request whose Host header or TLS SNI decides which virtual host answers. An address-only request skips all three, and where one proxy fronts several sites it can only reach whichever backend that proxy has been told to treat as its default.

Machine clients are worse, because they are not present to improvise. Service configuration is written in names almost everywhere — the database endpoint, the broker’s bootstrap list, the object storage endpoint, the identity provider’s issuer URL — and where cluster members identify each other by name, a peer arriving as anything else is not the peer the membership list expects to see. The backup repository is usually addressed by a name too, so a recovery whose resolvers are down cannot find the repository it is meant to restore from: the dependency cycle from the recovery graph lesson, wearing a different hat.

The consequence is a rule about what closes a recovery step. The evidence that a service is recovered is a name-based request from a client that resolves the way a real client resolves — same resolvers, same view, same trust anchors — with certificate validation on. Anything less validates the restore, not the recovery.

Time to live is a recovery-time parameter, and it is set in advance

Failing over is, at the last step, a change to a record. How quickly that change reaches the people who need it is governed by one number: the time to live the record carried when it was last answered.

That number is a standing trade-off. A long TTL means fewer queries and more tolerance of the authoritative servers being briefly unreachable — answers keep working while the servers do not. A short TTL reverses both: every client rechecks constantly, so an authoritative outage becomes a client outage far faster. The usual sensible position is a long TTL in steady state, which is precisely the position that makes a failover slow.

What a cached answer carries, and why lowering a TTL afterwards cannot reach it

A DNS answer is a set of records and, attached to each set, a duration in seconds. RFC 1035 defines it as the time interval that the record may be cached before the source of the information should be consulted again. A caching resolver stores the answer and counts that interval down; it holds no pointer to the zone, no version, and it does not check back before the countdown reaches zero. Nothing in the protocol lets an authoritative server tell a resolver that an answer it already handed out has changed.

This is why sequence matters. The TTL is a property of the answer at the moment it was served. Lowering the TTL in the zone changes the answers given from now on; it cannot reach a resolver already holding a copy stamped with the old value. A resolver that cached a 24-hour answer one minute before you lowered the TTL keeps serving that answer for almost 24 hours, and only then learns the new value. The reduction therefore has to land at least one full old-TTL before the change it is meant to accelerate — and because many resolvers each cached at different times, one old TTL is the floor, not the estimate.

Absence is cached too. RFC 2308 specifies that a negative response — the name does not exist, or exists with no record of that type — is cached for an interval derived from the zone’s SOA record, bounded by the SOA MINIMUM field and by the TTL of the SOA record itself. A client that asks while the zone is missing or the record not yet created remembers the absence, and keeps remembering it after the record appears.

Two more SOA timers belong in a recovery discussion. refresh and retry govern how often a secondary asks its primary for a new copy of the zone. expire is the one to write down: it is how long a secondary keeps answering for the zone while it cannot reach the primary at all, after which it stops serving that zone. A primary down longer than expire takes the secondaries with it, turning a single-server outage into a zone-wide one on a timer nobody was watching.

The record’s own TTL is not the only clock, either. The delegation NS set in the parent zone carries a TTL you do not control, bounding how fast a change of authoritative servers takes effect, and above the resolvers sit application-level caches — wherever a runtime resolves a name once and holds the address for the life of the process. No TTL planning reaches those; only restarting the client does.

Keep an inventory of the records that participate in a failover — ingress addresses, the database endpoint clients use, the API name, the VPN endpoint, the mail exchangers — and record two TTLs for each: the steady-state value and the change-window value. Reducing them is a numbered step before the window opens; raising them is a numbered step after it closes, because a reduced TTL left in place changes the failure characteristics of the whole zone.

Read both values directly rather than trusting what was configured. A recursive query shows what a shared resolver is handing out and how much countdown remains; the same query against an authoritative server with recursion disabled shows what the zone says right now.

NAME=api.example.com
RESOLVER=192.0.2.53
AUTH=ns1.example.com

# What the shared resolver is serving, and how long it will keep serving it.
dig +noall +answer "$NAME" A @"$RESOLVER"
sleep 5
dig +noall +answer "$NAME" A @"$RESOLVER"

# What the zone itself says at this moment.
dig +noall +answer +norecurse "$NAME" A @"$AUTH"

Where the second answer comes from the same cache as the first, its TTL is lower by roughly the seconds that passed between the two. That countdown is what a failover actually races, and it started before you were involved.

Anti-pattern: we will lower the TTL when we need it

The failover plan opens with a step that reduces the TTL on the records being repointed. It reads as prudent, it is in the runbook, and it does nothing for the callers who are the reason the failover is happening — every resolver holding an answer keeps holding it for the remainder of the interval it was given, and the new low value applies only to whoever asks next.

A TTL reduction works if and only if it precedes the incident by at least the old TTL, which puts it on the maintenance calendar rather than on the incident timeline. Where the failover cannot be foreseen, the honest options are a permanently low TTL on those records — accepting the extra query load and the tighter dependency on the authoritative servers — or a failover that does not depend on a record change at all, such as an address that moves.

Lease databases and IPAM records decide whether the estate gets its addresses back

Addressing carries the same split: configuration on one side, live state on the other, and only one comes back from a repository.

A DHCP server’s configuration is its subnet declarations, pools, option sets and — depending on the implementation — its reservations. A rebuild from configuration management reproduces all of that and comes up with an empty lease database. The bindings are the state: which client holds which address, and until when. RFC 2131 makes the lease a duration held by both parties, so a client whose lease has not expired keeps using its address regardless of what the server remembers. A server with no record of that binding is free to offer the same address to somebody else, and the duplicate assignment presents as an intermittent, address-specific fault that looks nothing like a DHCP problem.

The reverse ordering is equally awkward. A client that renews while the server is rebuilt can be refused and pushed onto a different address, breaking everything keyed to the old one: the firewall rules restored in the previous lesson, monitoring targets, access control lists, any static record naming it. And where the DHCP server writes forward and reverse records on a client’s behalf, those records exist only in the zone the name server is running, so the lease database and the zone have to agree or the estate ends up with names pointing at addresses their holders no longer have.

IPAM is the third piece and the one most often kept somewhere that does not survive. It records which addresses are allocated, to what, and which are free — the source saying this range is a pool, that range static, this address a load balancer’s and never to be handed out. Reconstruct a firewall from configuration and you get rules referencing addresses; reconstruct hosts from infrastructure as code and you get machines asking for addresses. IPAM makes the two sets agree, and without it that reconciliation is done by hand, under pressure, by whoever remembers most.

Classify addressing state by how it is recovered, not by which team owns it

Three questions separate the artefacts cleanly. Which of this comes back from the configuration repository? Which is a live database that must be backed up and restored like any other? Which exists only inside the estate being recovered?

A lease database and an IPAM database are databases, and they earn the backup treatment any database earns, including a restore that has actually been performed. An IPAM that is a spreadsheet on a file share, or a wiki page authenticated against the identity provider you are rebuilding, is a recovery artefact stored inside its own failure domain.

Reconciliation after the rebuild is then a step with an owner: does the address plan the restored configuration assumes match what is assigned now?

A resolution failure looks exactly like a total outage

Name resolution is also missing from recovery plans because its failures are misread when they happen.

When resolution stops, everything stops at once, from every client, with no pattern. Requests fail, agents lose their backends, dashboards go blank because the monitoring resolves names too, and the tickets say the site is down. The signature is indistinguishable from a total service outage, and worse, the tooling used to investigate also depends on resolution, so the investigation starts failing in ways that read as more evidence of a platform failure. The on-call engineer reaches for the application runbook, and nothing in it applies.

Three queries settle it, and they belong in the runbook rather than in somebody’s memory. Ask the client’s own resolver for a name that should work. Ask the authoritative server with recursion disabled, sidestepping every cache between you and the zone. Then contact the service by address, separating “the name does not resolve” from “the service is not there”. If the name fails and the address works, this is a resolution incident and the application runbook is the wrong document.

That fixes resolution’s place in the recovery order. It is an early node in the dependency graph and it needs its own validation gate — a query that must return the right answer before the next step begins. The gate cannot be “the monitoring went green”, because the monitoring resolves names through the thing being validated, and a check that depends on its subject reports success at the wrong time.

Production discipline

Back up zone content and resolver configuration as separate artefacts with named owners. A zone transfer captures what a zone actually contains, dynamically written records included; the checked-in file captures only what was committed. The resolver inventory, the trust anchors and the credential controlling the delegation belong in the same bundle, and none of them is recoverable from the others.
Reduce the TTL on failover-path records before the window, and restore it after. A reduction applied during an incident reaches only clients that had not already cached; answers in circulation keep their original countdown. Record a steady-state and a change-window TTL for every record in that path.
Treat the DHCP lease database and the IPAM record as data, not configuration. A configuration-only rebuild produces correct pools and no bindings, which is how a rebuilt server hands out an address a client is still legitimately using. Back them up and restore them as the databases they are, then reconcile the address plan against reality.
Close a recovery step with a name-based check from a realistic client. A request to an address with certificate validation disabled proves the process is listening. It does not exercise resolution, certificate validation, the Host header or SNI routing, and it is not the test the business runs.
Put resolution early in the recovery graph with its own validation gate. Its failures present as total outages, so the runbook needs the three queries that tell them apart, and the gate proving resolution works must not itself depend on the resolution it is proving.

Cross-course references

OPNsense for Production Network & Security Administrators — Part XVII (DHCP) and Part XVIII (DNS and Unbound) are where the lease database, the reservations and the resolver views discussed here are actually configured; that material describes the running system, while this lesson is about which parts of it survive a rebuild from configuration alone.
Linux for Production Sysadmins — Part XXIII (DNS) covers the client-side half of the split this lesson insists on, from /etc/resolv.conf and stub resolvers to search domains, and Part LXIII (Cluster Time, DNS and Identity Dependencies) treats those same services as start-up constraints, which is exactly how the recovery order has to treat them.
Observability for Production Sysadmins — Part LXV (DNS Monitoring) builds the probes that turn a resolution failure into its own distinct alert instead of a wall of red application checks, which is the detection gap that makes the misdiagnosis in this lesson’s final section so common.