Dependencies: DNS, certificates, identity - the cascade risks

DR recovery often fails not at the data restore but at the dependency layer. DNS does not resolve, certificates have expired, identity is not available. This lesson covers the cascade risks and how to design for them.

Common cascade failures

• DNS: restored hosts cannot reach the central authentication server, cannot resolve external services.
• Time: freshly provisioned hosts have no time source, so clocks drift outside the window that Kerberos and TLS tolerate.
• Certificates: restored TLS certificates have expired.
• Identity: restored hosts cannot authenticate to central identity (LDAP, AD).
• Secrets: encryption keys are in a vault that the restored network cannot reach.
• Network: routing or firewall changes are needed to reach backup services.

Each of these can independently block recovery.

DNS in DR

For DR, the DNS infrastructure must be available:

• Run DNS on the recovery site (read-only copy of zone).
• Use a managed DNS service (Route 53, Cloudflare) that is inherently multi-region.
• Keep a bootstrap /etc/hosts for the handful of names the recovery itself depends on.

The bootstrap file is the one control here that works with no resolver at all. nsswitch.conf consults files before dns, so these entries resolve even when every nameserver is unreachable:

# The names the recovery procedure itself needs, and nothing else.
# Keep it short - every entry here is a stale record waiting to happen.
sudo tee -a /etc/hosts <<'EOF'
10.20.0.10  vault.example.com
10.20.0.11  ldap.example.com
10.20.0.12  backup.example.com
EOF

# Verify the resolution order actually honours it
getent hosts vault.example.com
grep '^hosts:' /etc/nsswitch.conf      # expect: files ... dns

Lower the TTLs on the records you plan to fail over before the cutover, not during it — you cannot shorten a record that resolvers have already cached:

dig +noall +answer +ttlid app.example.com

When restoring, ensure the recovery site can resolve critical names — and test that by stopping the resolver, not by assuming.

Time in DR

Time is the dependency nobody puts on the list, and it sits underneath two that everybody does.

Kerberos and Active Directory reject a ticket whose timestamp is outside a five-minute skew window. So a recovery site whose clocks are wrong cannot authenticate anyone — even though identity is up, replicated, and answering. TLS validity checks compare notBefore and notAfter against the local clock, so a badly skewed host rejects a perfectly valid certificate, or accepts an expired one. TOTP break-glass codes are derived from the current 30-second window and fail the same way. And log correlation across the recovery timeline — the thing you need most while running an incident — is unusable when hosts disagree about when events happened.

Freshly provisioned instances and hardware coming up cold are exactly where clocks are wrong. A machine that has been powered off for a year comes back with a dead RTC battery or a BIOS default. A cloud instance in a new region may have no route to the NTP server the golden image was baked with.

Verify time before attempting any identity or TLS operation:

# Is the clock disciplined, and how far off is it?
chronyc tracking          # System time offset small; 'Leap status: Normal'
chronyc sources -v        # At least one source with a '*' (selected)
timedatectl status        # 'System clock synchronized: yes'

chronyc tracking is the one that answers the question. A “System time” offset of a few milliseconds and Leap status: Normal means the clock is disciplined. A large offset, or Leap status: Not synchronised, means chrony is running but has not agreed with anything — which looks identical to healthy in systemctl status chronyd.

The order matters. The recovery site needs a reachable time source provisioned before identity, not alongside it, because identity cannot be validated without it. Options, in decreasing order of resilience:

• A stratum-1 or stratum-2 source in the recovery site (GPS appliance, or the hypervisor/cloud provider’s local NTP endpoint such as 169.254.169.123 on AWS).
• The recovery site’s domain controllers peered to an external pool, with hosts pointed at them — matching the primary site’s topology so nothing changes at cutover.
• A public pool, only if the recovery site has outbound access and you have confirmed it is not blocked. UDP/123 egress is a common firewall omission in a site that is normally idle.

Certificates in DR

TLS certificates have expiry. A backup from 6 months ago may have a certificate that expired in between.

Solutions:

• Short certificate lifetimes: 90 days or less.
• Automated renewal: Let’s Encrypt, internal CA.
• Backup the certificate and key: the backup must include the private key (encrypted).

# Check certificate expiry
openssl x509 -enddate -noout -in /etc/ssl/cert.pem

If the certificate is close to expiry, renew before DR.

Identity in DR

AD/LDAP may not be available during a DR. The restored host cannot authenticate users.

Solutions:

• Local fallback accounts: every host has a local break-glass account.
• Cached credentials: SSSD caches credentials for offline use (covered in Part XXVII).
• Identity in the recovery site: replicate identity to the recovery site.

Secrets in DR

Encryption keys, API tokens, certificates - all live in secrets management. The recovery site must have access.

Solutions:

• HashiCorp Vault with disaster recovery replication.
• AWS Secrets Manager (inherently multi-region).
• Local copies of critical secrets (encrypted).

Network in DR

The recovery site’s network must reach:

• Backup storage (S3, NFS, etc.).
• Vault or secrets manager.
• Identity provider (AD, LDAP).
• DNS resolvers.
• Internet (for updates, package repos).

Solutions:

• VPN to the recovery site.
• Outbound internet access (for the recovery site).
• Pre-configured firewall rules.

Test the dependency chain

The DR test should verify that all dependencies are reachable:

- DNS resolves
- Time synchronised (chronyc tracking, offset small, Leap status: Normal)
- TLS certificates valid
- Identity available
- Secrets accessible
- Network connectivity
- Backup storage reachable

A test that misses one of these is incomplete. Time is listed second on purpose: certificate validity and identity are both evaluated against the local clock, so verifying them on a skewed host tells you nothing.

Knowledge check