Secrets, identity and third-party dependencies in DR

Names that point at the recovery site and certificates that validate there, which is where the previous lesson finished, deliver traffic to the door. They do not get anything through it. What remains is the set of dependencies the estate does not own and mostly never wrote down: the secret manager and the identity provider that everything else authenticates against, the credentials that must already be valid at the instant a process starts, and a group of external parties whose systems hold firm opinions about where your traffic comes from. Almost all of them are invisible during a rehearsal, and the reason they are invisible is this lesson’s central argument.

The recovery site comes up sealed

At the recovery site the secret manager exists as a replica, as a restored instance, or as a machine still to be built, and in all three forms it begins in a state where it holds encrypted data it cannot decrypt. The Vault seal documentation describes that state directly: a sealed instance knows where its storage is and can do nothing with the contents until a quorum of key shares reconstructs the key that decrypts the rest. Sealing is not a fault. It is the property that makes the storage safe to replicate offsite at all, and it is the same property that makes a recovery site inert until a human supplies something from outside it.

So the operative question is where that something is kept, and in most estates the honest answer is in the primary. The unseal shares live in the corporate password manager, which authenticates through the identity provider. The identity provider at the recovery site is either a replica that federates back to the primary or a restore — and restoring an identity store is a rollback of credential state, with all the consequences the identity bootstrap lesson worked through. The multi-factor service that guards administrative sign-in is a hosted product whose own administrative account is federated to the same provider. To unseal you must sign in; to sign in you need the identity provider; the identity provider reads its database password from the secret manager while starting. That is a loop, and a loop has no correct order to execute.

Two things make the recovery-site version of this loop harder than the cold-start version. The first is that the recovery site’s participation was granted by the primary. A replica was enrolled with a token or a client certificate the primary issued, and that material has a lifetime; when it stops being renewed the replica does not necessarily report anything alarming about itself, because from its own point of view it is running. The second is that the primary is normally reachable, so during every rehearsal each edge of the loop is quietly satisfied by the site that is supposed to be gone.

The test that settles a proposed remedy is the one from the bootstrap lesson, sharpened for failover: is the thing that satisfies this dependency inside the failure domain that just failed? Unseal shares in the primary’s password manager fail it. An emergency administrator account in the identity provider being recovered fails it. The runbook in the wiki fails it, because the wiki is a service that starts after the estate does.

Credentials that must already be valid before a process starts

Below the two platform systems sits a larger and duller population: the service accounts, machine identities and API credentials that individual processes consume. Their defining property in a failover is timing. A service does not merely need its credential to exist somewhere; it needs the credential to exist, to be valid, and to be reachable at the instant it starts, which places every issuer strictly upstream of every consumer even where the relationship is invisible in normal running.

Four shapes behave differently and fail at different moments. Static service account passwords survive being copied to a new site but not being rotated after the recovery-site configuration was written. Machine and workload identities are bound to a host, so a rebuilt or newly provisioned recovery-site host is a new identity that needs an enrolment step over an authenticated path to a running issuer — and that issuer was in the loop above. Short-lived tokens do not exist at rest at all and must be minted before anything can use them, which is why an estate that has correctly eliminated long-lived credentials has, by construction, made its issuer a total dependency of everything it runs. Mutual-TLS pairs fail closed at both ends, so a mismatch leaves two services individually healthy and unable to speak to each other.

Expiry deserves separate attention because it is the failure that a warm site accumulates while nobody touches it. Material provisioned when the recovery site was built has a validity window that keeps running whether or not the site is used, so a failover eighteen months later finds certificates, tokens and joined machine accounts that expired long ago. Reissuing them requires the certificate authority and the identity provider, which is the loop again, reached from a worse starting position.

The subtler timing problem is that a process can start successfully on a cached credential and fail later. A daemon that read a token at start-up and holds it for the token’s lifetime looks completely healthy for that lifetime; the failure arrives at the first renewal, well after the exercise has been declared a success. An exercise short enough to miss the renewal has measured the cache rather than the dependency.

Access controls keyed on an address that failover changes

Move outside the estate and a different class of dependency appears, one that is enforced by an organisation you cannot page. A bank accepts settlement files over SFTP from a named list of source addresses. A payment API applies the same rule. A partner site-to-site tunnel is configured with your gateway address as its peer and will not negotiate with another. A supplier pins the client certificate you enrolled with, and the recovery site holds a different one. A mail provider authorises your domain to send from specific addresses, published in DNS, so a failover that moves egress makes outbound customer notifications fail authentication at exactly the moment they matter most.

The general form is a policy conditioned on the network origin of the request, independent of the credential presented. AWS documents the mechanism plainly in its global condition context keys, where a key such as aws:SourceIp lets a policy grant access only to requests arriving from stated addresses; firewall allowlists, API gateway rules and appliance access lists are the same idea in other syntax. The credential is correct, the request is well-formed, and the answer is a refusal that reads like an authentication failure.

What makes this a disaster recovery problem rather than a networking one is where the entry lives. Typically it was added during onboarding, by a support engineer, in response to a ticket. It is not in the contract. The contract names an availability target and a support channel, and says nothing about how quickly a source address can be changed, who may request the change, or what happens when the request arrives from a domain that is itself failing authentication. So the lead time for the change is unbounded from your side and unaffected by the urgency of your incident.

The remedy is administrative and has to be done in advance. Build a register of external parties, and for each one record whether access is conditioned on source address, whether the provider will hold a second address permanently rather than swapping one for another, the measured lead time for a change, the named individuals authorised to request it, and how such a request is authenticated when your usual mail domain is not delivering. Then pre-authorise the recovery site’s ranges now, while it is a routine ticket rather than an emergency, and confirm the entry by using it.

SaaS and licences: dependencies with no recovery site of their own

A hosted platform is a peculiar dependency because it has exactly one instance from your point of view and you cannot build a second. If it is unaffected by your disaster you may still be locked out of it, because sign-in federates to the identity provider you are trying to recover, and the fallback local administrator, if one exists, is a credential someone has to be able to find. If it is affected by the same regional event, you have no action available at all; its restoration is on its own schedule and your recovery time inherits it unmodified.

Two further properties are routinely missed. The first is that hosted platforms hold estate state — the incident tracker, the source host, the DNS provider’s zone configuration, the CDN rules, the paging rotation — and for that data their export is your only backup, so it belongs in the backup inventory like any database. The second is that the tooling used to run a recovery is frequently hosted: if the runbook, the chat where coordination happens and the paging system all authenticate through the same federated login, the incident response capability shares a failure domain with the estate.

Licences are the same problem wearing different clothing. Licences bind to something durable — a MAC address, a machine identifier derived from hardware, a hardware token, or a licence server that must be reachable. Every one of those bindings is invalidated by the act of standing a service up somewhere else on new hardware, which is the act disaster recovery consists of. Some products additionally require an online activation exchange with the vendor, which assumes a network path from the recovery site that may not exist or may not be permitted. Some vendors grant disaster recovery use under the existing agreement, some require a separately purchased entitlement, and some require a phone call during business hours in another timezone.

So the licence register looks like the third-party register: the product, what the licence is bound to, whether the agreement grants recovery-site use in writing, what the reissue path is, and the date on which a licence was last actually issued for a recovery-site host during an exercise. The last column is the only one that constitutes evidence.

The exercise: make the primary genuinely unreachable

Every dependency in this lesson has the same detection method, and it is not a review. It is an exercise in which the primary cannot answer.

The distinction between stopped and unreachable is worth stating precisely, because the two produce different client behaviour. A stopped service refuses connections and the client learns immediately. An unreachable one accepts nothing and says nothing, so the client waits for its timeout, retries, and waits again. Real disasters produce the second, and it is the second that turns a start-up sequence into something much slower than the sum of its parts. An exercise that cleanly shuts the primary down is therefore easier to arrange and measures the gentler failure.

The faithful and safe way to produce unreachability is to remove the paths from the recovery site rather than to damage the primary. Blackhole its networks, then prove the blackhole holds instead of assuming it.

PRIMARY_NET=10.20.0.0/16
DR_HOST=dr-app-01

# refuse every route back to the primary before the drill begins
ssh "$DR_HOST" sudo ip route add blackhole "$PRIMARY_NET"

# prove the path is gone rather than assuming it
ssh "$DR_HOST" ip route get 10.20.4.9

Do the same for name resolution, so a recovery-site resolver cannot forward to a primary-site one, and withhold production credentials from the operators so that the only material available is the offline kit. Restart every process rather than letting running ones continue, because a running process is holding caches the drill is meant to invalidate. Then let the exercise run long enough to cross a token renewal and a certificate validity check, and capture what each host is still talking to while it runs.

DR_HOST=dr-app-01
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
EVIDENCE="/srv/dr/evidence/egress-${STAMP}.txt"

ssh "$DR_HOST" ss -tn state established > "$EVIDENCE"

Record each failure with four fields: the dependency, the layer it sits in, the moment it appeared, and whether the fix required somebody outside the organisation. The fourth field is the one that predicts real recovery time, because internal work compresses under pressure and an external lead time does not.

Recovery material only the estate can produce is not recovery material

Every item the exercise names — the unseal shares, the licence file, the DNS provider’s API credential, the partner contact list, the repository passphrase — is subject to a single test, and the escrow capture measured what happens when an item fails it. A repository’s passphrase was kept inside the directory the repository protected. The site was destroyed. The repository survived complete.

$ restic snapshots

Fatal: wrong password or no key found
>>> exit code: 12

The capture then had a recovery team add a second passphrase of its own to the same repository with restic key add, producing a second key — 66c34166d8443d16e8c5899fe3f792e749cb24fa3a90ac90bbe8348979b57d90 — alongside the production one, after which restic key list showed both opening the same repository and neither able to derive the other. Production was destroyed a second time.

$ restic --password-file /work/recovery-pass snapshots

ID        Time                 Host          Tags        Paths        Size
--------------------------------------------------------------------------
b96ba7cf  2026-08-28 14:04:52  17dffded9807              /work/prod2  38 B
--------------------------------------------------------------------------
Timestamps shown in local time
1 snapshots
>>> exit code: 0

The bytes were identical in both cases and so was the repository; the entire difference between exit code 12 and exit code 0 was custody. The same capture also exported Borg key material as a printed paper block, which survives the loss of every system that ever held the repository and is still useless without its passphrase — which is the argument for escrowing the two halves separately, to different custodians.

Generalise past encryption keys. Anything required to start the recovery must be producible without the estate, and the recovery site is part of the estate, so material held only there fails the test as surely as material held only at the primary. That is what makes the drill and the escrow one subject rather than two: the drill enumerates the material, and custody decides whether the material is reachable when the enumeration turns out to matter.

Production discipline

1. Run the failover exercise with the primary genuinely unreachable, not merely stopped. Blackhole its networks and its resolvers from the recovery site, withhold production credentials, restart every process so no cache survives, and run long enough to cross a token renewal — a rehearsal the primary can still answer measures the estate with the primary in it.
2. Treat the sealed secret manager and the federated identity provider as the first two items of the runbook, not as infrastructure. A sealed instance needs a quorum of shares supplied from outside it, and every proposed source for those shares has to survive the question of whether it sits inside the failure domain that just failed.
3. Inventory credentials by when they are needed and when they expire. Machine identities need enrolment against a running issuer, short-lived tokens do not exist at rest, mutual-TLS pairs fail closed at both ends, and material provisioned when the recovery site was built has been expiring ever since.
4. Register every external party with a source-address dependency, and pre-authorise the recovery ranges. The allowlist entry lives in an onboarding ticket rather than in the contract, so the change lead time is set by the other organisation; record it, name who may request the change, and confirm the second address by using it.
5. Hold recovery material where the estate cannot be the only source of it. The measured cost of getting this wrong is an intact repository returning Fatal: wrong password or no key found with exit code 12, against exit code 0 for the same repository opened with a passphrase kept elsewhere.

Cross-course references

• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XII (Secret Management Platforms) and Part XIII (Dynamic Credentials and Workload Identity) describe the steady-state design of the two systems this lesson finds sealed and unreachable at the recovery site, and Part XVIII (Incidents and Recovery) is where the unseal and reissue procedures assumed here are built in detail.
• Terraform for Production Sysadmins — Part XXVIII (Disaster Recovery and Resilience) plans the rebuild of a recovery estate, and Part XIX (Security: Credentials, Secrets, and Audit) covers the provider credentials that rebuild needs; both are downstream of this lesson, because a plan that runs at the recovery site still has to obtain credentials from systems that may be sealed.
• Git, CI/CD & GitOps for Infrastructure Engineers — Part XLIII (OIDC and Short-Lived Credentials) explains the exchange by which a pipeline obtains a credential with no long-lived secret, which is the mechanism that makes the issuer a total dependency during a failover, and Part XCVII (CI/CD Disaster Recovery) treats the delivery platform itself as a hosted dependency of the kind this lesson registers.

Quiz