← All break/fix scenarios in Backup & DR

intermediatebdr-dr-firewall~45 min

The recovered services can reach each other and nothing else

Reported symptoms

●Failover to the second site completed inside the planned sequence and every recovered service is healthy against every other recovered service: the order service reaches its database, its queue and its cache, and all internal health checks are green
●Settlement pricing calls to the third-party endpoint api.rates-vendor.example fail; the TLS handshake completes and the endpoint returns HTTP 403 with a body naming the source address it saw
●Staff sign-in fails because the LDAPS bind to the shared group directory at ldaps.identity.corp.example (192.0.2.40) never establishes a connection at all, returning connection refused rather than a timeout
●Both destinations were answering normally from the primary site less than an hour before the failover, and neither vendor nor directory team reports an outage
●The two failures present differently — one refused by the application, one refused by the network — which initially reads as two unrelated incidents
●The first theory on the bridge is that the recovery site edge firewall, rebuilt from IaC three weeks earlier, is missing its outbound rules

Evidence

· The NAT policy on rbdr-dr-edge translates the recovered application subnet to 198.51.100.24, while the primary site has always egressed from 203.0.113.10
· The 403 body returned by api.rates-vendor.example names 198.51.100.24 as the source address the vendor observed, which is only possible if the request reached the vendor
· The vendor portal shows one entry in the API key allowlist, 203.0.113.10/32, created when the integration was commissioned and never amended
· The group identity team supplies the source object group attached to their 636/tcp rule; it contains the primary site prefix only, and their platform rejects rather than drops non-matching sources
· A capture on the egress interface of rbdr-dr-edge shows the SYN leaving with source 198.51.100.24 and a RST arriving one round trip later from 192.0.2.40 itself, not from a local gateway and not as an ICMP administratively-prohibited message
· Connection tracking on rbdr-dr-edge holds an ESTABLISHED flow to the vendor address with bytes counted in both directions, which is what a completed TLS handshake and an HTTP response look like
· A control request over the same egress path to a public endpoint with no source allowlist succeeds, so the path, the NAT policy and DNS are all working
· The recovery site dependency inventory, generated from the asset register by filtering to configuration items the estate owns, lists neither the vendor endpoint nor the group directory
· The DR runbook step that pre-registers recovery site addresses with dependency owners reads its list from that same inventory, so it ran, reported success, and registered nothing

Diagnosis and resolutionclick to reveal

Root cause

Two defects. The first one caused the outage; the second one is the reason nobody knew it was coming. **The immediate cause.** Both far ends authorise callers by source address, and the source address is a property of the site, not of the service. The vendor binds the API key to 203.0.113.10/32 and the group identity team attaches an object group containing the primary prefix to their 636/tcp rule. Failover moved the workload to an edge that translates to 198.51.100.24, an address neither far end has ever been told about. Nothing about the recovered services changed; the identity they present to anyone outside the site did. The two failures look unrelated because the enforcement points sit at different layers. The vendor terminates TLS and refuses in the application, so the client sees HTTP 403. The identity team's firewall refuses before the handshake, so the client sees connection refused. Same cause, two surfaces. **The control that should have caught it.** The recovery site dependency inventory was generated from the asset register by filtering to configuration items the estate owns. That filter is exactly wrong for this question. A dependency is anything the service needs in order to work, and the two that broke are a commercial third party and a service run by another team in the group — neither is an owned asset, so neither appeared. The runbook step that pre-registers recovery addresses with dependency owners read from that inventory, found nothing external, and completed successfully. The alternative hypothesis, and the one the bridge started with, is that the recovery edge is blocking its own outbound traffic. It is eliminated by the packets. A local block leaves no flow in connection tracking, produces either silence or an ICMP administratively-prohibited message sourced from the local gateway, and never yields a response carrying the far end's address. Here the capture shows the SYN leaving the edge and the RST returning from 192.0.2.40, connection tracking holds an ESTABLISHED flow to the vendor with bytes both ways, and the 403 body quotes back the recovery site's own egress address. Traffic left. It was refused at the destination.

Remediation

Start the request that has somebody else's clock on it first. Ask the vendor to add 198.51.100.24/32 to the API key allowlist and the group identity team to add the recovery site prefix to their source object group, and be explicit that this is an addition, not a replacement: an edit that swaps the primary address for the recovery one converts this incident into a failback incident. While that is in flight, establish what the service can do without each dependency. Sign-in through the shared directory and settlement pricing are not equally degradable, and the answer determines whether the site is partially open or closed. An interim path exists only if the primary edge survives: outbound traffic for the two affected destinations can be routed through the registered address. Treat it as a stopgap and state the cost out loud — it makes the recovered estate depend on the site it just failed away from, which is precisely the coupling the second site exists to break. Then record both addresses in the dependency record for each far end, and re-run the registration step against the corrected inventory rather than the one that produced this outage.

Verification

The vendor call is repeated from a host inside the recovery site, not from a laptop or a jump box with different egress, and returns a success status instead of 403. The same request from the primary site still succeeds, which is what proves the entry was added rather than swapped. The LDAPS bind completes from the recovery site and the sign-in path is exercised end to end by a real account, because a TCP connection that establishes proves only that the firewall changed. Each far end supplies its allowlist as it now stands, showing both site addresses. A ticket that says the change was made is not the same artefact as the rule that enforces it. A day of observed egress destinations from the recovery site is diffed against the dependency inventory and the difference is empty. That is the check that says the inventory now describes reality rather than the asset register.

Prevention

**Build the dependency inventory from observed traffic, not from the asset register.** Every distinct destination outside the recovery boundary is a dependency whether or not the estate owns it, bought it, or can name its owner. An inventory filtered to owned configuration items cannot contain a vendor, and this one did not. **For every external dependency, record what the far end keys on.** Source address, API key bound to an address, client certificate, mutual TLS subject, DNS name. Then ask the only question that matters for DR: which of those changes when the workload moves sites. Source address always does. **Pre-register recovery site egress addresses permanently, in advance.** Registration is somebody else's change process with somebody else's lead time, measured in vendor working days, and a failover is the worst possible moment to discover that. Both addresses should be present in every allowlist at all times. **Prefer an identity that survives a site change.** Where the far end supports client certificates or workload identity, the credential travels with the service and the egress address stops being load-bearing. Where it does not, the address is a documented DR dependency with an owner and a renewal check. **Make the DR exercise reach outward.** An exercise scoped to the estate's own systems validates exactly the part of the plan the inventory already covered, which is how a control that omits every external dependency passes every rehearsal. **Consider presenting a stable egress address from both sites** — a shared NAT pool advertised from either location, or egress via a third site — while being honest that this makes that address a dependency in its own right, with its own failure mode.

Reported symptoms

Failover to the second site completed inside the planned sequence. Storage came up, the databases opened, the application tier started, and the order service reaches its database, its queue and its cache. Measured from the inside, the recovery worked; measured from the outside, the estate is deaf.

Settlement pricing calls to api.rates-vendor.example fail: the TLS handshake completes and the endpoint returns HTTP 403, with a body naming the source address it saw. Staff sign-in fails differently — the LDAPS bind to the shared group directory ldaps.identity.corp.example at 192.0.2.40 never establishes a connection, returning an immediate refusal rather than a timeout.

Both destinations answered from the primary site an hour earlier, and neither far end reports an outage. The two failures look different enough that the bridge opens with them as separate incidents, and the first theory is the recovery edge: rbdr-dr-edge was rebuilt from IaC three weeks ago, and somebody remembers a review comment about outbound rules.

Evidence provided

Read-only / Safethe address this site presents to the world— The NAT policy on rbdr-dr-edge. The primary site has always egressed from 203.0.113.10.

$ sudo nft list table ip nat

The recovered application subnet is translated to 198.51.100.24.

Read-only / Safethe vendor call, from a recovery site host— Run from inside the recovery site, so the request takes the same egress path as the failing service.

$ curl -sS -o /dev/null -w '%{http_code}' "https://$VENDOR_HOST/v1/rates"

The status is 403, and the response body names 198.51.100.24 as the source address the vendor observed. The vendor portal shows one allowlist entry for the API key: 203.0.113.10/32, created at commissioning and never amended.

Read-only / Safethe directory bind, on the wire— Capturing at the egress interface answers where the refusal is generated.

$ sudo tcpdump -ni "$WAN_IF" -c 20 "host $DIR_IP and port 636"

The SYN leaves with source 198.51.100.24. One round trip later a RST arrives from 192.0.2.40 itself — not from a local gateway, and not as an ICMP administratively-prohibited message. The group identity team supplies the source object group on their 636/tcp rule: it holds the primary site prefix only, and their platform rejects rather than drops.

Read-only / Safewhat the edge thinks happened to the vendor flow— Connection tracking state and byte counters for the flow to the vendor address.

$ sudo conntrack -L -d "$VENDOR_IP"

The flow is ESTABLISHED with bytes counted in both directions.

Read-only / Safea control destination with no allowlist— Same egress path, same NAT policy, a destination that authorises nobody by address.

$ curl -sS -o /dev/null -w '%{http_code}' "https://$CONTROL_HOST/"

It succeeds.

Finally, the recovery site dependency inventory, generated from the asset register by filtering to configuration items the estate owns. It lists neither destination. The DR runbook step that pre-registers recovery addresses with dependency owners reads that inventory — so it ran, found nothing to register, and reported success.

Work the evidence before reading on

One failure is a 403, the other a TCP refusal. What must be true of the path for either response to exist at all?
The 403 quotes an address back. Which one, and what does the vendor knowing it establish?
If the recovery edge were dropping this traffic, what would conntrack hold, and what would the capture show instead of a RST from the destination?
The pre-registration step completed successfully. What did it read, and what could that source never have contained?

Root cause

The far ends authorise by source address, and failover changed it

Both dependencies key access on where the traffic comes from: the vendor binds the API key to 203.0.113.10/32, and the identity team’s object group holds the primary prefix. Neither control is a property of the service; both are properties of the site. The recovered services present 198.51.100.24, which no far end has ever been told about.

That is the whole defect. The failures diverge only because the enforcement points sit at different layers: the vendor terminates TLS and refuses in the application, so the client sees a 403, while the identity team’s firewall refuses before the handshake.

The dependency inventory stopped at the boundary of the estate

The inventory was generated by filtering the asset register to configuration items the estate owns. That filter answers a question about ownership; the question that matters at failover is about need. Neither a commercial third party nor another team’s service is an owned asset. Both were needed; neither was listed. So the pre-registration step, reading that inventory, had nothing to act on and exited clean — and every DR exercise since passed with the same gap.

Resolution

Start the request that runs on somebody else’s clock. Ask the vendor to add 198.51.100.24/32 to the API key allowlist, and the identity team to add the recovery prefix to their object group. Say explicitly that this is an addition: an edit that swaps the primary address for the recovery one turns this incident into a failback incident three days from now.

While that is in flight, decide what the service can do without each dependency; settlement pricing and staff sign-in are not equally degradable, and that answer determines whether the site opens partially or stays closed.

An interim path exists only if the primary edge survives: route the affected destinations through the registered address. Say the cost out loud — it couples the recovered estate to the site it just failed away from.

Verification

The vendor call, repeated from a host inside the recovery site, succeeds. Repeated from the primary site, it still succeeds — that second check proves the entry was added rather than swapped.

The LDAPS bind completes from the recovery site and a real account signs in end to end, because a connection that establishes proves only that the firewall changed. Each far end supplies its allowlist as it now stands, showing both addresses.

A day of observed egress destinations is diffed against the dependency inventory, and the difference is empty. That is the check that says the inventory describes reality, not the asset register.

Prevention

Build the dependency inventory from observed traffic. Every destination outside the recovery boundary is a dependency, owned or not. An inventory filtered to owned configuration items cannot contain a vendor.

Record what each far end keys on — source address, API key bound to an address, client certificate, mutual TLS subject — then ask the question DR cares about: which of those changes when the workload moves sites. The address always does.

Pre-register recovery egress addresses in advance. Registration runs on somebody else’s change process at somebody else’s lead time; both addresses belong in every allowlist at all times.

Prefer an identity that survives a site change. Where the far end supports client certificates or workload identity, the credential travels with the service and the address stops being load-bearing.

Make the DR exercise reach outward. An exercise that only calls the estate’s own systems validates exactly the part of the plan the inventory covered — which is how this gap passed every rehearsal.