Declaring a disaster, failing over, and failing back

Restoring a system and declaring a disaster are different activities, and the course has spent an entire part on the first one. The boundary is worth stating precisely, because most of what goes wrong in DR goes wrong at exactly this line.

A restore rebuilds one system inside a working environment. DNS answers. The identity service authenticates. The network routes. The monitoring is watching. You are putting one piece back into a structure that is otherwise intact.

Disaster recovery replaces the environment itself. The DNS that would have pointed at the new host is in the site that is gone. So is the identity service that would have authorised the engineer doing the work, the certificate authority that would have issued the replacement certificate, and the monitoring that would have told you whether it worked.

Everything specific to DR follows from that difference: the decision to invoke it, the point of no return, the risk of both sites serving at once, and the return journey.

The decision

The most expensive minutes in a disaster are the ones spent deciding whether it is one. Two failure modes, both common:

• Declaring too late. The team spends three hours trying to repair the primary because each individual symptom looked fixable. The RTO clock has been running the whole time.
• Declaring too early. Failover happens, the primary recovers twenty minutes later, and now there is a failback to perform - a second migration, with its own risk, for an outage that would have resolved itself.

The way out is not better judgement under pressure. It is criteria agreed in advance, in writing, that can be applied without one.

DECLARATION CRITERIA - service: payments-api
Declare a disaster if ANY of the following holds:

  D1  The primary site has been unreachable from two
      independent external vantage points for >= 20 minutes
      AND the site operator has no restoration ETA.

  D2  The primary storage array reports an unrecoverable
      fault affecting the production LUNs, confirmed by
      the vendor.

  D3  Encryption of production data by an unauthorised
      party is confirmed on two or more hosts.

  D4  The estimated time to repair, from whoever owns the
      failing component, exceeds 90 minutes.

Authority to declare: the on-call incident commander.
No second approval is required. Escalation is a
notification, not a gate.

The commit point

Every failover has a step after which returning is no longer a switch back but a reconciliation. Find it, name it in the runbook, and put a checklist in front of it.

For almost every architecture, the commit point is the first write accepted at the recovery site. Before that, the recovery site is a warm copy and abandoning the failover costs nothing. After it, the two sites hold different data and any return requires deciding what happens to the writes each of them took.

1. Phase A - reversible. Provision, restore, start services bound to loopback or behind a closed firewall, run every verification. If the primary recovers here, stop and discard the work.
2. Phase B - the gate. Fence the primary. Confirm it cannot write to shared storage, cannot answer for the service name, and cannot authenticate as the service. This is the checklist, and it is the last reversible moment.
3. Phase C - commit. Open the recovery site to traffic and cut DNS or routing over. From this point the sites have diverged.
4. Phase D - operate. Run at the recovery site, with a known and growing delta that will have to be reconciled at failback.

$ dig +short payments-api.example.com @198.51.100.53; dig +short payments-api.example.com @203.0.113.53

203.0.113.40
203.0.113.40

Partial failover is a design decision, not a shortcut

Moving some services and leaving others is superficially attractive - fail over the database, leave the application servers - and it breaks in a specific way.

Every dependency edge that now crosses the WAN pays the WAN latency, per call. An application making 200 sequential queries per request over a link with 25 ms round-trip time adds five seconds to every request. The service is technically up and practically unusable, which is a worse position than being down, because it is harder to explain and harder to decide about.

Failback

Failback is the phase that is missing from most DR plans, and it is where data is most often lost - because by the time it happens the incident is officially over, the attention is gone, and it feels like a formality.

It is not a formality. It is a second migration, in the opposite direction, and it has its own RPO: everything written at the recovery site since the commit point. That delta is real business data and there is no backup of it at the primary.

1. Do not rush it. The recovery site is serving. Schedule the failback into a maintenance window rather than doing it the moment the primary is repaired.
2. Rebuild the primary as if it were new. It was down for an unknown reason and may be in an unknown state. Reprovision rather than restarting what was there, and reapply configuration management.
3. Reverse the replication. The recovery site is now the source of truth; the primary must be seeded from it, not the other way round. For PostgreSQL this is pg_rewind or a fresh pg_basebackup from the promoted node, never restarting the old primary and hoping it catches up.
4. Let it catch up and measure the lag until it is near zero. Failing back with an hour of lag discards an hour of data.
5. Quiesce writes at the recovery site, confirm zero lag, then cut over. This is a planned brief outage, which is the whole reason it belongs in a window.
6. Fence the recovery site, exactly as the primary was fenced in the other direction. Split brain is symmetric.
7. Verify at the primary with the same checks used at failover, then stand the recovery site back down to warm.

Testing the parts nobody tests

The last row is the one that gets cut for time, every year, and it is the one that covers the phase where data is most likely to be lost. A DR programme that has never rehearsed a failback has tested half of its plan.

Tabletop exercises are undervalued for the opposite reason: they are cheap, they need no infrastructure, and they reliably find the failures that are not technical - the approver who is not on a rota, the criteria that three people interpret differently, the runbook stored where the disaster will take it.

Knowledge check