Runbook: Disaster recovery - full cluster loss

This runbook covers the worst-case DR scenario: total cluster loss. The procedure is detailed; the discipline is quarterly testing.

When to use this runbook

Use this runbook when:

• An entire cluster or region is down.
• Recovery from a single host or service is not possible.
• A major event (data centre loss, ransomware) requires a full rebuild.

Inputs

Gather before starting:

• The recovery site (cloud region, alternate data centre).
• Access to backups (offsite, immutable).
• Access to vault or secrets manager.
• Communication channels (war room, on-call).
• Decision authority (who can authorise the cutover).

Procedure

Phase 1: Communicate (0-15 min)

- Page on-call incident manager
- Open war room (video + chat)
- Notify: security, operations, business stakeholders
- Identify scope:
  - Which hosts are down
  - Which services are affected
  - What data is at risk
- Estimate RTO impact
- Communicate to customers (status page)

Phase 2: Provision (15-60 min)

- Verify recovery site reachable
- Provision instances per host (terraform, ansible, etc.)
- Configure networking:
  - VPC, subnets, routing
  - Security groups
  - DNS resolvers
- Set up bastion access
- Verify backup access
- Verify time sync (chronyc tracking) on every provisioned host

Time is a Phase 2 item, not a Phase 3 one. Kerberos tickets and TLS notBefore/notAfter windows are evaluated against the local clock, so restoring identity onto a skewed host proves nothing: authentication fails fleet-wide with a generic pre-authentication error while DNS, the vault and the domain controllers all look healthy. Freshly provisioned hosts in an idle recovery site are exactly where clocks are wrong and where UDP/123 egress is commonly blocked. A large System time offset, or a Leap status other than Normal, is a blocker for that host - not a warning.

Phase 3: Restore (60-180 min)

For each host:

1. Install OS (autoinstall)
2. Apply configuration management
3. Restore /etc from backup
4. Restore data from backup
5. Restore secrets from vault
6. Start services
7. Verify

Run hosts in parallel where possible. Sequence for dependencies (e.g. database before application).

Phase 4: Validate (180-240 min)

For each service:

- Smoke test (curl /health)
- Synthetic monitoring
- Data integrity check (compare with last known good)
- Performance check (latency, throughput)

Phase 4.5: Isolate the failed region (before any cutover)

This phase is a hard gate. Do not proceed to Phase 5 until it is confirmed and timestamped.

- Disable the original region's load balancer pool / health checks
- Revoke or rotate the original region's database write credentials
- Demote the original database to read-only, or stop it
- Stop or power off the original instances via the cloud API / OOB
- Block the original region's replication and API egress
- RECORD: who confirmed isolation, by what method, at what time (UTC)

Pick whichever of these you can actually execute. A region that is unreachable is not the same as a region that is fenced: an unreachable region can come back. Power events and network partitions - the most common triggers for this runbook - very often end with the original region returning to life halfway through your cutover.

Phase 5: Cutover (240 min)

- Confirm Phase 4.5 isolation timestamp is recorded
- Update DNS to point to new region
- Update load balancer pool
- Verify traffic flows
- Watch the fenced region for any write attempt (it should see none)
- Communicate: "service restored"

Phase 6: Post-incident

Within 24-48 hours:

- Post-mortem meeting
- Timeline reconstruction
- What worked, what did not
- Update the runbook
- Improve the backup / DR

Rollback

Rollback in this runbook is not “put DNS back”. The scope of this runbook is a lost region: by definition there is usually no serviceable region to go back to. Decide with one question.

Has the recovery region accepted a single write?

• No - rollback is available. Unfence the original region, restore DNS, and confirm there is exactly one active write path before you announce recovery.
• Yes - rollback is closed. Pointing DNS back now sends clients to a dataset that is missing every write made since cutover, while the recovery region keeps taking new ones. That is not a rollback; it is the creation of a second production database. Keep the recovery region authoritative, leave the original fenced, and escalate to the data team for reconciliation.

Where the isolation was performed for a security reason - ransomware, suspected compromise - rollback is closed regardless of writes, until security signs off. Returning traffic to a compromised region is a security regression, not a recovery.

Common patterns

Escalation

Escalate when:

• The original region cannot be positively fenced - escalate before cutover, not after.
• Writes are observed in both regions. Freeze both write paths first, then escalate.
• Recovery is not on track within RTO.
• Data is missing or corrupt.
• A dependency is unavailable.
• Security implications are unclear.
• The runbook does not cover the situation.

Bring: timeline, current state, attempted actions, blockers.

Knowledge check