Full cluster loss recovery - the worst-case scenario

Full cluster loss is the worst-case DR scenario: every host in a region is gone. This lesson covers the runbook and the discipline for recovering.

The scenario

A data centre is destroyed, a region has an extended outage, or ransomware has encrypted every host. Every node in the cluster is gone.

The recovery: rebuild the cluster in a different region from backups, restore data, restart services.

The runbook

FULL CLUSTER LOSS RUNBOOK
=========================
Scenario: data centre X is destroyed
Goal: bring up the service in data centre Y within 4 hours
RPO: 1 hour

Phase 1: Communicate (0-15 min)
- Notify incident management
- Notify security, operations, business
- Establish a war room (video call, chat channel)
- Identify scope: how many hosts, which services, which data

Phase 2: Provision (15-60 min)
- Provision new capacity in the recovery site
  (cloud instances, new hardware)
- Configure network: VPC, subnets, security groups
- Bring up DNS in the recovery site

Phase 3: Restore (60-180 min)
- For each host:
  - Install OS
  - Apply configuration management
  - Restore data from backup
  - Restore secrets from vault
  - Start services
- Restore central identity (LDAP, AD)
- Restore monitoring (Prometheus, Grafana)

Phase 4: Validate (180-240 min)
- Smoke test each service
- Validate data integrity
- Check monitoring and alerting
- Test user-facing workflows

Phase 5: Cutover (240 min)
- Update DNS to point to new region
- Update load balancer
- Communicate to users: "service restored"
- Monitor for issues

Phase 6: Post-incident (next day)
- Post-mortem
- Update runbook
- Improve backup and DR

Communication

Communication is critical:

• Incident management.
• Security (if the loss is security-related).
• Operations (who is doing what).
• Business (impact and ETA).
• Customers (status updates).

Pre-written templates save time during the incident.

Coordination

Full cluster loss is a multi-team effort:

• Operations: rebuild infrastructure.
• Security: investigate the cause.
• Application: restore services.
• Database: restore data.
• Network: provision connectivity.

A runbook with clear responsibilities prevents confusion.

Test quarterly

The DR test for full cluster loss:

• Run in a separate region from production.
• Bring up the full stack from backups.
• Validate against the baseline.
• Time the whole procedure.

Annual minimum; quarterly preferred. The cost of the test is hours. The cost of unpreparedness is days.

Knowledge check