Disaster scenarios - what to plan for · Linux

Disaster scenarios are the events that disrupt service. The DR plan must address each scenario. This lesson covers the common ones and how to design for each.

Common scenarios

Scenario	Likelihood	Impact	Plan
Single disk failure	High	Low (one host)	RAID, snapshots
Host hardware failure	Medium	Medium (one host, RTO = restore time)	Backup, automation
Data centre loss	Low	High (many hosts)	Offsite backup, cross-region
Ransomware	Medium	High (data + backups)	Immutable backups, segmentation
Accidental deletion	High	Medium	Backups with versioning
Database corruption	Medium	High	Application-consistent backups
Configuration error	High	High	Version control, configuration management
Network partition	Medium	High	Multi-region, redundant paths
Security breach	Medium	High	Incident response, backups

Plan for each

Single disk failure

RAID (1, 5, 6, 10) for redundancy.
Hot spare or fast replacement.
Verify RAID rebuild after replacement.

Host hardware failure

Backups in a different host.
Restore procedure tested.
Hardware replacement SLA (next-business-day).

Data centre loss

Offsite backups (different region).
Cross-region replication for critical data.
DR runbook for bringing up in new region.

Ransomware

Immutable backups (S3 Object Lock, tape rotation).
Network segmentation (production backups on isolated network).
Tested restore from immutable backup.
Retention long enough to outlive attacker dwell time.
Cyber insurance.

Ransomware does not behave like the other scenarios on this list, and planning it like a hardware failure produces a plan that fails on the day.

Your RPO for ransomware is not your backup interval

Encryption is the last stage of an intrusion. The attacker gains access, establishes persistence, moves laterally, exfiltrates, and only then encrypts. The gap between initial access and encryption is dwell time, and it is routinely measured in weeks.

So the most recent backups are the ones most likely to be contaminated. They contain either already-encrypted files or the attacker’s persistence mechanism - the cron job, the authorized_keys entry, the trojaned binary, the backdoored service unit. Restoring the newest backup is the fastest way to be encrypted a second time.

The usable recovery point is the last backup predating initial compromise. That is bounded by dwell time, not by how often you take backups. Taking hourly backups does not give you an hourly ransomware RPO.

The consequence for design: retention must exceed your worst-case dwell time, or there is no clean restore point left to return to. A 30-day retention policy against a 60-day dwell time means every surviving backup is contaminated, and no amount of immutability helps.

The consequence for response: forensics gates the restore.

1. Isolate. Do not power off - memory is evidence and some
   variants destroy keys on shutdown. Pull the network.
2. Preserve. Snapshot disks and capture volatile state
   before anyone starts "fixing" things.
3. Establish time of initial access from logs, EDR
   telemetry and the backup timeline. This is the step that
   sets the real RPO, and it takes days, not hours.
4. Select a restore point PREDATING that timestamp. Never
   default to the newest backup.
5. Restore into an isolated network. Scan the restored
   data before it is reachable from anything.
6. Rotate every credential present in the restored data -
   service accounts, SSH keys, API tokens, database
   passwords. They are all in the attacker's hands.
7. Only then reconnect.

Two habits make this survivable: keep one immutable copy older than your dwell-time estimate, and rehearse a restore from a copy at least 30 days old, so you find out in a drill rather than in an incident that your oldest usable backup is not usable.

Accidental deletion

Backups with versioning (BorgBackup, S3 versioning).
“Soft delete” with retention period.
Restore procedure tested.

Database corruption

Application-consistent backups (FLUSH + snapshot).
Point-in-time recovery (PITR) for transactions.
Tested restore.

Configuration error

Configuration management (Ansible, etc.) with version control.
Rollback procedure.
Tested restore.

Scenario matrix

For each scenario, document:

Likelihood (high/medium/low).
Impact (high/medium/low).
Detection method.
Response procedure.
RPO and RTO.
Test frequency.

SCENARIO: Ransomware on production
Likelihood: medium
Impact: high
Detection: file integrity monitoring, anomalous behaviour,
           canary files, EDR alerts
Response: isolate (network, not power), preserve evidence,
          establish time of initial access, THEN select a
          restore point predating it
RPO: NOT the backup interval. Bounded by attacker dwell
     time, commonly weeks. Retention must exceed realistic
     dwell time or there is no clean restore point.
RTO: days, not hours - forensics gates the restore
Test: quarterly restore from an immutable copy at least
      30 days old, into an isolated network

Contrast that with the hardware-failure row, where RPO genuinely is the backup interval because nothing is adversarial about a dead disk. Ransomware is the one scenario in the matrix where the attacker chooses when you find out, and they choose the moment your backups are least useful.

Prioritise

Prioritise scenarios by likelihood * impact:

High likelihood + high impact: top priority.
High likelihood + low impact: monitor.
Low likelihood + high impact: insurance / runbook.
Low likelihood + low impact: skip.

The top 3-5 scenarios get the most attention. The rest get basic coverage.