DR testing: fire drills, partial failures, and chaos engineering

A DR plan that has never been executed is a wishlist. This lesson covers the spectrum of DR testing — from tabletop walks to live chaos engineering — and shows how to design tests that actually find weaknesses in your plan.

The DR testing spectrum

Five levels of testing, increasing in realism and risk:

Level	Description	Risk	Time	What it tests
1	Tabletop walkthrough	None	1–2 hours	Plan completeness, decision points
2	Single-component failure in a test environment	None	2–4 hours	Recovery procedures work end-to-end
3	Single-node failure in production	Low	1 hour	Real impact on real workload
4	Multi-site failover in production	Medium	Half-day	Full plan, communication, real RTO
5	Chaos engineering (random failures in production)	High	Continuous	Resilience to unexpected failure modes

For a new DR plan: start at level 1, advance as confidence grows. For a mature plan: levels 1, 4, and 5 should be regular practice.

Level 1: Tabletop walkthrough

A meeting where the team reads the DR runbook out loud and discusses each step. Takes 1–2 hours. Tests:

Completeness — are there steps missing?
Decision points — who calls DR? When?
Communication — what gets announced? To whom?
Dependencies — what does step 5 require that step 4 didn’t provide?

Run a tabletop quarterly. Invite:

The on-call engineer
The team lead or manager
Someone who was not involved in writing the runbook (catches assumptions)
Someone from another team (e.g., network, security) — catches cross-team dependencies

Document the questions raised and the gaps identified. Update the runbook.

Level 2: Test environment failover

A full failover in a non-production environment. Tests:

Procedures work end-to-end — the commands run, the systems come up
Time estimates are accurate — is the 30-minute RTO real?
Tooling works — PBS sync, ZFS replication, DNS automation, all in one place

Use a separate PVE cluster as the DR site. Replicate from production to the test cluster. Fail over, validate, fail back.

Recommended: once a quarter, scheduled during a low-traffic window on the test cluster (which can be any time since it’s not production).

Level 3: Single-node failure in production

Pull one node out of the cluster and watch the world not end. This tests:

HA failover works for the actual production VMs
HA group configuration is correct
Capacity headroom is sufficient
Monitoring catches the failure
On-call engineers know what to do

# Schedule: pick a low-traffic time. Have someone ready to abort.

# 1. Announce the drill in chat
echo "DR drill starting. pve-02 will be cordoned in 10 minutes."

# 2. Cordon the node (prevent new VMs from being scheduled there)
ha-manager group modify production-grp --nodes "pve-01:100,pve-02:0,pve-03:100"

# 3. Wait 5 minutes for current operations to settle

# 4. Pull the network cable (or stop corosync)
ssh pve-02 systemctl stop corosync

# 5. Watch the cluster
watch -n 5 'ha-manager status; pvecm status'

# 6. After validation (10–30 minutes), restore the node
ssh pve-02 systemctl start corosync

# 7. Wait for HA to rebalance if desired
# 8. Document the result

The “cordon first” step is important — without it, new HA-migrated VMs might land on pve-02 right before you take it down. Cordoning ensures the drill exercises the actual failure mode, not a scheduling race.

Level 4: Full multi-site failover

The big one. Twice a year. Schedule during a low-traffic window (weekend, after hours). Coordinate with the team and customers.

T-minus 7 days: Announce the exercise window
T-minus 3 days: Walk through the runbook (level 1)
T-minus 1 day:  Verify replication is current
T-minus 1 hour: Stand up the war room (video call, screen-share)
T-0:           Declare disaster (real announcement to real channels)
T+0 to T+30:  Run the failover procedure (real, not simulated)
T+30:          Validate DR site is operational
T+30 to T+90:  Hold steady, observe, document
T+90:          Begin failback
T+90 to T+180: Reverse-replication, validation
T+180:         Hand back to primary
T+1 week:      Debrief meeting, update runbook

The single most important thing about a level-4 test: do not cheat. If the failover takes 90 minutes instead of the documented 30, that’s the real RTO — update the plan, don’t paper over it.

Level 5: Chaos engineering

Once you have a tested DR plan, occasional chaos engineering surfaces failure modes nobody predicted. The principles from principlesofchaos.org apply:

Start with a hypothesis. “If pve-02 loses network for 5 minutes, the cluster continues serving VMs with no more than 30 seconds of disruption.”
Vary the experiments. Different failure modes (network, process, storage), different scopes (one node, one VM, one service).
Run in production. Production is the only environment where the real failure modes happen. Test in production.
Automate the experiments. A scheduled chaos run that exercises failure modes on a regular cadence finds problems before they find you.
Minimize blast radius. Each experiment should affect the smallest possible scope. Use cordoning, canary groups, and feature flags.

For a PVE cluster, a chaos script:

#!/bin/bash
# /usr/local/bin/chaos.sh — runs a random failure mode against
# the cluster. Use sparingly.

set -euo pipefail
NODES=(pve-01 pve-02 pve-03)

chaos_pick_failure() {
  local modes=("network_isolate" "process_kill" "disk_pressure")
  echo "${modes[$((RANDOM % 3))]}"
}

case "$(chaos_pick_failure)" in
  network_isolate)
    TARGET="${NODES[$((RANDOM % 3))]}"
    echo "Isolating $TARGET from cluster network for 60s"
    ssh "$TARGET" "iptables -I OUTPUT -d 239.192.0.0/16 -j DROP"
    sleep 60
    ssh "$TARGET" "iptables -D OUTPUT -d 239.192.0.0/16 -j DROP"
    ;;
  process_kill)
    TARGET="${NODES[$((RANDOM % 3))]}"
    TARGET_PID=$(ssh "$TARGET" "pidof pveproxy" | head -1)
    echo "Killing pveproxy on $TARGET (PID $TARGET_PID)"
    ssh "$TARGET" "kill -9 $TARGET_PID"
    ;;
  disk_pressure)
    TARGET="${NODES[$((RANDOM % 3))]}"
    TARGET_VG=$(ssh "$TARGET" "vgs --noheadings -o vg_name | head -1 | tr -d ' '")
    echo "Filling disk on $TARGET until 95%"
    ssh "$TARGET" "dd if=/dev/zero of=/var/lib/vz/fill.bin bs=1M count=100000 || true"
    sleep 300
    ssh "$TARGET" "rm -f /var/lib/vz/fill.bin"
    ;;
esac

Run this monthly, during low-traffic windows, with monitoring on high alert.

Partial failures are the most useful tests

A clean “node goes down, HA recovers” is the easy case. The real world has partial failures:

One node’s network is dropping packets at 5% loss. Corosync has retransmits but doesn’t disconnect. HA doesn’t fire. The cluster “works” but slowly.
One VM’s disk is unhealthy. VM continues running, HA doesn’t move it, but I/O is failing on writes.
Storage backend is degraded. Ceph with one OSD down. The cluster continues but recovery is in progress and impacting performance.
One service is hung but not crashed. HA doesn’t fire because the process is still running.

Test these scenarios specifically. Each one is harder to detect and recover from than a clean crash.

Building a culture of testing

The hardest part of DR testing isn’t technical — it’s organizational.

Make it routine. Quarterly fire drills scheduled a year in advance, on the calendar.
Make it blameless. When the test reveals a gap, document the gap, fix it, move on. Don’t punish the on-call for finding problems.
Make it cross-team. DR usually involves networking, security, facilities, and customer communication. Get them in the room.
Make it documented. Every test produces a report. The report feeds the runbook.
Make it visible. The team knows when DR tests are happening. The org knows the DR plan exists. The customers know what to expect.

Common mistakes

Testing only the happy path. The first 10 minutes of a failover work. The interesting failure modes are in the next 90.
Skipping failback. Failback is harder and breaks more often. Test it.
Testing once. A test from two years ago proves the plan worked two years ago. Test quarterly.
Skipping the documentation update. A test that finds gaps but doesn’t fix the runbook is a waste.
Cheating. If the failover takes 90 minutes and the runbook says 30, that’s the real RTO. Don’t pretend.

Production considerations

Test on production where safe. Test environments have different workloads, different monitoring, different failure modes. The real failure modes only appear in production.
Have a kill switch. Any chaos experiment must be reversible. If something goes wrong, you abort.
Notify before testing. A surprise failover during business hours is a customer incident. Even a low-risk level-3 test should be announced.
Track metrics. Test frequency, RTO achieved vs target, RPO achieved vs target, gaps identified and fixed.

Key takeaways

Test at multiple levels: tabletop quarterly, full failover semi-annually, chaos engineering monthly.
Partial failures are more useful than clean crashes.
Document every test and update the runbook with findings.
Make testing routine, blameless, and visible.